How to limit jobs to run one at a time for a given tenant?

[ashleyww93]

We have a complex product, and some jobs can take a long time to complete.

We have a bunch of queues, some of which have restrictions on them such as: "Only allow a single job from any queue to run for a given tenant at a given time".

For example, lets say we have these queues:

1. SubscriptionCheck (Single job per tenant)
2. ProductData (Single job per tenant)
3. GenerateFilters

And our payloads always include a tenantId for queues that have the single job restriction.

In this example, if a job is running for SubscriptionCheck for Tenant A, then even if there are 10 other jobs for Tenant A in SubscriptionCheck or ProductData they should NOT be run, and then ideally once that job finishes, the old job in any queue for Tenant A would be triggered next.

Currently, we handle blocking jobs in Redis by using locks, but I would love to use trigger.dev instead.

Is this possible with trigger.dev?

[TriPSs]

You could checkout concurrent-run-executions

[matt-aitken]

@ashleyww93 sorry for the slow response to this, I don't get notifications for these so totally missed this.

This isn't currently possible, we don't have per-tenant concurrency controls.

In v3, which we're working on, we will have more concurrency controls. You'll be able to define queues and attach them to Jobs (including sharing them across Jobs, which is what you need to do for your use case). Then when you trigger the Job you can specify a concurrencyKey – that will effectively create a copy of that queue with that key.

To be more concrete:

const serialTenantQueue = queue({
  name: "serial-tenant-queue",
  concurrencyLimit: 1,
});

export const subscriptionCheck = task({
  id: "subscription-check",
  queue: serialTenantQueue,
  run: async ({ payload }: { payload: number }) => {
    return "something";
  },
});

export const productData = task({
  id: "product-data",
  queue: serialTenantQueue,
  run: async ({ payload }: { payload: number }) => {
    return "something";
  },
});

subscriptionCheck.trigger({
  payload: 123,
  options: {
    //this effectively creates a serialTenantQueue for each unique concurrencyKey
    concurrencyKey: "usr-12345",
  },
});

//this won't run until the above task is finished because they share the same queue + concurrencyKey
productData.trigger({
  payload: 123,
  options: {
    //this effectively creates a serialTenantQueue for each unique concurrencyKey
    concurrencyKey: "usr-12345",
  },
});

[ashleyww93]

No problem about the delay.

It’s good to see this is in progress. Do you know roughly when this might be available?

We currently use long running servers for our jobs, so we are looking this ticket (critical for us) and then on if we move to serverless jobs or stay long running (which I see an open discussion for)

[matt-aitken]

I'm closing this as we have per-tenant queue support in version 3. Learn more and get early access: https://trigger.dev/blog/v3-developer-preview-launch/