Batching Jobs in workers #1742
mbrimmer83
started this conversation in
General
Replies: 1 comment
-
Thanks for your interest in BullMQ. The batch functionality is actually in the roadmap of the Pro version. This functionality will allow you to fetch atomically from the queue up to a defined number of jobs, and pass them to the processor function. The only way to process in batches without this functionality would be if you represent a job as an array of jobs and add the array to the queue as one job. It is not the same as batches but in the cases when you can add several jobs in a batch it could have a similar performance benefit. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
We are evaluating BullMQ to process events in our system. Our two requirements are to rate limit the jobs(don't overwhelm the downstream system) and that we send the processed jobs in bulk to a downstream system(1000 at a time per request). Workers can easily be rate limited, but is there a strategy to process a batch of jobs together and then bulk-send them downstream?
Beta Was this translation helpful? Give feedback.
All reactions