Is there a way for multiple consumers?

[bennyKY]

Hi, Bull uses Redis pubsub so it wont allow multiple consumers.
But Bullmq uses Redis streams so it should be feasible.
Is there a way to do it with Bullmq? If not maybe someone can advise how to achieve it without Rabbit or Kafka?

[manast]

Bull and BullMQ both support multiple consumers, just create as many workers as you see fit and they will consume your jobs in parallell.

[bennyKY]

@manast
I think i wasnt clear, sorry for that. Im talking about broadcasting. Multiple services that consume from a queue. As far as i worked with Bull (1 year in production) if 2 services listening to messages from a queue only 1 services will receive the message.

[manast]

Ok, so you mean fanout (https://en.wikipedia.org/wiki/Fan-out_(software)), i.e. same job to many consumers. That is not supported and there are no plans for it, in that case using XTREAMS is a really good solution if you do not want to experiment with more complex solutions such as Kafka.

[bennyKY]

@manast thanks a lot for your answers, sorry didnt find anything related to XTREAMS, can you please give me a reference?

[manast]

@bennyKY I meant streams as you wrote on the issue, sorry for the confusion: https://redis.io/commands#stream

[audenaert]

@manast Came across this while searching for the same goal as the OP.

It looks like I could achieve a similar result by listening to queue events. Seems like a bit of an abuse of the intention of BullMQ, but wondering if something along the following lines would get a similar result.

const queue = new Queue('Cars');
await queue.add('paint', { color: 'red' });

const worker = new Worker('paint', paintHandler);
...
const queueEvents = new QueueEvents('Paint');
queueEvents.on('completed', queueListener1);
queueEvents.on('completed', queueListener2);
queueEvents.on('completed', queueListener3);

The downside here (I assume) is that I can't set these up to run in multiple workers because each listener would be called in each worker. That could lead to surprising behavior unless my listeners are idempotent. For example, if the app that's listening scales horizontally, each listener would be called on each running instance.

Am I understanding how Queue listeners work correctly?

[notflip]

@audenaert Did you ever find an answer to your question?

[manast]

The answer to the question is that there will be no surprising behaviour here, the job will be processed by one of the workers and all the global listeners (queueEvents instances), listening to the 'completed' event will receive the event.

[adamlamaa]

Had to eliminate BullMQ because it does not provide fanout, looking at RabbitMQ or LavinMQ instead.

[manast]

Can you elaborate what do you mean with "fanout" ?

[adamlamaa]

Did not see your detailed reply before my follow up question. Thanks for sharing your perspective. I will have a look at just using Redis on its own.

[manast]

Btw, you should probably use the BullMQ version instead of bull: https://docs.bullmq.io/guide/nestjs

[adamlamaa]

The architecture I am looking to implement is the ability to broadcast an event, which can then be picked up by multiple subscribers. Each subscriber can then evaluate on their own if they need to perform any actions as a result of the event.

Enabling a microservice setup, where logic can be split up and the code be structured in a more event driven way.

It was my understanding that in the queue world the pattern that fits this need is fanout.

And yes, was using the nestjs version, thanks!

[adamlamaa]

Update: I was not able to acheive the dream setup I had invisioned in my head without any MQ.

Ended up using BullMQ with a FlowProducer.

[yisibl]

Is this a needed feature for this issue? #2854

[manast]

For fanout you could also use custom global events, not sure if it would fullfil the requirements for a pubsub scenario but seems to be quite close: https://docs.bullmq.io/guide/events/create-custom-events