Timeout if the broker handling client request is shut down

[rbnacharya]

I implemented PIGATO with following architecture,

I placed 2 brokers and 2 workers

broker 1 - have registered 2 workers
broker 2 - have registered 2 workers

Client use both the broker, and when we request it just go through any one,
and, if I stopped the broker that is handling request then client will receive timeout, and is not redirected to other broker.
(this seem to be message get lost)

Is there any way to distinguish this timeout from timeout when client fails (or client didn't send the response? )

[prdn]

Zeromq allows clients to use multiple brokers in the Smsense that if a client sends multiple requests, these are ventilated to brokers with a round robin logic. Otherwise you will have duplication.

Right now the retry in case of timeout is left to the user that can handle resubmit the request again in case of timeout.

However we can do it automatically adding an option to the request that forces client to retry sending it.

Sounds good?

[TCMiranda]

Would help me as well. As you just said, i'm doing it manually with a requests interface.

[TCMiranda]

Seems like zeromq is not meant to provide this kind of security, right?

[prdn]

Zeromq is a socket layer basically so you can use the available patterns to create what you need.
This is pretty straightforward so will be available soon

[rbnacharya]

@prdn It seem to be good, but for a task we will not be able to know if client failed or did the broker.

Eg. If there is any write command issued to worker (and after request reached to worker, if broker goes down), and worker finished the write, If we retry sending by setting in client, then there will be multiple writes right? Which is a big problem.

[prdn]

Yes that's true. The client retry will be optional and disabled by default. So you can use in a scenario where you a duplicate requests doesn't affect your system.

Think about a query for some data. It would be helpful to resubmit the request automatically in case of timeout with a limit of attempts.

In more delicate scenarios you should handle the timeout manually so you can check before resend.

Have I answered your question?

[rbnacharya]

Yes, Thanks.

But, Will not it be good if PIGATO support adding multiple broker to worker, so that worker can send response to any available or in anyway do that. (as an enhancement )

[prdn]

That's not easy to do this unless a client sends the same request to multiple brokers. Because a broker needs to route the response back to the client.
But sending a request to multiple workers can result in multiple executions. Otherwise we should synchronise brokers but this is against the scalability pattern in my opinion

[rbnacharya]

I guess you are right, but I don't understand how will that be against scalability pattern. Because in my opinion if broker knows another broker's socket, it can just ping (or send heart beat).

[prdn]

Yes but it may happen that broker for any reason fail to synchronise. What would happen if they are in charge both to forward the same request to an available worker?

[TCMiranda]

The message could contain the data necessary to reply to the client.

[rbnacharya]

A use-case can be,

1. Client send request to Broker (any),
2. Broker decide which worker (and broker) handle request [can be any shortest path algorithm], and create metadata about that. The metadata is shared to all broker before forwarding message (with metadata) to worker.
3. Worker, when sending response if destined broker failed, then send to another active broker (with metadata), and another broker help to transfer data to the client.

[TCMiranda]

In this case, all clients must always be connected to all brokers, right? Is it bad?

[prdn]

Clients may not be aware of all available brokers.
Clients may connect to multiple brokers but not be connected to all the brokers.
Suppose you have 20 brokers and 10000 clients. All clients cannot connect to all brokers or your distributed system will be overloaded and slow.

A solution could be introduce routing solution for failed deliveries.
If a worker recognise a failed delivery tries to ask other brokers to forward the reply.
If a client receives multiple replies for the same request discards all but the first.

[leonpegg]

@prdn This seems like a logical solution but not something that should be implemented in the Client/Broker/Worker core.
We could maybe look at introducing a plugin system for extra features, if a client/broker/worker tries to use a feature provided by a plugin and its not present it should still be able to use all core features and the client/worker/broker should handle the missing plugin problem.

This would allow us to keep the core clean and usable even when plugins are provided by third party's

[TCMiranda]

I can't see much where to fit some plugins pattern, maybe emitting some events to the worker and the client, so it would be easy and possible to take actions from the client when, for example, a message delivery to the worker failed.

Those kinds of messages could broadcast to all brokers, so one of them could delivers it to the client (if needed, like a broker went down)

[prdn]

Take a look here #21
I have described a possible workflow for the map-reduce functionality.
Everything should be a Service. The Broker should only offer basic features to let people develop special workers implementing complex features.

Special Workers will be deployed in external projects (i.e pigato-mapreduce)
I'm pretty sure that we can rely on this idea also for Broker-Broker communication.

[rbnacharya]

@prdn
A question,
Is there any events in Broker, that execute on,
like onMessage received, or onClientConnected , onWorkerConnected ?

If not, Don't you think it will be good to have?

[leonpegg]

@rbnacharya No currently there are only the error event fired.

I have been looking at adding client/worker join to the broker as its something we require.

There is nothing stopping extra event being added and its straight forward to do with minimal effort