delegate sendMessage to the provider service

[sjfink]

The implementation already maintains provider service instances to maintain long-lived kafka connections.

We currently use these connections for feeds but not for outgoing messages.

We should consider implementing sendMessage via delegation to the provider service, to avoid the overhead of setting up a connection for each message.

[jberstler]

I assume that the benefit of this is that the total time from when the action is invoked and the message is produced to the Kafka instance is on average lower, by way of not taking the time to establish a Kafka connection every time a message is produced.

Implementing this means that the action must have a reliable communication method to the provider service in order to signal the delegation intent. In order for this to to have the above stated benefit, the time of signaling delegation to the provider service, plus the time to (potentially) re-use an existing Kafka connection to produce a message must be less than the time it takes to produce a message using a new Kafka connection. In the worst case (when the established connection does not yet exist) this is always slower than the current implementation, meaning that the best case should be significantly faster in order to make up the difference.

My gut feeling is that HTTP connection overhead is at least that of establishing a Kafka connection. While redis may (I really don't know...) be faster, I'm not certain that it is 100% reliable for this kind of scenario. While Kafka is reliable, obviously the connection overhead is the same.

@sjfink what do you think?

[sjfink]

You might be right -- I don't know the overhead of establishing an HTTP connection vs. a kafka connection. I'd suggest doing an experiment and let's see what the numbers say.

Note that I suspect overhead on the message producer is the most important thing in most cases -- for asynchronous patterns the latency after it produces a message is likely less important.

I suspect a typical web server is engineered to support thousands of connections per second. Kafka -- maybe not so much?

I think a higher-level claim is that we should think about a "real" solution that avoids establishing a new connection each time. I realize this might be a major, intrusive change to the system ... it might involve having some sort of "connection proxy" running on each invoker and a low-overhead low-latency communication channel to the proxy. There are lots of engineering issues -- I haven't thought through the details. I'm sure there are lots of implementation choices (grpc?). Might be a research topic to figure out best way.