Use Broadway for stream processing

[benkeil]

Can we extend broadway to use it for stream processing?

I have something like the Kafka Streams DSL in mind.

Let's take join as an example. Like in flow we would define the type of the join, 2 producers, functions to get the id's and a function how to join. But in addition we need also a storage to process infinite streams. This storage could have different implementations, like just RAM, ets, AWS DynamoDB or what ever.

[josevalim]

Hi @benkeil! We welcome folks to explore this idea. From your example above, the functionality that we would need is to add multiple producers of different types to a Broadway topology.

Then everything else can be built on top of Broadway's existing functionality. Joins should partition the producers based on the join ID. The storage can be tackled as a separate concern and so on. Pull requests are welcome!

[whatyouhide]

@josevalim can this be built in userland, if we add multiple producers? If so, we can close this and just keep #36 around?

Otherwise, if we decided to go with no multiple producers, we can probably close this?

[josevalim]

Fair!