Kafka Streams?

[OneCricketeer]

Currently this project is plain consumer & producer. Can we get Kafka Streams configuration?

Maybe borrow some ideas from https://github.com/streamthoughts/azkarra-streams

Link streamthoughts/azkarra-streams#28

[natalie-zamani]

I'd love to support Kafka Streams, it's just a matter of me setting aside the time to do it. I welcome any PRs for this before I can tackle it, though!

[OneCricketeer]

I could probably get started on at least the config objects

[OneCricketeer]

Putting my thoughts here

1. Keep TopicsConfig the way it is
2. define generic config map like

streams:
    applicationId:
    ... # all other real kafka configs
    config:  # name TBD
      input-topic: foo
      output-topic: bar

That way, you could start

kstream.from(props.get("input-topic")).map(this::process).to(props.get("output-topic"))

[OneCricketeer]

More thoughts.

Provide ability to chain together processor api via config

stream:
    appid: foo
    source: topic # from this
    output: next # to this 
    steps:
        - step:
           name: filter # human readable 
           type: my-filter # polymorphism type 
        - step: 
            name: convert
            type: my-mapper 
            through: [topic2] # intermediate topics (optional) 
            config:  # optional map 
                prop: foobar 
        

Stream is evaluated top to bottom

[natalie-zamani]

@Cricket007 Though I like the idea, I think it would be out of the scope of this module to define the steps in YAML, at least initially. A better starting point would be the configuration and management of the streams clients first. What do you think?

[OneCricketeer]

I understand. I just wanted to somewhere to write down my thoughts