Windows and aggregation

[illotum]

While I understand that you intend to be compatible with Kafka Streams, are there any plans to do windowed processing in WASM? I am most interested in sliding window aggregation, triggered by timeout or incremental doesn't matter.

[emaxerrno]

hi @illotum - great idea. We haven't added this to the roadmap yet, but good thinking. I bet you can do it yourself.

Currently the async engine is stateful so you could buffer by window time and then callback. but not out of the box atm. building a window aggregator is not too challenging if the windows fit inmemory.

[emaxerrno]

Also turning this into a discussion.

[illotum]

Right, should've gone here from the start. Can the engine replay disrupted batches? Or does it offer fault tolerance of some other type?

[emaxerrno]

the only policy we have now is at-least-once for local, so you'd just need to track the offset but this is different from other conditions like WAN based failure policies. it is very rare we actually replay an offset unless we crash. so you already get replays.

[emaxerrno]

also, @graphcareful just added a policy that when you register the wasm script you can start from stored, earliest, latest so you get to choose if you want to reproc the full stream.

[graphcareful]

Yeah the stored policy is currently supported, so in the case of failures your coprocessor will reprocess from where it left off.