Feature Request: Documentation of some example uses of Kinesis

[ryanwalls]

I have some ideas of what is possible, but it would be great to have some concrete examples in the docs of how you guys use Kinesis. We're debating using it as a way to aggregate metrics about total activity running time for workflows with a given tag... not sure if that is within the scope or how delayed data results would be. Never really used Kinesis.

[sclasen]

Hey @ryanwalls we use it in a CQRS kind of manner.

The kinesis replication hander puts json marshalled SerializedState structs onto kinesis, using the workflow id as the 'key'

We then use https://github.com/sclasen/streambolt in two ways.

First we have an operational workflow always running in each swf domain, that every hour updates a boltdb database, by reading a previous snapshot, updating that with whatever is currently in kinesis, and writes that as a 'snapshot' into s3.

Then all of our instances that want to be aware of whats going on without heavyweight queries into SWF simply grab the latest snapshot on startup, and then continuously consume the kinesis stream and apply it to to the local boltdb. (We use the stateVersion of the SerializedState as an 'optimistic lock' and only apply later versions)

At that point instances can just query the boltdb locally to service requests.

The streambolt stuff should totally be flexible enough for your usecase, you can write the data that comes in in the SerializedState object to bolt in any way that works for you.

[ryanwalls]

@sclasen Thanks! Will have to see if we can adopt this pattern. Good stuff.

[ryanwalls]

Would be handy to have some of this in the documentation of the project somewhere. Have you guys considered maybe using something like gitbook or other tool to write some more in depth documentation? I feel the current examples only scratch the surface of what's possible. Speed vs documentation tradeoffs are hard...