streaming interfaces

[mikedewar]

Build interfaces for:

• SQS
• Kinesis
• Kafka
• RabbitMQ
• DBus

any others? Ignore any of the above?

[nikhan]

Always down for more but I think Rabbit should be at the top of the list. I don't know anything about Kinesis, Kafka, DBus.

[mikedewar]

• Sources: any block that isn't a regular block
• Store: an in-memory store, that needs to be locked
• Interface: an interface onto an external streaming service

[mikedewar]

Source can:

• GetType()

Interface can:

• Serve()
• Stop()

Store can:

• Lock()
• Unlock()
• Get()
• Set()

[mikedewar]

I'd love to ditch Get and Set - is there a good reason the API needs the generic access afforded by these methods?

[nikhan]

the reason you can get and set is if you want to populate a store with a predefined value conveniently... like for instance, lets say you have a dictionary of country codes on to country names... or a set of precalculated weights or something.

er, what are the reasons to get rid of them?

[mikedewar]

feels like they should be done "in" streamtools, rather than through the API. A one-off read of a country-code JSON should be achievable directly...

[nikhan]

ok, how does one load a store?

[mikedewar]

so I mean - it feels like something that should be able to be done in streamtools directly, with maybe a file-stream interface that pushes its contents line by line upon connect. It's not doable right now.

[mikedewar]

actually (and this is OT now) this could be pretty powerful in the Unix's right? FileConnect specifies a path, receive dumps the file then acts like tail -f, and I guess send appends? Then you'd be able to connect to anything that portrays itself as a Unix file thinger including, I think /dev/stdout?

So I guess I want to add

• files

to the list above.

[mikedewar]

also I guess we should have a disconnect block.

[AKAMEDIASYSTEM]

uh