Write Once - Read Many

[wobbet]

I'm working on a project that I think Litestream would be great for.

I have a daemon that is collecting data and writing to a SQLite database and RESTful API layer that allows reading (but not writing). If the number of readers needs to scale (questionable) I would want to

• Daemon writes to db
• Litestream replicates to common storage
• Readers pull updated data from common storage. "Eventual consistency" is acceptable within some yet to be determined criteria.

The straightforward way to do it is to just have the reader apps copy the entire db file over on a schedule.

If I have a list of all the readers that are running, can I have a different Litestream instance watching the common storage DB and stream changes to the readers? Or is copying the full file over on a schedule "better"?

Thanks!

rjsjr

[benbjohnson]

@wobbet I have live replication (#8) almost working. That'll let you have one writer node and multiple read-only nodes streaming changes in real-time. Is that what you're looking for? I'm hoping to have it finished and cleaned up by the end of the month and then spend most of March doing testing and performance improvements. It'll release with v0.4.0 at the end of March.

[benbjohnson]

Yeah, path 2 is what live replicas will do. I don't have latency numbers but it should be pretty fast. Probably about 10ms plus whatever network latency you have between machines.

Path 3 is something I'm considering in the future. Essentially, it'd be longer latency but it would allow readers to poll something like S3 to check for new changes and integrate them on a regular interval like every minute or hour or something.

[wobbet]

So... if I detect I need to auto-scale and spin up a new reader instance I will need to do something like

• Tell the writer that I'm alive
• Writer updates the Litestream config
• Litestream sends full replica to new, empty reader (???)
• Litestream adds updates as it goes along

And when the reader scales down

• Tells the writer that it's going away
• Writer updates Litestream config
• Litestream no longer pushes updates to that target

Is that reasonable? For some unknown definition of reasonable...

rjsjr

[benbjohnson]

It's a bit simpler than that. The streaming API will be over HTTP so you'll just spin up a reader and it just needs to know who the writer is and will connect via a long-running HTTP connection. If it's empty, it'll get a snapshot of the database. After that it'll stream all updates down to the reader. When you scale down the reader, it'll just disconnect that HTTP connection and be done.

I may add daisy-chaining as well in the future if folks need additional scaling. That would allow you to connect a reader to another reader which is then connected to the writer. It would help lessen the load on the writer. It can also help if you are geographically distributed and have a cluster of nodes in each region and want to minimize cross-region bandwidth costs.

[wobbet]

Y'all are AWESOME! I am incredibly excited!

[lastlegion]

Excited for this!