Use Cases

[alexellis]

Hey, you messaged me today asking for feedback. I wrote you it, then you asked me to copy it here, so here we go

I would like to see some usecases. Can it make single node k3s more durable?

I use Ghost with SQLite in a container, how can I replicate it to S3?

Does this have an application with Android?

Will you be adding WAL2HTTP so I can trigger functions on row changes?

[benbjohnson]

Hi @alexellis, thanks for posting here.

Can it make single node k3s more durable?

Yes, I would think so. Is the SQLite database on a Docker mount? I haven't used k3s before so I'm not too familiar. Litestream should replicate any SQLite 3 database.

I use Ghost with SQLite in a container, how can I replicate it to S3?

If you're running it from a mount that's visible outside the container then you should just be able to run the litestream command and point at that file. I haven't tried running Litestream from inside a container yet but I don't see why it wouldn't work.

Does this have an application with Android?

How do you mean? Like a release build for Android? I don't think there's anything preventing it from working on Android (at least maybe as a library if not a process) but I haven't tried it yet. What's the use case you're thinking of?

Will you be adding WAL2HTTP so I can trigger functions on row changes?

Probably not. That gets a lot more complicated. The WAL only has the individual page data so I'd need to look at the ptrmap pages to traverse up the B-tree to figure out what table the page belongs to.

[readall]

Sorry if I am hijacking this discussion thread, as the thread is named Use cases, my query may fit here.
There are other cloud storage services which are S3 compatible (for example Wasabi).

Is it possible for Litestream to work with non-aws S3 compatible (the cloud storage api remains same as S3)? This will allow flexibility for users who look for alternatives for various reasons.

[aschrijver]

Thanks for considering. The rclone landing page has a big list. I guess it is not needed to natively support multiple of these, just providing others the opportunity to do so - i.e. by separating the storage parts to facilitate this - would be a boon. Thank you, I will follow this great project.

[nkev]

@aschrijver Couldn't you use rclone to mount a local volume and have litestream write to that local volume (remote location)?

[aschrijver]

Yes, it might be an option. I haven't yet looked into code and specifics that much (currently planning an app based on postgres). It may be easy to facilitate, and possibly only changes to Litestream would help make it a breeze to do so. Thanks!

[tomberek]

Random thought. Is there anything specific to S3 being used? Otherwise github.com/philippgille/gokv or similar can give instant access to a large number of key/value stores.

[benbjohnson]

There's nothing specific for S3 right now although there could be optimizations in the future. I'm hesitant to add gokv for two reasons. First, it adds a layer of abstraction and makes it more difficult to tweak/optimize any specific implementation. Second, I'm not sure it adds a lot of value in that people won't use most of those K/V stores and it's a lot to try to support. Replicating SQLite WAL frames to another local K/V store or to a database server doesn't really make much sense.

The S3 API is pretty ubiquitous among blob stores so I'll probably stick with that until there's a good reason to expand to other backing stores.

[nhaberl]

I really like your motivation for this project, I can totally feel with you.
So my question is, would it be possible to replicate 100s of databases (just a few megs) concurrently?
It seems litetsream acts as a single background process.

I would like to use it in a multi tenant environment...where also the replication process acts per tenant and not globally per machine.

Or would you see any performance issues?
My guess is that performance decreases with every additional tenant?

Anyway, keep goin...its nice project!

[benbjohnson]

I haven't tested with hundreds of databases. Currently, you'd probably see some performance degradation. Although, I may change the Litestream monitor to something like fsnotify to monitor for database changes rather then polling. Once that's implemented, I would expect much per-database overhead.

[nhaberl]

You mean you would not expect much overhead ?
...just to be sure

[benbjohnson]

Sorry, yes, I would not expect much overhead after the fsnotify change is implemented.

[nhaberl]

More than welcome to see then 🤩

[nkev]

@benbjohnson

I may change the Litestream monitor to something like fsnotify to monitor for database changes

Checkout this repo. It does exactly that.