Bidirectional streaming?

[pachacamac]

Thank you for creating this. Definitely nicer than scheduled rsync tasks :)

I was wondering if it is possible to use it in a bidirectional way so having e.g. two servers receiving (mostly distinct but not always) data, syncing to each other. Or will this most likely result in inconsistencies or crashes?

[benbjohnson]

Hi Marc. 👋

Unfortunately, I don't think Litestream would be a good fit for that. Litestream works by replicating whole physical pages of the database so it would potentially overwrite updates if they were on the same page from each database.

Have you looked at using the SQLite session extention to exchange changesets between databases?

[pachacamac]

Hey Ben,

Thanks for getting back so quickly :) Gotcha! I'll check out the session extension but most likely will end up playing with rqlite as it sounds more targeted to what I was looking for.

Cheers

[warmchang]

e.g. two servers receiving (mostly distinct but not always) data, syncing to each other.

Hi Marc,
The rqlite is base on Raft consensus algorithm, so the system is better to have 2N + 1 nodes (odd number).

[jmillerv]

Have you looked at using the SQLite session extention to exchange changesets between databases?

Would it be possible to use the session extension in conjunction with litestream? I'm still investigating this as an option for my use case.

[benbjohnson]

Would it be possible to use the session extension in conjunction with litestream?

I think there's some interesting setups you could make with the session extension but I'm not sure how it'd work with Litestream. Litestream relies on physical replication (e.g. the raw page data) so there's no way to merge changes from two databases which both act as primaries.

You could make a system that shares updates based on the session extension to a bunch of servers but that gets more into the realm of CRDTs and that's a whole different can of worms.

What's the use case you're trying to solve for?

[jmillerv]

Ideally, it's something akin to a p2p relational database.

But specifically I want a device to create an SQLite database and be able to have 3-10 other devices stay in sync with each other with each device capable of writes.

These devices would largely be behind residential IP addresses. It's fine if there has to be some central coordination.

Eventually consistent is also fine.