Question about the Firebase example / future Firebase support

[cristiano-belloni]

Hello,
Thanks for this interesting library! Love hitchhiker trees!
I just tried the Firebase example locally and I see that the application starts with a blank db everytime, then just gets all the records from the database. That wouldn't scale when the number of entities is high. Also, since the db is not persisted locally, there is no way to sync only the latest updates (and for this reason deletions / retractions are "real" deletions, instead of tombstoning).

I guess it might be possible to make a full syncing prototype working with Firebase as a backend - probably with tombstoning, a lastUpdated special field with the server timestamp of the last modification and the client using the REST api to query and get only the latest updates, while Firebase would automatically do conflict resolution.

Do you plan to support something similar? I've looked at similar solutions (rxdb, dexie, y.js), but they only support websocket / rest / graphql sync, which are not really serverless. Firebase has the advantage of integrating auth, permissions, lambdas and database in a single package, so (imho) having a supported Firebase backend would be a killer feature that no other project really has.

[rubinovitz]

Hi @cristiano-belloni, super appreciate you sharing this idea here!

Presently, we are working on persisting locally with indexedDB and then will be adding a CRDT for syncing.

The tombstoning idea is really interesting for a more near-term solution. Tagging @smothers for thoughts.

[cristiano-belloni]

Yes, obviously client persistence is needed to do any syncing on the server. I like the simplicity of having the addTransactListener callback to push entities to the server and the transactSilently method to push to the database / client persistence layer after pulling from the server. If it continues working like that, it would be easy to write a Firebase adapter - records can be pushed with a lastUpdated field and they can be partially queried using the same field. Timestamps would be generated on the server, ensuring strict ordering and deleted entities can be tombstoned and even compressed (all content removed, apart from the lastUpdated and deleted field), since Firebase charges per bytes downloaded. In the end it would be a simple CRDT layer built a complex CRDT-like database. Wouldn't mind even trying myself (in js/ts, though, since I love Clojure but I'm just dipping my toe in it :))

[cristiano-belloni]

(also, user permissions would essentially be for free - but then it's worth knowing what should happen if the DB refuses an operation due to auth rules but the client layer persists the transaction because it knows nothing about authentication. That might desync one client from all the others)

[becomingbabyman]

Hey @cristiano-belloni. We're clearly thinking along the same lines.

The Firebase example is just a demonstration of how you can connect to a backend API. It's a bit overly simplified. As you've said, you'd probably make some slightly different decisions to make it scale, and it does not cover the local-first use case at all.

That said, local-first is very much on our roadmap. We're working with the Datahike team to handle offline caching and sync. Authorization of local-first data is something we'll be addressing with our hosted service.

In the meantime, I'm happy to help talk through alternative architectures to achieve the things you mention.

Our hope in releasing homebase-react for basic client-side state management ahead of all of these local-first features is that we can seamlessly upgrade clients without changing the API.