Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add an implementation of kvdb on top of IndexedDB #182

Closed
tomaka opened this issue Jun 25, 2019 · 3 comments · Fixed by #202
Closed

Add an implementation of kvdb on top of IndexedDB #182

tomaka opened this issue Jun 25, 2019 · 3 comments · Fixed by #202
Assignees

Comments

@tomaka
Copy link
Contributor

tomaka commented Jun 25, 2019

Add an implementation of the kvdb traits on top of the Web IndexedDB API: https://developer.mozilla.org/fr/docs/Web/API/API_IndexedDB

The bindings to IndexedDB are already present in the web_sys crate (https://docs.rs/web-sys/0.3.24/web_sys/struct.IdbFactory.html?search=Idb), and we should use this crate to access IndexedDB.

The use-case for this is running Substrate (or parity-ethereum?) from within a browser.

I'm personally not very familiar with neither kvdb nor IndexedDB, so there might or might not be problems that I can't think of.

@ordian
Copy link
Member

ordian commented Jul 1, 2019

I think the main problem is that kvdb exposes sync API like fn get(&self, col: Option<u32>, key: &[u8]) -> io::Result<Option<DBValue>>, whereas IndexedDB provides an async API with js-style callbacks.

From https://developer.mozilla.org/en-US/docs/Web/API/IDBObjectStoreSync:

Important: The synchronous version of the IndexedDB API was originally intended for use only with Web Workers, and was eventually removed from the spec because its need was questionable. It may however be reintroduced in the future if there is enough demand from web developers.

@tomaka
Copy link
Contributor Author

tomaka commented Jul 1, 2019

Oh, I see.

Maybe a better approach is simply to keep the database in memory (like kvdb-memory), and save a serialized version in the local storage from time to time and on shutdown.

The use case is a light client whose database is probably very small. I imagine that the light client will somehow warp-sync in the future, and storing the database is mostly an optimization.

@ordian ordian self-assigned this Jul 22, 2019
@ordian
Copy link
Member

ordian commented Aug 2, 2019

While trying to make this work, I've encountered a bunch of issues:

Some problems of localStorage:

  • The API allows you to use only JS string (UTF-16) as keys and values
  • It usually has a limit of 10MB (https://arty.name/localstorage.html)
  • localStorage API is not available in WebWorkers, meaning it's not possible to save a serialized version in a background thread.

IndexDB API is not ready yet

I'm not sure what is the expected size of of the substrate light client db, but we could implement it with localStorage first and migrate to IndexDB later (I imagine having an in-memory hashmap in both cases, reading the whole db into the memory on startup and writing async in case of indexdb).

wasm-bindgen limitations

This means that kvdb KeyValueDB interface would need some breaking changes. Alternatively, we could create another (async?) interface just for the browser use-case.

EDIT: the wasm-bindgen limitations are only relevant if we want to expose the KeyValueDB interface to JS, but looks like we don't.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants