From 6e4c3ebb69f0fa79cae02c46d322a511318502a7 Mon Sep 17 00:00:00 2001 From: luojiyin Date: Mon, 3 Mar 2025 02:46:58 +0000 Subject: [PATCH 1/5] fetch by hand --- ...indexeddb-cookies-opfs-sqlite-wasm.html.md | 302 +++++++++++++++++- 1 file changed, 301 insertions(+), 1 deletion(-) diff --git a/_drafts/Article/Translation/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html.md b/_drafts/Article/Translation/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html.md index ee8bfc24..7df0346c 100644 --- a/_drafts/Article/Translation/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html.md +++ b/_drafts/Article/Translation/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html.md @@ -7,4 +7,304 @@ translator: "" reviewer: "" --- -Skip to main content \ No newline at end of file +So you are building that web application and you want to **store data inside of your users browser**. Maybe you just need to store some small flags or you even need a fully fledged database. + +The types of web applications we build have changed significantly. In the early years of the web we served static html files. Then we served dynamically rendered html and later we build **single page applications** that run most logic on the client. And for the coming years you might want to build so called [local first apps](https://rxdb.info/offline-first.html) that handle big and complex data operations solely on the client and even work when offline, which gives you the opportunity to build **zero-latency** user interactions. + +In the early days of the web, **cookies** were the only option for storing small key-value assignments.. But JavaScript and browsers have evolved significantly and better storage APIs have been added which pave the way for bigger and more complex data operations. + +In this article, we will dive into the various technologies available for storing and querying data in a browser. We'll explore traditional methods like **Cookies**, **localStorage**, **WebSQL**, **IndexedDB** and newer solutions such as **OPFS** and **SQLite via WebAssembly**. We compare the features and limitations and through performance tests we aim to uncover how fast we can write and read data in a web application with the various methods. + +note + +You are reading this in the [RxDB](https://rxdb.info/) docs. RxDB is a JavaScript database that has different storage adapters which can utilize the different storage APIs. **Since 2017** I spend most of my time working with these APIs, doing performance tests and building [hacks](https://rxdb.info/slow-indexeddb.html) and plugins to reach the limits of browser database operation speed. + +[![Image 1: JavaScript Database](https://rxdb.info/files/logo/rxdb_javascript_database.svg)](https://rxdb.info/) + +The available Storage APIs in a modern Browser[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#the-available-storage-apis-in-a-modern-browser "Direct link to The available Storage APIs in a modern Browser") +------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + +First lets have a brief overview of the different APIs, their intentional use case and history: + +### What are Cookies[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#what-are-cookies "Direct link to What are Cookies") + +Cookies were first introduced by [netscape in 1994](https://www.baekdal.com/thoughts/the-original-cookie-specification-from-1997-was-gdpr-compliant/). Cookies store small pieces of key-value data that are mainly used for session management, personalization, and tracking. Cookies can have several security settings like a time-to-live or the `domain` attribute to share the cookies between several subdomains. + +Cookies values are not only stored at the client but also sent with **every http request** to the server. This means we cannot store much data in a cookie but it is still interesting how good cookie access performance compared to the other methods. Especially because cookies are such an important base feature of the web, many performance optimizations have been done and even these days there is still progress being made like the [Shared Memory Versioning](https://blog.chromium.org/2024/06/introducing-shared-memory-versioning-to.html) by chromium or the asynchronous [CookieStore API](https://developer.mozilla.org/en-US/docs/Web/API/Cookie_Store_API). + +### What is LocalStorage[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#what-is-localstorage "Direct link to What is LocalStorage") + +The [localStorage API](https://rxdb.info/articles/localstorage.html) was first proposed as part of the [WebStorage specification in 2009](https://www.w3.org/TR/2009/WD-webstorage-20090423/#the-localstorage-attribute). LocalStorage provides a simple API to store key-value pairs inside of a web browser. It has the methods `setItem`, `getItem`, `removeItem` and `clear` which is all you need from a key-value store. LocalStorage is only suitable for storing small amounts of data that need to persist across sessions and it is [limited by a 5MB storage cap](https://rxdb.info/articles/localstorage.html#understanding-the-limitations-of-local-storage). Storing complex data is only possible by transforming it into a string for example with `JSON.stringify()`. The API is not asynchronous which means if fully blocks your JavaScript process while doing stuff. Therefore running heavy operations on it might block your UI from rendering. + +> There is also the **SessionStorage** API. The key difference is that localStorage data persists indefinitely until explicitly cleared, while sessionStorage data is cleared when the browser tab or window is closed. + +### What is IndexedDB[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#what-is-indexeddb "Direct link to What is IndexedDB") + +IndexedDB was first introduced as "Indexed Database API" [in 2015](https://www.w3.org/TR/IndexedDB/#sotd). + +[IndexedDB](https://rxdb.info/rx-storage-indexeddb.html) is a low-level API for storing large amounts of structured JSON data. While the API is a bit hard to use, IndexedDB can utilize indexes and asynchronous operations. It lacks support for complex queries and only allows to iterate over the indexes which makes it more like a base layer for other libraries then a fully fledged database. + +In 2018, IndexedDB version 2.0 [was introduced](https://hacks.mozilla.org/2016/10/whats-new-in-indexeddb-2-0/). This added some major improvements. Most noticeable the `getAll()` method which improves performance dramatically when fetching bulks of JSON documents. + +IndexedDB [version 3.0](https://w3c.github.io/IndexedDB/) is in the workings which contains many improvements. Most important the addition of `Promise` based calls that makes modern JS features like `async/await` more useful. + +### What is OPFS[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#what-is-opfs "Direct link to What is OPFS") + +The [Origin Private File System](https://rxdb.info/rx-storage-opfs.html) (OPFS) is a [relatively new](https://caniuse.com/mdn-api_filesystemfilehandle_createsyncaccesshandle) API that allows web applications to store large files directly in the browser. It is designed for data-intensive applications that want to write and read **binary data** in a simulated file system. + +OPFS can be used in two modes: + +* Either asynchronous on the [main thread](https://rxdb.info/rx-storage-opfs.html#using-opfs-in-the-main-thread-instead-of-a-worker) +* Or in a WebWorker with the faster, asynchronous access with the `createSyncAccessHandle()` method. + +Because only binary data can be processed, OPFS is made to be a base filesystem for library developers. You will unlikely directly want to use the OPFS in your code when you build a "normal" application because it is too complex. That would only make sense for storing plain files like images, not to store and query [JSON data](https://rxdb.info/articles/json-based-database.html) efficiently. I have build a [OPFS based storage](https://rxdb.info/rx-storage-opfs.html) for RxDB with proper indexing and querying and it took me several months. + +### What is WASM SQLite[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#what-is-wasm-sqlite "Direct link to What is WASM SQLite") + +![Image 2: WASM SQLite](https://rxdb.info/files/icons/sqlite.svg) + +[WebAssembly](https://webassembly.org/) (Wasm) is a binary format that allows high-performance code execution on the web. Wasm was added to major browsers over the course of 2017 which opened a wide range of opportunities on what to run inside of a browser. You can compile native libraries to WebAssembly and just run them on the client with just a few adjustments. WASM code can be shipped to browser apps and generally runs much faster compared to JavaScript, but still about [10% slower then native](https://www.usenix.org/conference/atc19/presentation/jangda). + +Many people started to use compiled SQLite as a database inside of the browser which is why it makes sense to also compare this setup to the native APIs. + +The compiled byte code of SQLite has a size of [about 938.9 kB](https://sqlite.org/download.html) which must be downloaded and parsed by the users on the first page load. WASM cannot directly access any persistent storage API in the browser. Instead it requires data to flow from WASM to the main-thread and then can be put into one of the browser APIs. This is done with so called [VFS (virtual file system) adapters](https://www.sqlite.org/vfs.html) that handle data access from SQLite to anything else. + +### What was WebSQL[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#what-was-websql "Direct link to What was WebSQL") + +WebSQL **was** a web API [introduced in 2009](https://www.w3.org/TR/webdatabase/) that allowed browsers to use SQL databases for client-side storage, based on SQLite. The idea was to give developers a way to store and query data using SQL on the client side, similar to server-side databases. WebSQL has been **removed from browsers** in the current years for multiple good reasons: + +* WebSQL was not standardized and having an API based on a single specific implementation in form of the SQLite source code is hard to ever make it to a standard. +* WebSQL required browsers to use a [specific version](https://developer.chrome.com/blog/deprecating-web-sql#reasons_for_deprecating_web_sql) of SQLite (version 3.6.19) which means whenever there would be any update or bugfix to SQLite, it would not be possible to add that to WebSQL without possible breaking the web. +* Major browsers like firefox never supported WebSQL. + +Therefore in the following we will **just ignore WebSQL** even if it would be possible to run tests on in by setting specific browser flags or using old versions of chromium. + +* * * + +Feature Comparison[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#feature-comparison "Direct link to Feature Comparison") +------------------------------------------------------------------------------------------------------------------------------------------------------------- + +Now that you know the basic concepts of the APIs, lets compare some specific features that have shown to be important for people using RxDB and browser based storages in general. + +### Storing complex JSON Documents[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#storing-complex-json-documents "Direct link to Storing complex JSON Documents") + +When you store data in a web application, most often you want to store complex JSON documents and not only "normal" values like the `integers` and `strings` you store in a server side database. + +* Only IndexedDB works with JSON objects natively. +* With SQLite WASM you can [store JSON](https://www.sqlite.org/json1.html) in a `text` column since version 3.38.0 (2022-02-22) and even run deep queries on it and use single attributes as indexes. + +Every of the other APIs can only store strings or binary data. Of course you can transform any JSON object to a string with `JSON.stringify()` but not having the JSON support in the API can make things complex when running queries and running `JSON.stringify()` many times can cause performance problems. + +### Multi-Tab Support[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#multi-tab-support "Direct link to Multi-Tab Support") + +A big difference when building a Web App compared to [Electron](https://rxdb.info/electron-database.html) or [React-Native](https://rxdb.info/react-native-database.html), is that the user will open and close the app in **multiple browser tabs at the same time**. Therefore you have not only one JavaScript process running, but many of them can exist and might have to share state changes between each other to not show **outdated data** to the user. + +> If your users' muscle memory puts the left hand on the **F5** key while using your website, you did something wrong! + +Not all storage APIs support a way to automatically share write events between tabs. + +Only localstorage has a way to automatically share write events between tabs by the API itself with the [storage-event](https://rxdb.info/articles/localstorage.html#localstorage-vs-indexeddb) which can be used to observe changes. + +``` +// localStorage can observe changes with the storage event.// This feature is missing in IndexedDB and othersaddEventListener("storage", (event) => {}); +``` + +There was the [experimental IndexedDB observers API](https://stackoverflow.com/a/33270440) for chrome, but the proposal repository has been archived. + +To workaround this problem, there are two solutions: + +* The first option is to use the [BroadcastChannel API](https://github.com/pubkey/broadcast-channel) which can send messages across browser tabs. So whenever you do a write to the storage, you also send a notification to other tabs to inform them about these changes. This is the most common workaround which is also used by RxDB. Notice that there is also the [WebLocks API](https://developer.mozilla.org/en-US/docs/Web/API/Web_Locks_API) which can be used to have mutexes across browser tabs. +* The other solution is to use the [SharedWorker](https://developer.mozilla.org/en-US/docs/Web/API/SharedWorker) and do all writes inside of the worker. All browser tabs can then subscribe to messages from that **single** SharedWorker and know about changes. + +### Indexing Support[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#indexing-support "Direct link to Indexing Support") + +The big difference between a database and storing data in a plain file, is that a database is writing data in a format that allows running operations over indexes to facilitate fast performant queries. From our list of technologies only **IndexedDB** and **WASM SQLite** support for indexing out of the box. In theory you can build indexes on top of any storage like localstorage or OPFS but you likely should not want to do that by yourself. + +In IndexedDB for example, we can fetch a bulk of documents by a given index range: + +``` +// find all products with a price between 10 and 50const keyRange = IDBKeyRange.bound(10, 50);const transaction = db.transaction('products', 'readonly');const objectStore = transaction.objectStore('products');const index = objectStore.index('priceIndex');const request = index.getAll(keyRange);const result = await new Promise((res, rej) => { request.onsuccess = (event) => res(event.target.result); request.onerror = (event) => rej(event);}); +``` + +Notice that IndexedDB has the limitation of [not having indexes on boolean values](https://github.com/w3c/IndexedDB/issues/76). You can only index strings and numbers. To workaround that you have to transform boolean to numbers and backwards when storing the data. + +### WebWorker Support[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#webworker-support "Direct link to WebWorker Support") + +When running heavy data operations, you might want to move the processing away from the JavaScript main thread. This ensures that our app keeps being responsive and fast while the processing can run in parallel in the background. In a browser you can either use the [WebWorker](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API), [SharedWorker](https://developer.mozilla.org/en-US/docs/Web/API/SharedWorker) or the [ServiceWorker](https://developer.mozilla.org/en-US/docs/Web/API/Service_Worker_API) API to do that. In RxDB you can use the [WebWorker](https://rxdb.info/rx-storage-worker.html) or [SharedWorker](https://rxdb.info/rx-storage-shared-worker.html) plugins to move your storage inside of a worker. + +The most common API for that use case is spawning a **WebWorker** and doing most work on that second JavaScript process. The worker is spawned from a separate JavaScript file (or base64 string) and communicates with the main thread by sending data with `postMessage()`. + +Unfortunately **LocalStorage** and **Cookies** [cannot be used in WebWorker or SharedWorker](https://stackoverflow.com/questions/6179159/accessing-localstorage-from-a-webworker) because of the design and security constraints. WebWorkers run in a separate global context from the main browser thread and therefore cannot do stuff that might impact the main thread. They have no direct access to certain web APIs, like the DOM, localStorage, or cookies. + +Everything else can be used from inside a WebWorker. The fast version of OPFS with the `createSyncAccessHandle` method can **only** [be used in a WebWorker](https://rxdb.info/rx-storage-opfs.html#opfs-limitations), and **not on the main thread**. This is because all the operations of the returned `AccessHandle` are **not async** and therefore block the JavaScript process, so you do want to do that on the main thread and block everything. + +* * * + +Storage Size Limits[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#storage-size-limits "Direct link to Storage Size Limits") +---------------------------------------------------------------------------------------------------------------------------------------------------------------- + +* **Cookies** are limited to about `4 KB` of data in [RFC-6265](https://datatracker.ietf.org/doc/html/rfc6265#section-6.1). Because the stored cookies are send to the server with every HTTP request, this limitation is reasonable. You can test your browsers cookie limits [here](http://www.ruslog.com/tools/cookies.html). Notice that you should never fill up the full `4 KB` of your cookies because your websserver will not accept too long headers and reject the requests with `HTTP ERROR 431 - Request header fields too large`. Once you have reached that point you can not even serve updated JavaScript to your user to clean up the cookies and you will have locked out that user until the cookies get cleaned up manually. + +* **LocalStorage** has a storage size limitation that varies depending on the browser, but generally ranges from 4 MB to 10 MB per origin. You can test your localStorage size limit [here](https://arty.name/localstorage.html). + + * Chrome/Chromium/Edge: 5 MB per domain + * Firefox: 10 MB per domain + * Safari: 4-5 MB per domain (varies slightly between versions) +* **IndexedDB** does not have a specific fixed size limitation like localStorage. The maximum storage size for IndexedDB depends on the browser implementation. The upper limit is typically based on the available disc space on the user's device. In chromium browsers it can use up to 80% of total disk space. You can get an estimation about the storage size limit by calling `await navigator.storage.estimate()`. Typically you can store gigabytes of data which can be tried out [here](https://demo.agektmr.com/storage/). Notice that we have a full article about [storage max size limits of IndexedDB](https://rxdb.info/articles/indexeddb-max-storage-limit.html) that covers this topic. + +* **OPFS** has the same storage size limitation as IndexedDB. Its limit depends on the available disc space. This can also be tested [here](https://demo.agektmr.com/storage/). + + +* * * + +Performance Comparison[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#performance-comparison "Direct link to Performance Comparison") +------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + +Now that we've reviewed the features of each storage method, let's dive into performance comparisons, focusing on initialization times, read/write latencies, and bulk operations. + +Notice that we only run simple tests and for your specific use case in your application the results might differ. Also we only compare performance in google chrome (version 128.0.6613.137). Firefox and Safari have similar **but not equal** performance patterns. You can run the test by yourself on your own machine from this [github repository](https://github.com/pubkey/localstorage-indexeddb-cookies-opfs-sqlite-wasm). For all tests we throttle the network to behave like the average german internet speed. (download: 135,900 kbit/s, upload: 28,400 kbit/s, latency: 125ms). Also all tests store an "average" JSON object that might be required to be stringified depending on the storage. We also only test the performance of storing documents by id because some of the technologies (cookies, OPFS and localstorage) do not support indexed range operations so it makes no sense to compare the performance of these. + +### Initialization Time[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#initialization-time "Direct link to Initialization Time") + +Before you can store any data, many APIs require a setup process like creating databases, spawning WebAssembly processes or downloading additional stuff. To ensure your app starts fast, the initialization time is important. + +The APIs of localStorage and Cookies do not have any setup process and can be directly used. IndexedDB requires to open a database and a store inside of it. WASM SQLite needs to download a WASM file and process it. OPFS needs to download and start a worker file and initialize the virtual file system directory. + +Here are the time measurements from how long it takes until the first bit of data can be stored: + +| Technology | Time in Milliseconds | +| --- | --- | +| IndexedDB | 46 | +| OPFS Main Thread | 23 | +| OPFS WebWorker | 26.8 | +| WASM SQLite (memory) | 504 | +| WASM SQLite (IndexedDB) | 535 | + +Here we can notice a few things: + +* Opening a new IndexedDB database with a single store takes surprisingly long +* The latency overhead of sending data from the main thread to a WebWorker OPFS is about 4 milliseconds. Here we only send minimal data to init the OPFS file handler. It will be interesting if that latency increases when more data is processed. +* Downloading and parsing WASM SQLite and creating a single table takes about half a second. Using also the IndexedDB VFS to store data persistently adds additional 31 milliseconds. Reloading the page with enabled caching and already prepared tables is a bit faster with 420 milliseconds (memory). + +### Latency of small Writes[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#latency-of-small-writes "Direct link to Latency of small Writes") + +Next lets test the latency of small writes. This is important when you do many small data changes that happen independent from each other. Like when you stream data from a websocket or persist pseudo randomly happening events like mouse movements. + +| Technology | Time in Milliseconds | +| --- | --- | +| Cookies | 0.058 | +| LocalStorage | 0.017 | +| IndexedDB | 0.17 | +| OPFS Main Thread | 1.46 | +| OPFS WebWorker | 1.54 | +| WASM SQLite (memory) | 0.17 | +| WASM SQLite (IndexedDB) | 3.17 | + +Here we can notice a few things: + +* LocalStorage has the lowest write latency with only 0.017 milliseconds per write. +* IndexedDB writes are about 10 times slower compared to localStorage. +* Sending the data to the WASM SQLite process and letting it persist via IndexedDB is slow with over 3 milliseconds per write. + +The OPFS operations take about 1.5 milliseconds to write the JSON data into one document per file. We can see the sending the data to a webworker first is a bit slower which comes from the overhead of serializing and deserializing the data on both sides. If we would not create on OPFS file per document but instead append everything to a single file, the performance pattern changes significantly. Then the faster file handle from the `createSyncAccessHandle()` only takes about 1 millisecond per write. But this would require to somehow remember at which position the each document is stored. Therefore in our tests we will continue using one file per document. + +### Latency of small Reads[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#latency-of-small-reads "Direct link to Latency of small Reads") + +Now that we have stored some documents, lets measure how long it takes to read single documents by their `id`. + +| Technology | Time in Milliseconds | +| --- | --- | +| Cookies | 0.132 | +| LocalStorage | 0.0052 | +| IndexedDB | 0.1 | +| OPFS Main Thread | 1.28 | +| OPFS WebWorker | 1.41 | +| WASM SQLite (memory) | 0.45 | +| WASM SQLite (IndexedDB) | 2.93 | + +Here we can notice a few things: + +* LocalStorage reads are **really really fast** with only 0.0052 milliseconds per read. +* The other technologies perform reads in a similar speed to their write latency. + +### Big Bulk Writes[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#big-bulk-writes "Direct link to Big Bulk Writes") + +As next step, lets do some big bulk operations with 200 documents at once. + +| Technology | Time in Milliseconds | +| --- | --- | +| Cookies | 20.6 | +| LocalStorage | 5.79 | +| IndexedDB | 13.41 | +| OPFS Main Thread | 280 | +| OPFS WebWorker | 104 | +| WASM SQLite (memory) | 19.1 | +| WASM SQLite (IndexedDB) | 37.12 | + +Here we can notice a few things: + +* Sending the data to a WebWorker and running it via the faster OPFS API is about twice as fast. +* WASM SQLite performs better on bulk operations compared to its single write latency. This is because sending the data to WASM and backwards is faster if it is done all at once instead of once per document. + +### Big Bulk Reads[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#big-bulk-reads "Direct link to Big Bulk Reads") + +Now lets read 100 documents in a bulk request. + +| Technology | Time in Milliseconds | +| --- | --- | +| Cookies | 6.34 | +| LocalStorage | 0.39 | +| IndexedDB | 4.99 | +| OPFS Main Thread | 54.79 | +| OPFS WebWorker | 25.61 | +| WASM SQLite (memory) | 3.59 | +| WASM SQLite (IndexedDB) | 5.84 (35ms without cache) | + +Here we can notice a few things: + +* Reading many files in the OPFS webworker is about **twice as fast** compared to the slower main thread mode. +* WASM SQLite is surprisingly fast. Further inspection has shown that the WASM SQLite process keeps the documents in memory cached which improves the latency when we do reads directly after writes on the same data. When the browser tab is reloaded between the writes and the reads, finding the 100 documents takes about **35 milliseconds** instead. + +Performance Conclusions[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#performance-conclusions "Direct link to Performance Conclusions") +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + +* LocalStorage is really fast but remember that is has some downsides: + * It blocks the main JavaScript process and therefore should not be used for big bulk operations. + * Only Key-Value assignments are possible, you cannot use it efficiently when you need to do index based range queries on your data. +* OPFS is way faster when used in the WebWorker with the `createSyncAccessHandle()` method compare to using it directly in the main thread. +* SQLite WASM can be fast but the you have to initially download the full binary and start it up which takes about half a second. This might not be relevant at all if your app is started up once and the used for a very long time. But for web-apps that are opened and closed in many browser tabs many times, this might be a problem. + +* * * + +Possible Improvements[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#possible-improvements "Direct link to Possible Improvements") +---------------------------------------------------------------------------------------------------------------------------------------------------------------------- + +There is a wide range of possible improvements and performance hacks to speed up the operations. + +* For IndexedDB I have made a list of [performance hacks here](https://rxdb.info/slow-indexeddb.html). For example you can do sharding between multiple database and webworkers or use a custom index strategy. +* OPFS is slow in writing one file per document. But you do not have to do that and instead you can store everything at a single file like a normal database would do. This improves performance dramatically like it was done with the RxDB [OPFS RxStorage](https://rxdb.info/rx-storage-opfs.html). +* You can mix up the technologies to optimize for multiple scenarios at once. For example in RxDB there is the [localstorage meta optimizer](https://rxdb.info/rx-storage-localstorage-meta-optimizer.html) which stores initial metadata in localstorage and "normal" documents inside of IndexedDB. This improves the initial startup time while still having the documents stored in a way to query them efficiently. +* There is the [memory-mapped](https://rxdb.info/rx-storage-memory-mapped.html) storage plugin in RxDB which maps data directly to memory. Using this in combination with a shared worker can improve pageloads and query time significantly. +* [Compressing](https://rxdb.info/key-compression.html) data before storing it might improve the performance for some of the storages. +* Splitting work up between [multiple WebWorkers](https://rxdb.info/rx-storage-worker.html) via [sharding](https://rxdb.info/rx-storage-sharding.html) can improve performance by utilizing the whole capacity of your users device. + +Here you can see the [performance comparison](https://rxdb.info/rx-storage-performance.html) of various RxDB storage implementations which gives a better view of real world performance: + +![Image 3: RxStorage performance - browser](https://rxdb.info/files/rx-storage-performance-browser.png) + +Future Improvements[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#future-improvements "Direct link to Future Improvements") +---------------------------------------------------------------------------------------------------------------------------------------------------------------- + +You are reading this in 2024, but the web does not stand still. There is a good chance that browser get enhanced to allow faster and better data operations. + +* Currently there is no way to directly access a persistent storage from inside a WebAssembly process. If this changes in the future, running SQLite (or a similar database) in a browser might be the best option. +* Sending data between the main thread and a WebWorker is slow but might be improved in the future. There is a [good article](https://surma.dev/things/is-postmessage-slow/) about why `postMessage()` is slow. +* IndexedDB lately [got support](https://developer.chrome.com/blog/maximum-idb-performance-with-storage-buckets) for storage buckets (chrome only) which might improve performance. + +Follow Up[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#follow-up "Direct link to Follow Up") +---------------------------------------------------------------------------------------------------------------------------------- + +* Share my [announcement tweet](https://x.com/rxdbjs/status/1846145062847062391) --\> +* Reproduce the benchmarks at the [github repo](https://github.com/pubkey/localstorage-indexeddb-cookies-opfs-sqlite-wasm) +* Learn how to use RxDB with the [RxDB Quickstart](https://rxdb.info/quickstart.html) +* Check out the [RxDB github repo](https://github.com/pubkey/rxdb) and leave a star ⭐ \ No newline at end of file From 4df8d03b5c51cd8875bc9ae527f2f05160d0563c Mon Sep 17 00:00:00 2001 From: luojiyin Date: Mon, 3 Mar 2025 02:48:59 +0000 Subject: [PATCH 2/5] url clear --- ...indexeddb-cookies-opfs-sqlite-wasm.html.md | 154 ++++++++++++------ 1 file changed, 106 insertions(+), 48 deletions(-) diff --git a/_drafts/Article/Translation/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html.md b/_drafts/Article/Translation/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html.md index 7df0346c..decc077d 100644 --- a/_drafts/Article/Translation/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html.md +++ b/_drafts/Article/Translation/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html.md @@ -9,7 +9,7 @@ reviewer: "" So you are building that web application and you want to **store data inside of your users browser**. Maybe you just need to store some small flags or you even need a fully fledged database. -The types of web applications we build have changed significantly. In the early years of the web we served static html files. Then we served dynamically rendered html and later we build **single page applications** that run most logic on the client. And for the coming years you might want to build so called [local first apps](https://rxdb.info/offline-first.html) that handle big and complex data operations solely on the client and even work when offline, which gives you the opportunity to build **zero-latency** user interactions. +The types of web applications we build have changed significantly. In the early years of the web we served static html files. Then we served dynamically rendered html and later we build **single page applications** that run most logic on the client. And for the coming years you might want to build so called [local first apps][4] that handle big and complex data operations solely on the client and even work when offline, which gives you the opportunity to build **zero-latency** user interactions. In the early days of the web, **cookies** were the only option for storing small key-value assignments.. But JavaScript and browsers have evolved significantly and better storage APIs have been added which pave the way for bigger and more complex data operations. @@ -17,9 +17,9 @@ In this article, we will dive into the various technologies available for storin note -You are reading this in the [RxDB](https://rxdb.info/) docs. RxDB is a JavaScript database that has different storage adapters which can utilize the different storage APIs. **Since 2017** I spend most of my time working with these APIs, doing performance tests and building [hacks](https://rxdb.info/slow-indexeddb.html) and plugins to reach the limits of browser database operation speed. +You are reading this in the [RxDB][5] docs. RxDB is a JavaScript database that has different storage adapters which can utilize the different storage APIs. **Since 2017** I spend most of my time working with these APIs, doing performance tests and building [hacks][6] and plugins to reach the limits of browser database operation speed. -[![Image 1: JavaScript Database](https://rxdb.info/files/logo/rxdb_javascript_database.svg)](https://rxdb.info/) +[![Image 1: JavaScript Database][1]][5] The available Storage APIs in a modern Browser[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#the-available-storage-apis-in-a-modern-browser "Direct link to The available Storage APIs in a modern Browser") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- @@ -28,53 +28,53 @@ First lets have a brief overview of the different APIs, their intentional use ca ### What are Cookies[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#what-are-cookies "Direct link to What are Cookies") -Cookies were first introduced by [netscape in 1994](https://www.baekdal.com/thoughts/the-original-cookie-specification-from-1997-was-gdpr-compliant/). Cookies store small pieces of key-value data that are mainly used for session management, personalization, and tracking. Cookies can have several security settings like a time-to-live or the `domain` attribute to share the cookies between several subdomains. +Cookies were first introduced by [netscape in 1994][7]. Cookies store small pieces of key-value data that are mainly used for session management, personalization, and tracking. Cookies can have several security settings like a time-to-live or the `domain` attribute to share the cookies between several subdomains. -Cookies values are not only stored at the client but also sent with **every http request** to the server. This means we cannot store much data in a cookie but it is still interesting how good cookie access performance compared to the other methods. Especially because cookies are such an important base feature of the web, many performance optimizations have been done and even these days there is still progress being made like the [Shared Memory Versioning](https://blog.chromium.org/2024/06/introducing-shared-memory-versioning-to.html) by chromium or the asynchronous [CookieStore API](https://developer.mozilla.org/en-US/docs/Web/API/Cookie_Store_API). +Cookies values are not only stored at the client but also sent with **every http request** to the server. This means we cannot store much data in a cookie but it is still interesting how good cookie access performance compared to the other methods. Especially because cookies are such an important base feature of the web, many performance optimizations have been done and even these days there is still progress being made like the [Shared Memory Versioning][8] by chromium or the asynchronous [CookieStore API][9]. ### What is LocalStorage[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#what-is-localstorage "Direct link to What is LocalStorage") -The [localStorage API](https://rxdb.info/articles/localstorage.html) was first proposed as part of the [WebStorage specification in 2009](https://www.w3.org/TR/2009/WD-webstorage-20090423/#the-localstorage-attribute). LocalStorage provides a simple API to store key-value pairs inside of a web browser. It has the methods `setItem`, `getItem`, `removeItem` and `clear` which is all you need from a key-value store. LocalStorage is only suitable for storing small amounts of data that need to persist across sessions and it is [limited by a 5MB storage cap](https://rxdb.info/articles/localstorage.html#understanding-the-limitations-of-local-storage). Storing complex data is only possible by transforming it into a string for example with `JSON.stringify()`. The API is not asynchronous which means if fully blocks your JavaScript process while doing stuff. Therefore running heavy operations on it might block your UI from rendering. +The [localStorage API][10] was first proposed as part of the [WebStorage specification in 2009][11]. LocalStorage provides a simple API to store key-value pairs inside of a web browser. It has the methods `setItem`, `getItem`, `removeItem` and `clear` which is all you need from a key-value store. LocalStorage is only suitable for storing small amounts of data that need to persist across sessions and it is [limited by a 5MB storage cap][12]. Storing complex data is only possible by transforming it into a string for example with `JSON.stringify()`. The API is not asynchronous which means if fully blocks your JavaScript process while doing stuff. Therefore running heavy operations on it might block your UI from rendering. > There is also the **SessionStorage** API. The key difference is that localStorage data persists indefinitely until explicitly cleared, while sessionStorage data is cleared when the browser tab or window is closed. ### What is IndexedDB[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#what-is-indexeddb "Direct link to What is IndexedDB") -IndexedDB was first introduced as "Indexed Database API" [in 2015](https://www.w3.org/TR/IndexedDB/#sotd). +IndexedDB was first introduced as "Indexed Database API" [in 2015][13]. -[IndexedDB](https://rxdb.info/rx-storage-indexeddb.html) is a low-level API for storing large amounts of structured JSON data. While the API is a bit hard to use, IndexedDB can utilize indexes and asynchronous operations. It lacks support for complex queries and only allows to iterate over the indexes which makes it more like a base layer for other libraries then a fully fledged database. +[IndexedDB][14] is a low-level API for storing large amounts of structured JSON data. While the API is a bit hard to use, IndexedDB can utilize indexes and asynchronous operations. It lacks support for complex queries and only allows to iterate over the indexes which makes it more like a base layer for other libraries then a fully fledged database. -In 2018, IndexedDB version 2.0 [was introduced](https://hacks.mozilla.org/2016/10/whats-new-in-indexeddb-2-0/). This added some major improvements. Most noticeable the `getAll()` method which improves performance dramatically when fetching bulks of JSON documents. +In 2018, IndexedDB version 2.0 [was introduced][15]. This added some major improvements. Most noticeable the `getAll()` method which improves performance dramatically when fetching bulks of JSON documents. -IndexedDB [version 3.0](https://w3c.github.io/IndexedDB/) is in the workings which contains many improvements. Most important the addition of `Promise` based calls that makes modern JS features like `async/await` more useful. +IndexedDB [version 3.0][16] is in the workings which contains many improvements. Most important the addition of `Promise` based calls that makes modern JS features like `async/await` more useful. ### What is OPFS[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#what-is-opfs "Direct link to What is OPFS") -The [Origin Private File System](https://rxdb.info/rx-storage-opfs.html) (OPFS) is a [relatively new](https://caniuse.com/mdn-api_filesystemfilehandle_createsyncaccesshandle) API that allows web applications to store large files directly in the browser. It is designed for data-intensive applications that want to write and read **binary data** in a simulated file system. +The [Origin Private File System][17] (OPFS) is a [relatively new][18] API that allows web applications to store large files directly in the browser. It is designed for data-intensive applications that want to write and read **binary data** in a simulated file system. OPFS can be used in two modes: -* Either asynchronous on the [main thread](https://rxdb.info/rx-storage-opfs.html#using-opfs-in-the-main-thread-instead-of-a-worker) +* Either asynchronous on the [main thread][19] * Or in a WebWorker with the faster, asynchronous access with the `createSyncAccessHandle()` method. -Because only binary data can be processed, OPFS is made to be a base filesystem for library developers. You will unlikely directly want to use the OPFS in your code when you build a "normal" application because it is too complex. That would only make sense for storing plain files like images, not to store and query [JSON data](https://rxdb.info/articles/json-based-database.html) efficiently. I have build a [OPFS based storage](https://rxdb.info/rx-storage-opfs.html) for RxDB with proper indexing and querying and it took me several months. +Because only binary data can be processed, OPFS is made to be a base filesystem for library developers. You will unlikely directly want to use the OPFS in your code when you build a "normal" application because it is too complex. That would only make sense for storing plain files like images, not to store and query [JSON data][20] efficiently. I have build a [OPFS based storage][17] for RxDB with proper indexing and querying and it took me several months. ### What is WASM SQLite[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#what-is-wasm-sqlite "Direct link to What is WASM SQLite") -![Image 2: WASM SQLite](https://rxdb.info/files/icons/sqlite.svg) +![Image 2: WASM SQLite][2] -[WebAssembly](https://webassembly.org/) (Wasm) is a binary format that allows high-performance code execution on the web. Wasm was added to major browsers over the course of 2017 which opened a wide range of opportunities on what to run inside of a browser. You can compile native libraries to WebAssembly and just run them on the client with just a few adjustments. WASM code can be shipped to browser apps and generally runs much faster compared to JavaScript, but still about [10% slower then native](https://www.usenix.org/conference/atc19/presentation/jangda). +[WebAssembly][21] (Wasm) is a binary format that allows high-performance code execution on the web. Wasm was added to major browsers over the course of 2017 which opened a wide range of opportunities on what to run inside of a browser. You can compile native libraries to WebAssembly and just run them on the client with just a few adjustments. WASM code can be shipped to browser apps and generally runs much faster compared to JavaScript, but still about [10% slower then native][22]. Many people started to use compiled SQLite as a database inside of the browser which is why it makes sense to also compare this setup to the native APIs. -The compiled byte code of SQLite has a size of [about 938.9 kB](https://sqlite.org/download.html) which must be downloaded and parsed by the users on the first page load. WASM cannot directly access any persistent storage API in the browser. Instead it requires data to flow from WASM to the main-thread and then can be put into one of the browser APIs. This is done with so called [VFS (virtual file system) adapters](https://www.sqlite.org/vfs.html) that handle data access from SQLite to anything else. +The compiled byte code of SQLite has a size of [about 938.9 kB][23] which must be downloaded and parsed by the users on the first page load. WASM cannot directly access any persistent storage API in the browser. Instead it requires data to flow from WASM to the main-thread and then can be put into one of the browser APIs. This is done with so called [VFS (virtual file system) adapters][24] that handle data access from SQLite to anything else. ### What was WebSQL[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#what-was-websql "Direct link to What was WebSQL") -WebSQL **was** a web API [introduced in 2009](https://www.w3.org/TR/webdatabase/) that allowed browsers to use SQL databases for client-side storage, based on SQLite. The idea was to give developers a way to store and query data using SQL on the client side, similar to server-side databases. WebSQL has been **removed from browsers** in the current years for multiple good reasons: +WebSQL **was** a web API [introduced in 2009][25] that allowed browsers to use SQL databases for client-side storage, based on SQLite. The idea was to give developers a way to store and query data using SQL on the client side, similar to server-side databases. WebSQL has been **removed from browsers** in the current years for multiple good reasons: * WebSQL was not standardized and having an API based on a single specific implementation in form of the SQLite source code is hard to ever make it to a standard. -* WebSQL required browsers to use a [specific version](https://developer.chrome.com/blog/deprecating-web-sql#reasons_for_deprecating_web_sql) of SQLite (version 3.6.19) which means whenever there would be any update or bugfix to SQLite, it would not be possible to add that to WebSQL without possible breaking the web. +* WebSQL required browsers to use a [specific version][26] of SQLite (version 3.6.19) which means whenever there would be any update or bugfix to SQLite, it would not be possible to add that to WebSQL without possible breaking the web. * Major browsers like firefox never supported WebSQL. Therefore in the following we will **just ignore WebSQL** even if it would be possible to run tests on in by setting specific browser flags or using old versions of chromium. @@ -91,30 +91,30 @@ Now that you know the basic concepts of the APIs, lets compare some specific fea When you store data in a web application, most often you want to store complex JSON documents and not only "normal" values like the `integers` and `strings` you store in a server side database. * Only IndexedDB works with JSON objects natively. -* With SQLite WASM you can [store JSON](https://www.sqlite.org/json1.html) in a `text` column since version 3.38.0 (2022-02-22) and even run deep queries on it and use single attributes as indexes. +* With SQLite WASM you can [store JSON][27] in a `text` column since version 3.38.0 (2022-02-22) and even run deep queries on it and use single attributes as indexes. Every of the other APIs can only store strings or binary data. Of course you can transform any JSON object to a string with `JSON.stringify()` but not having the JSON support in the API can make things complex when running queries and running `JSON.stringify()` many times can cause performance problems. ### Multi-Tab Support[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#multi-tab-support "Direct link to Multi-Tab Support") -A big difference when building a Web App compared to [Electron](https://rxdb.info/electron-database.html) or [React-Native](https://rxdb.info/react-native-database.html), is that the user will open and close the app in **multiple browser tabs at the same time**. Therefore you have not only one JavaScript process running, but many of them can exist and might have to share state changes between each other to not show **outdated data** to the user. +A big difference when building a Web App compared to [Electron][28] or [React-Native][29], is that the user will open and close the app in **multiple browser tabs at the same time**. Therefore you have not only one JavaScript process running, but many of them can exist and might have to share state changes between each other to not show **outdated data** to the user. > If your users' muscle memory puts the left hand on the **F5** key while using your website, you did something wrong! Not all storage APIs support a way to automatically share write events between tabs. -Only localstorage has a way to automatically share write events between tabs by the API itself with the [storage-event](https://rxdb.info/articles/localstorage.html#localstorage-vs-indexeddb) which can be used to observe changes. +Only localstorage has a way to automatically share write events between tabs by the API itself with the [storage-event][30] which can be used to observe changes. ``` // localStorage can observe changes with the storage event.// This feature is missing in IndexedDB and othersaddEventListener("storage", (event) => {}); ``` -There was the [experimental IndexedDB observers API](https://stackoverflow.com/a/33270440) for chrome, but the proposal repository has been archived. +There was the [experimental IndexedDB observers API][31] for chrome, but the proposal repository has been archived. To workaround this problem, there are two solutions: -* The first option is to use the [BroadcastChannel API](https://github.com/pubkey/broadcast-channel) which can send messages across browser tabs. So whenever you do a write to the storage, you also send a notification to other tabs to inform them about these changes. This is the most common workaround which is also used by RxDB. Notice that there is also the [WebLocks API](https://developer.mozilla.org/en-US/docs/Web/API/Web_Locks_API) which can be used to have mutexes across browser tabs. -* The other solution is to use the [SharedWorker](https://developer.mozilla.org/en-US/docs/Web/API/SharedWorker) and do all writes inside of the worker. All browser tabs can then subscribe to messages from that **single** SharedWorker and know about changes. +* The first option is to use the [BroadcastChannel API][32] which can send messages across browser tabs. So whenever you do a write to the storage, you also send a notification to other tabs to inform them about these changes. This is the most common workaround which is also used by RxDB. Notice that there is also the [WebLocks API][33] which can be used to have mutexes across browser tabs. +* The other solution is to use the [SharedWorker][34] and do all writes inside of the worker. All browser tabs can then subscribe to messages from that **single** SharedWorker and know about changes. ### Indexing Support[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#indexing-support "Direct link to Indexing Support") @@ -126,33 +126,33 @@ In IndexedDB for example, we can fetch a bulk of documents by a given index rang // find all products with a price between 10 and 50const keyRange = IDBKeyRange.bound(10, 50);const transaction = db.transaction('products', 'readonly');const objectStore = transaction.objectStore('products');const index = objectStore.index('priceIndex');const request = index.getAll(keyRange);const result = await new Promise((res, rej) => { request.onsuccess = (event) => res(event.target.result); request.onerror = (event) => rej(event);}); ``` -Notice that IndexedDB has the limitation of [not having indexes on boolean values](https://github.com/w3c/IndexedDB/issues/76). You can only index strings and numbers. To workaround that you have to transform boolean to numbers and backwards when storing the data. +Notice that IndexedDB has the limitation of [not having indexes on boolean values][35]. You can only index strings and numbers. To workaround that you have to transform boolean to numbers and backwards when storing the data. ### WebWorker Support[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#webworker-support "Direct link to WebWorker Support") -When running heavy data operations, you might want to move the processing away from the JavaScript main thread. This ensures that our app keeps being responsive and fast while the processing can run in parallel in the background. In a browser you can either use the [WebWorker](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API), [SharedWorker](https://developer.mozilla.org/en-US/docs/Web/API/SharedWorker) or the [ServiceWorker](https://developer.mozilla.org/en-US/docs/Web/API/Service_Worker_API) API to do that. In RxDB you can use the [WebWorker](https://rxdb.info/rx-storage-worker.html) or [SharedWorker](https://rxdb.info/rx-storage-shared-worker.html) plugins to move your storage inside of a worker. +When running heavy data operations, you might want to move the processing away from the JavaScript main thread. This ensures that our app keeps being responsive and fast while the processing can run in parallel in the background. In a browser you can either use the [WebWorker][36], [SharedWorker][34] or the [ServiceWorker][37] API to do that. In RxDB you can use the [WebWorker][38] or [SharedWorker][39] plugins to move your storage inside of a worker. The most common API for that use case is spawning a **WebWorker** and doing most work on that second JavaScript process. The worker is spawned from a separate JavaScript file (or base64 string) and communicates with the main thread by sending data with `postMessage()`. -Unfortunately **LocalStorage** and **Cookies** [cannot be used in WebWorker or SharedWorker](https://stackoverflow.com/questions/6179159/accessing-localstorage-from-a-webworker) because of the design and security constraints. WebWorkers run in a separate global context from the main browser thread and therefore cannot do stuff that might impact the main thread. They have no direct access to certain web APIs, like the DOM, localStorage, or cookies. +Unfortunately **LocalStorage** and **Cookies** [cannot be used in WebWorker or SharedWorker][40] because of the design and security constraints. WebWorkers run in a separate global context from the main browser thread and therefore cannot do stuff that might impact the main thread. They have no direct access to certain web APIs, like the DOM, localStorage, or cookies. -Everything else can be used from inside a WebWorker. The fast version of OPFS with the `createSyncAccessHandle` method can **only** [be used in a WebWorker](https://rxdb.info/rx-storage-opfs.html#opfs-limitations), and **not on the main thread**. This is because all the operations of the returned `AccessHandle` are **not async** and therefore block the JavaScript process, so you do want to do that on the main thread and block everything. +Everything else can be used from inside a WebWorker. The fast version of OPFS with the `createSyncAccessHandle` method can **only** [be used in a WebWorker][41], and **not on the main thread**. This is because all the operations of the returned `AccessHandle` are **not async** and therefore block the JavaScript process, so you do want to do that on the main thread and block everything. * * * Storage Size Limits[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#storage-size-limits "Direct link to Storage Size Limits") ---------------------------------------------------------------------------------------------------------------------------------------------------------------- -* **Cookies** are limited to about `4 KB` of data in [RFC-6265](https://datatracker.ietf.org/doc/html/rfc6265#section-6.1). Because the stored cookies are send to the server with every HTTP request, this limitation is reasonable. You can test your browsers cookie limits [here](http://www.ruslog.com/tools/cookies.html). Notice that you should never fill up the full `4 KB` of your cookies because your websserver will not accept too long headers and reject the requests with `HTTP ERROR 431 - Request header fields too large`. Once you have reached that point you can not even serve updated JavaScript to your user to clean up the cookies and you will have locked out that user until the cookies get cleaned up manually. +* **Cookies** are limited to about `4 KB` of data in [RFC-6265][42]. Because the stored cookies are send to the server with every HTTP request, this limitation is reasonable. You can test your browsers cookie limits [here][43]. Notice that you should never fill up the full `4 KB` of your cookies because your websserver will not accept too long headers and reject the requests with `HTTP ERROR 431 - Request header fields too large`. Once you have reached that point you can not even serve updated JavaScript to your user to clean up the cookies and you will have locked out that user until the cookies get cleaned up manually. -* **LocalStorage** has a storage size limitation that varies depending on the browser, but generally ranges from 4 MB to 10 MB per origin. You can test your localStorage size limit [here](https://arty.name/localstorage.html). +* **LocalStorage** has a storage size limitation that varies depending on the browser, but generally ranges from 4 MB to 10 MB per origin. You can test your localStorage size limit [here][44]. * Chrome/Chromium/Edge: 5 MB per domain * Firefox: 10 MB per domain * Safari: 4-5 MB per domain (varies slightly between versions) -* **IndexedDB** does not have a specific fixed size limitation like localStorage. The maximum storage size for IndexedDB depends on the browser implementation. The upper limit is typically based on the available disc space on the user's device. In chromium browsers it can use up to 80% of total disk space. You can get an estimation about the storage size limit by calling `await navigator.storage.estimate()`. Typically you can store gigabytes of data which can be tried out [here](https://demo.agektmr.com/storage/). Notice that we have a full article about [storage max size limits of IndexedDB](https://rxdb.info/articles/indexeddb-max-storage-limit.html) that covers this topic. +* **IndexedDB** does not have a specific fixed size limitation like localStorage. The maximum storage size for IndexedDB depends on the browser implementation. The upper limit is typically based on the available disc space on the user's device. In chromium browsers it can use up to 80% of total disk space. You can get an estimation about the storage size limit by calling `await navigator.storage.estimate()`. Typically you can store gigabytes of data which can be tried out [here][45]. Notice that we have a full article about [storage max size limits of IndexedDB][46] that covers this topic. -* **OPFS** has the same storage size limitation as IndexedDB. Its limit depends on the available disc space. This can also be tested [here](https://demo.agektmr.com/storage/). +* **OPFS** has the same storage size limitation as IndexedDB. Its limit depends on the available disc space. This can also be tested [here][45]. * * * @@ -162,7 +162,7 @@ Performance Comparison[​](https://rxdb.info/articles/localstorage-indexeddb-co Now that we've reviewed the features of each storage method, let's dive into performance comparisons, focusing on initialization times, read/write latencies, and bulk operations. -Notice that we only run simple tests and for your specific use case in your application the results might differ. Also we only compare performance in google chrome (version 128.0.6613.137). Firefox and Safari have similar **but not equal** performance patterns. You can run the test by yourself on your own machine from this [github repository](https://github.com/pubkey/localstorage-indexeddb-cookies-opfs-sqlite-wasm). For all tests we throttle the network to behave like the average german internet speed. (download: 135,900 kbit/s, upload: 28,400 kbit/s, latency: 125ms). Also all tests store an "average" JSON object that might be required to be stringified depending on the storage. We also only test the performance of storing documents by id because some of the technologies (cookies, OPFS and localstorage) do not support indexed range operations so it makes no sense to compare the performance of these. +Notice that we only run simple tests and for your specific use case in your application the results might differ. Also we only compare performance in google chrome (version 128.0.6613.137). Firefox and Safari have similar **but not equal** performance patterns. You can run the test by yourself on your own machine from this [github repository][47]. For all tests we throttle the network to behave like the average german internet speed. (download: 135,900 kbit/s, upload: 28,400 kbit/s, latency: 125ms). Also all tests store an "average" JSON object that might be required to be stringified depending on the storage. We also only test the performance of storing documents by id because some of the technologies (cookies, OPFS and localstorage) do not support indexed range operations so it makes no sense to compare the performance of these. ### Initialization Time[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#initialization-time "Direct link to Initialization Time") @@ -281,16 +281,16 @@ Possible Improvements[​](https://rxdb.info/articles/localstorage-indexeddb-coo There is a wide range of possible improvements and performance hacks to speed up the operations. -* For IndexedDB I have made a list of [performance hacks here](https://rxdb.info/slow-indexeddb.html). For example you can do sharding between multiple database and webworkers or use a custom index strategy. -* OPFS is slow in writing one file per document. But you do not have to do that and instead you can store everything at a single file like a normal database would do. This improves performance dramatically like it was done with the RxDB [OPFS RxStorage](https://rxdb.info/rx-storage-opfs.html). -* You can mix up the technologies to optimize for multiple scenarios at once. For example in RxDB there is the [localstorage meta optimizer](https://rxdb.info/rx-storage-localstorage-meta-optimizer.html) which stores initial metadata in localstorage and "normal" documents inside of IndexedDB. This improves the initial startup time while still having the documents stored in a way to query them efficiently. -* There is the [memory-mapped](https://rxdb.info/rx-storage-memory-mapped.html) storage plugin in RxDB which maps data directly to memory. Using this in combination with a shared worker can improve pageloads and query time significantly. -* [Compressing](https://rxdb.info/key-compression.html) data before storing it might improve the performance for some of the storages. -* Splitting work up between [multiple WebWorkers](https://rxdb.info/rx-storage-worker.html) via [sharding](https://rxdb.info/rx-storage-sharding.html) can improve performance by utilizing the whole capacity of your users device. +* For IndexedDB I have made a list of [performance hacks here][6]. For example you can do sharding between multiple database and webworkers or use a custom index strategy. +* OPFS is slow in writing one file per document. But you do not have to do that and instead you can store everything at a single file like a normal database would do. This improves performance dramatically like it was done with the RxDB [OPFS RxStorage][17]. +* You can mix up the technologies to optimize for multiple scenarios at once. For example in RxDB there is the [localstorage meta optimizer][48] which stores initial metadata in localstorage and "normal" documents inside of IndexedDB. This improves the initial startup time while still having the documents stored in a way to query them efficiently. +* There is the [memory-mapped][49] storage plugin in RxDB which maps data directly to memory. Using this in combination with a shared worker can improve pageloads and query time significantly. +* [Compressing][50] data before storing it might improve the performance for some of the storages. +* Splitting work up between [multiple WebWorkers][38] via [sharding][51] can improve performance by utilizing the whole capacity of your users device. -Here you can see the [performance comparison](https://rxdb.info/rx-storage-performance.html) of various RxDB storage implementations which gives a better view of real world performance: +Here you can see the [performance comparison][52] of various RxDB storage implementations which gives a better view of real world performance: -![Image 3: RxStorage performance - browser](https://rxdb.info/files/rx-storage-performance-browser.png) +![Image 3: RxStorage performance - browser][3] Future Improvements[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#future-improvements "Direct link to Future Improvements") ---------------------------------------------------------------------------------------------------------------------------------------------------------------- @@ -298,13 +298,71 @@ Future Improvements[​](https://rxdb.info/articles/localstorage-indexeddb-cooki You are reading this in 2024, but the web does not stand still. There is a good chance that browser get enhanced to allow faster and better data operations. * Currently there is no way to directly access a persistent storage from inside a WebAssembly process. If this changes in the future, running SQLite (or a similar database) in a browser might be the best option. -* Sending data between the main thread and a WebWorker is slow but might be improved in the future. There is a [good article](https://surma.dev/things/is-postmessage-slow/) about why `postMessage()` is slow. -* IndexedDB lately [got support](https://developer.chrome.com/blog/maximum-idb-performance-with-storage-buckets) for storage buckets (chrome only) which might improve performance. +* Sending data between the main thread and a WebWorker is slow but might be improved in the future. There is a [good article][53] about why `postMessage()` is slow. +* IndexedDB lately [got support][54] for storage buckets (chrome only) which might improve performance. -Follow Up[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#follow-up "Direct link to Follow Up") +Follow Up ---------------------------------------------------------------------------------------------------------------------------------- -* Share my [announcement tweet](https://x.com/rxdbjs/status/1846145062847062391) --\> -* Reproduce the benchmarks at the [github repo](https://github.com/pubkey/localstorage-indexeddb-cookies-opfs-sqlite-wasm) -* Learn how to use RxDB with the [RxDB Quickstart](https://rxdb.info/quickstart.html) -* Check out the [RxDB github repo](https://github.com/pubkey/rxdb) and leave a star ⭐ \ No newline at end of file +* Share my [announcement tweet][55] --\> +* Reproduce the benchmarks at the [github repo][47] +* Learn how to use RxDB with the [RxDB Quickstart][56] +* Check out the [RxDB github repo][57] and leave a star ⭐ + +[1]: https://rxdb.info/files/logo/rxdb_javascript_database.svg +[2]: https://rxdb.info/files/icons/sqlite.svg +[3]: https://rxdb.info/files/rx-storage-performance-browser.png +[4]: https://rxdb.info/offline-first.html +[5]: https://rxdb.info/ +[6]: https://rxdb.info/slow-indexeddb.html +[7]: https://www.baekdal.com/thoughts/the-original-cookie-specification-from-1997-was-gdpr-compliant/ +[8]: https://blog.chromium.org/2024/06/introducing-shared-memory-versioning-to.html +[9]: https://developer.mozilla.org/en-US/docs/Web/API/Cookie_Store_API +[10]: https://rxdb.info/articles/localstorage.html +[11]: https://www.w3.org/TR/2009/WD-webstorage-20090423/#the-localstorage-attribute +[12]: https://rxdb.info/articles/localstorage.html#understanding-the-limitations-of-local-storage +[13]: https://www.w3.org/TR/IndexedDB/#sotd +[14]: https://rxdb.info/rx-storage-indexeddb.html +[15]: https://hacks.mozilla.org/2016/10/whats-new-in-indexeddb-2-0/ +[16]: https://w3c.github.io/IndexedDB/ +[17]: https://rxdb.info/rx-storage-opfs.html +[18]: https://caniuse.com/mdn-api_filesystemfilehandle_createsyncaccesshandle +[19]: https://rxdb.info/rx-storage-opfs.html#using-opfs-in-the-main-thread-instead-of-a-worker +[20]: https://rxdb.info/articles/json-based-database.html +[21]: https://webassembly.org/ +[22]: https://www.usenix.org/conference/atc19/presentation/jangda +[23]: https://sqlite.org/download.html +[24]: https://www.sqlite.org/vfs.html +[25]: https://www.w3.org/TR/webdatabase/ +[26]: https://developer.chrome.com/blog/deprecating-web-sql#reasons_for_deprecating_web_sql +[27]: https://www.sqlite.org/json1.html +[28]: https://rxdb.info/electron-database.html +[29]: https://rxdb.info/react-native-database.html +[30]: https://rxdb.info/articles/localstorage.html#localstorage-vs-indexeddb +[31]: https://stackoverflow.com/a/33270440 +[32]: https://github.com/pubkey/broadcast-channel +[33]: https://developer.mozilla.org/en-US/docs/Web/API/Web_Locks_API +[34]: https://developer.mozilla.org/en-US/docs/Web/API/SharedWorker +[35]: https://github.com/w3c/IndexedDB/issues/76 +[36]: https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API +[37]: https://developer.mozilla.org/en-US/docs/Web/API/Service_Worker_API +[38]: https://rxdb.info/rx-storage-worker.html +[39]: https://rxdb.info/rx-storage-shared-worker.html +[40]: https://stackoverflow.com/questions/6179159/accessing-localstorage-from-a-webworker +[41]: https://rxdb.info/rx-storage-opfs.html#opfs-limitations +[42]: https://datatracker.ietf.org/doc/html/rfc6265#section-6.1 +[43]: http://www.ruslog.com/tools/cookies.html +[44]: https://arty.name/localstorage.html +[45]: https://demo.agektmr.com/storage/ +[46]: https://rxdb.info/articles/indexeddb-max-storage-limit.html +[47]: https://github.com/pubkey/localstorage-indexeddb-cookies-opfs-sqlite-wasm +[48]: https://rxdb.info/rx-storage-localstorage-meta-optimizer.html +[49]: https://rxdb.info/rx-storage-memory-mapped.html +[50]: https://rxdb.info/key-compression.html +[51]: https://rxdb.info/rx-storage-sharding.html +[52]: https://rxdb.info/rx-storage-performance.html +[53]: https://surma.dev/things/is-postmessage-slow/ +[54]: https://developer.chrome.com/blog/maximum-idb-performance-with-storage-buckets +[55]: https://x.com/rxdbjs/status/1846145062847062391 +[56]: https://rxdb.info/quickstart.html +[57]: https://github.com/pubkey/rxdb From fd4f7d2ef8581f4d865dc1f2dcae1be65735d2f4 Mon Sep 17 00:00:00 2001 From: luojiyin Date: Mon, 3 Mar 2025 02:54:21 +0000 Subject: [PATCH 3/5] url clear --- ...indexeddb-cookies-opfs-sqlite-wasm.html.md | 44 +++++++++---------- 1 file changed, 22 insertions(+), 22 deletions(-) diff --git a/_drafts/Article/Translation/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html.md b/_drafts/Article/Translation/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html.md index decc077d..42dc4033 100644 --- a/_drafts/Article/Translation/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html.md +++ b/_drafts/Article/Translation/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html.md @@ -21,24 +21,24 @@ You are reading this in the [RxDB][5] docs. RxDB is a JavaScript database that h [![Image 1: JavaScript Database][1]][5] -The available Storage APIs in a modern Browser[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#the-available-storage-apis-in-a-modern-browser "Direct link to The available Storage APIs in a modern Browser") +The available Storage APIs in a modern Browser ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- First lets have a brief overview of the different APIs, their intentional use case and history: -### What are Cookies[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#what-are-cookies "Direct link to What are Cookies") +### What are Cookies Cookies were first introduced by [netscape in 1994][7]. Cookies store small pieces of key-value data that are mainly used for session management, personalization, and tracking. Cookies can have several security settings like a time-to-live or the `domain` attribute to share the cookies between several subdomains. Cookies values are not only stored at the client but also sent with **every http request** to the server. This means we cannot store much data in a cookie but it is still interesting how good cookie access performance compared to the other methods. Especially because cookies are such an important base feature of the web, many performance optimizations have been done and even these days there is still progress being made like the [Shared Memory Versioning][8] by chromium or the asynchronous [CookieStore API][9]. -### What is LocalStorage[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#what-is-localstorage "Direct link to What is LocalStorage") +### What is LocalStorage The [localStorage API][10] was first proposed as part of the [WebStorage specification in 2009][11]. LocalStorage provides a simple API to store key-value pairs inside of a web browser. It has the methods `setItem`, `getItem`, `removeItem` and `clear` which is all you need from a key-value store. LocalStorage is only suitable for storing small amounts of data that need to persist across sessions and it is [limited by a 5MB storage cap][12]. Storing complex data is only possible by transforming it into a string for example with `JSON.stringify()`. The API is not asynchronous which means if fully blocks your JavaScript process while doing stuff. Therefore running heavy operations on it might block your UI from rendering. > There is also the **SessionStorage** API. The key difference is that localStorage data persists indefinitely until explicitly cleared, while sessionStorage data is cleared when the browser tab or window is closed. -### What is IndexedDB[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#what-is-indexeddb "Direct link to What is IndexedDB") +### What is IndexedD IndexedDB was first introduced as "Indexed Database API" [in 2015][13]. @@ -48,7 +48,7 @@ In 2018, IndexedDB version 2.0 [was introduced][15]. This added some major impro IndexedDB [version 3.0][16] is in the workings which contains many improvements. Most important the addition of `Promise` based calls that makes modern JS features like `async/await` more useful. -### What is OPFS[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#what-is-opfs "Direct link to What is OPFS") +### What is OPFS The [Origin Private File System][17] (OPFS) is a [relatively new][18] API that allows web applications to store large files directly in the browser. It is designed for data-intensive applications that want to write and read **binary data** in a simulated file system. @@ -59,7 +59,7 @@ OPFS can be used in two modes: Because only binary data can be processed, OPFS is made to be a base filesystem for library developers. You will unlikely directly want to use the OPFS in your code when you build a "normal" application because it is too complex. That would only make sense for storing plain files like images, not to store and query [JSON data][20] efficiently. I have build a [OPFS based storage][17] for RxDB with proper indexing and querying and it took me several months. -### What is WASM SQLite[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#what-is-wasm-sqlite "Direct link to What is WASM SQLite") +### What is WASM SQLite ![Image 2: WASM SQLite][2] @@ -69,7 +69,7 @@ Many people started to use compiled SQLite as a database inside of the browser w The compiled byte code of SQLite has a size of [about 938.9 kB][23] which must be downloaded and parsed by the users on the first page load. WASM cannot directly access any persistent storage API in the browser. Instead it requires data to flow from WASM to the main-thread and then can be put into one of the browser APIs. This is done with so called [VFS (virtual file system) adapters][24] that handle data access from SQLite to anything else. -### What was WebSQL[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#what-was-websql "Direct link to What was WebSQL") +### What was WebSQL WebSQL **was** a web API [introduced in 2009][25] that allowed browsers to use SQL databases for client-side storage, based on SQLite. The idea was to give developers a way to store and query data using SQL on the client side, similar to server-side databases. WebSQL has been **removed from browsers** in the current years for multiple good reasons: @@ -81,12 +81,12 @@ Therefore in the following we will **just ignore WebSQL** even if it would be po * * * -Feature Comparison[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#feature-comparison "Direct link to Feature Comparison") +Feature Comparison ------------------------------------------------------------------------------------------------------------------------------------------------------------- Now that you know the basic concepts of the APIs, lets compare some specific features that have shown to be important for people using RxDB and browser based storages in general. -### Storing complex JSON Documents[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#storing-complex-json-documents "Direct link to Storing complex JSON Documents") +### Storing complex JSON Documents When you store data in a web application, most often you want to store complex JSON documents and not only "normal" values like the `integers` and `strings` you store in a server side database. @@ -95,7 +95,7 @@ When you store data in a web application, most often you want to store complex J Every of the other APIs can only store strings or binary data. Of course you can transform any JSON object to a string with `JSON.stringify()` but not having the JSON support in the API can make things complex when running queries and running `JSON.stringify()` many times can cause performance problems. -### Multi-Tab Support[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#multi-tab-support "Direct link to Multi-Tab Support") +### Multi-Tab Support A big difference when building a Web App compared to [Electron][28] or [React-Native][29], is that the user will open and close the app in **multiple browser tabs at the same time**. Therefore you have not only one JavaScript process running, but many of them can exist and might have to share state changes between each other to not show **outdated data** to the user. @@ -116,7 +116,7 @@ To workaround this problem, there are two solutions: * The first option is to use the [BroadcastChannel API][32] which can send messages across browser tabs. So whenever you do a write to the storage, you also send a notification to other tabs to inform them about these changes. This is the most common workaround which is also used by RxDB. Notice that there is also the [WebLocks API][33] which can be used to have mutexes across browser tabs. * The other solution is to use the [SharedWorker][34] and do all writes inside of the worker. All browser tabs can then subscribe to messages from that **single** SharedWorker and know about changes. -### Indexing Support[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#indexing-support "Direct link to Indexing Support") +### Indexing Support The big difference between a database and storing data in a plain file, is that a database is writing data in a format that allows running operations over indexes to facilitate fast performant queries. From our list of technologies only **IndexedDB** and **WASM SQLite** support for indexing out of the box. In theory you can build indexes on top of any storage like localstorage or OPFS but you likely should not want to do that by yourself. @@ -128,7 +128,7 @@ In IndexedDB for example, we can fetch a bulk of documents by a given index rang Notice that IndexedDB has the limitation of [not having indexes on boolean values][35]. You can only index strings and numbers. To workaround that you have to transform boolean to numbers and backwards when storing the data. -### WebWorker Support[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#webworker-support "Direct link to WebWorker Support") +### WebWorker Support When running heavy data operations, you might want to move the processing away from the JavaScript main thread. This ensures that our app keeps being responsive and fast while the processing can run in parallel in the background. In a browser you can either use the [WebWorker][36], [SharedWorker][34] or the [ServiceWorker][37] API to do that. In RxDB you can use the [WebWorker][38] or [SharedWorker][39] plugins to move your storage inside of a worker. @@ -140,7 +140,7 @@ Everything else can be used from inside a WebWorker. The fast version of OPFS wi * * * -Storage Size Limits[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#storage-size-limits "Direct link to Storage Size Limits") +Storage Size Limit ---------------------------------------------------------------------------------------------------------------------------------------------------------------- * **Cookies** are limited to about `4 KB` of data in [RFC-6265][42]. Because the stored cookies are send to the server with every HTTP request, this limitation is reasonable. You can test your browsers cookie limits [here][43]. Notice that you should never fill up the full `4 KB` of your cookies because your websserver will not accept too long headers and reject the requests with `HTTP ERROR 431 - Request header fields too large`. Once you have reached that point you can not even serve updated JavaScript to your user to clean up the cookies and you will have locked out that user until the cookies get cleaned up manually. @@ -157,14 +157,14 @@ Storage Size Limits[​](https://rxdb.info/articles/localstorage-indexeddb-cooki * * * -Performance Comparison[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#performance-comparison "Direct link to Performance Comparison") +Performance Comparison ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Now that we've reviewed the features of each storage method, let's dive into performance comparisons, focusing on initialization times, read/write latencies, and bulk operations. Notice that we only run simple tests and for your specific use case in your application the results might differ. Also we only compare performance in google chrome (version 128.0.6613.137). Firefox and Safari have similar **but not equal** performance patterns. You can run the test by yourself on your own machine from this [github repository][47]. For all tests we throttle the network to behave like the average german internet speed. (download: 135,900 kbit/s, upload: 28,400 kbit/s, latency: 125ms). Also all tests store an "average" JSON object that might be required to be stringified depending on the storage. We also only test the performance of storing documents by id because some of the technologies (cookies, OPFS and localstorage) do not support indexed range operations so it makes no sense to compare the performance of these. -### Initialization Time[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#initialization-time "Direct link to Initialization Time") +### Initialization Time Before you can store any data, many APIs require a setup process like creating databases, spawning WebAssembly processes or downloading additional stuff. To ensure your app starts fast, the initialization time is important. @@ -186,7 +186,7 @@ Here we can notice a few things: * The latency overhead of sending data from the main thread to a WebWorker OPFS is about 4 milliseconds. Here we only send minimal data to init the OPFS file handler. It will be interesting if that latency increases when more data is processed. * Downloading and parsing WASM SQLite and creating a single table takes about half a second. Using also the IndexedDB VFS to store data persistently adds additional 31 milliseconds. Reloading the page with enabled caching and already prepared tables is a bit faster with 420 milliseconds (memory). -### Latency of small Writes[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#latency-of-small-writes "Direct link to Latency of small Writes") +### Latency of small Writes Next lets test the latency of small writes. This is important when you do many small data changes that happen independent from each other. Like when you stream data from a websocket or persist pseudo randomly happening events like mouse movements. @@ -208,7 +208,7 @@ Here we can notice a few things: The OPFS operations take about 1.5 milliseconds to write the JSON data into one document per file. We can see the sending the data to a webworker first is a bit slower which comes from the overhead of serializing and deserializing the data on both sides. If we would not create on OPFS file per document but instead append everything to a single file, the performance pattern changes significantly. Then the faster file handle from the `createSyncAccessHandle()` only takes about 1 millisecond per write. But this would require to somehow remember at which position the each document is stored. Therefore in our tests we will continue using one file per document. -### Latency of small Reads[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#latency-of-small-reads "Direct link to Latency of small Reads") +### Latency of small Reads Now that we have stored some documents, lets measure how long it takes to read single documents by their `id`. @@ -227,7 +227,7 @@ Here we can notice a few things: * LocalStorage reads are **really really fast** with only 0.0052 milliseconds per read. * The other technologies perform reads in a similar speed to their write latency. -### Big Bulk Writes[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#big-bulk-writes "Direct link to Big Bulk Writes") +### Big Bulk Writes As next step, lets do some big bulk operations with 200 documents at once. @@ -246,7 +246,7 @@ Here we can notice a few things: * Sending the data to a WebWorker and running it via the faster OPFS API is about twice as fast. * WASM SQLite performs better on bulk operations compared to its single write latency. This is because sending the data to WASM and backwards is faster if it is done all at once instead of once per document. -### Big Bulk Reads[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#big-bulk-reads "Direct link to Big Bulk Reads") +### Big Bulk Reads Now lets read 100 documents in a bulk request. @@ -265,7 +265,7 @@ Here we can notice a few things: * Reading many files in the OPFS webworker is about **twice as fast** compared to the slower main thread mode. * WASM SQLite is surprisingly fast. Further inspection has shown that the WASM SQLite process keeps the documents in memory cached which improves the latency when we do reads directly after writes on the same data. When the browser tab is reloaded between the writes and the reads, finding the 100 documents takes about **35 milliseconds** instead. -Performance Conclusions[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#performance-conclusions "Direct link to Performance Conclusions") +Performance Conclusions ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- * LocalStorage is really fast but remember that is has some downsides: @@ -276,7 +276,7 @@ Performance Conclusions[​](https://rxdb.info/articles/localstorage-indexeddb-c * * * -Possible Improvements[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#possible-improvements "Direct link to Possible Improvements") +Possible Improvements ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- There is a wide range of possible improvements and performance hacks to speed up the operations. @@ -292,7 +292,7 @@ Here you can see the [performance comparison][52] of various RxDB storage implem ![Image 3: RxStorage performance - browser][3] -Future Improvements[​](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html#future-improvements "Direct link to Future Improvements") +Future Improvements ---------------------------------------------------------------------------------------------------------------------------------------------------------------- You are reading this in 2024, but the web does not stand still. There is a good chance that browser get enhanced to allow faster and better data operations. From 96c413f3e767f5b843ae4d289a1fcb6d8e6e8575 Mon Sep 17 00:00:00 2001 From: luojiyin Date: Mon, 3 Mar 2025 03:12:16 +0000 Subject: [PATCH 4/5] finish translate --- ...indexeddb-cookies-opfs-sqlite-wasm.html.md | 298 +++++++++--------- 1 file changed, 149 insertions(+), 149 deletions(-) diff --git a/_drafts/Article/Translation/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html.md b/_drafts/Article/Translation/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html.md index 42dc4033..e43fd440 100644 --- a/_drafts/Article/Translation/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html.md +++ b/_drafts/Article/Translation/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html.md @@ -7,307 +7,307 @@ translator: "" reviewer: "" --- -So you are building that web application and you want to **store data inside of your users browser**. Maybe you just need to store some small flags or you even need a fully fledged database. +当你正在构建一个Web应用程序时，你可能希望**将数据存储在用户的浏览器中**。也许你只需要存储一些小标志，或者你甚至需要一个完整的数据库。 -The types of web applications we build have changed significantly. In the early years of the web we served static html files. Then we served dynamically rendered html and later we build **single page applications** that run most logic on the client. And for the coming years you might want to build so called [local first apps][4] that handle big and complex data operations solely on the client and even work when offline, which gives you the opportunity to build **zero-latency** user interactions. +我们构建的Web应用程序类型已经发生了显著变化。在Web的早期，我们提供静态的HTML文件。然后我们提供动态渲染的HTML，后来我们构建了**单页应用程序**，这些应用程序在客户端运行大部分逻辑。而在未来几年，你可能希望构建所谓的[本地优先应用程序][4]，这些应用程序在客户端处理大量复杂的数据操作，甚至在离线时也能工作，这为你提供了构建**零延迟**用户交互的机会。 -In the early days of the web, **cookies** were the only option for storing small key-value assignments.. But JavaScript and browsers have evolved significantly and better storage APIs have been added which pave the way for bigger and more complex data operations. +在Web的早期，**Cookies**是存储小型键值对的唯一选择。但JavaScript和浏览器已经显著发展，并添加了更好的存储API，为更大、更复杂的数据操作铺平了道路。 -In this article, we will dive into the various technologies available for storing and querying data in a browser. We'll explore traditional methods like **Cookies**, **localStorage**, **WebSQL**, **IndexedDB** and newer solutions such as **OPFS** and **SQLite via WebAssembly**. We compare the features and limitations and through performance tests we aim to uncover how fast we can write and read data in a web application with the various methods. +在本文中，我们将深入探讨在浏览器中存储和查询数据的各种技术。我们将探索传统方法，如**Cookies**、**localStorage**、**WebSQL**、**IndexedDB**，以及较新的解决方案，如**OPFS**和**通过WebAssembly的SQLite**。我们将比较这些技术的功能和限制，并通过性能测试揭示在Web应用程序中使用各种方法读写数据的速度。 -note +注意 -You are reading this in the [RxDB][5] docs. RxDB is a JavaScript database that has different storage adapters which can utilize the different storage APIs. **Since 2017** I spend most of my time working with these APIs, doing performance tests and building [hacks][6] and plugins to reach the limits of browser database operation speed. +你正在[RxDB][5]文档中阅读本文。RxDB是一个JavaScript数据库，具有不同的存储适配器，可以利用不同的存储API。**自2017年以来**，我大部分时间都在与这些API打交道，进行性能测试，并构建[技巧][6]和插件，以达到浏览器数据库操作速度的极限。 [![Image 1: JavaScript Database][1]][5] -The available Storage APIs in a modern Browser +现代浏览器中可用的存储API ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- -First lets have a brief overview of the different APIs, their intentional use case and history: +首先，让我们简要概述不同的API、它们的预期用例和历史： -### What are Cookies +### 什么是Cookies -Cookies were first introduced by [netscape in 1994][7]. Cookies store small pieces of key-value data that are mainly used for session management, personalization, and tracking. Cookies can have several security settings like a time-to-live or the `domain` attribute to share the cookies between several subdomains. +Cookies最早由[网景公司于1994年][7]引入。Cookies存储小型的键值数据，主要用于会话管理、个性化和跟踪。Cookies可以具有多个安全设置，如生存时间或`domain`属性，以在多个子域之间共享Cookies。 -Cookies values are not only stored at the client but also sent with **every http request** to the server. This means we cannot store much data in a cookie but it is still interesting how good cookie access performance compared to the other methods. Especially because cookies are such an important base feature of the web, many performance optimizations have been done and even these days there is still progress being made like the [Shared Memory Versioning][8] by chromium or the asynchronous [CookieStore API][9]. +Cookies的值不仅存储在客户端，还会随**每个HTTP请求**发送到服务器。这意味着我们不能在Cookie中存储太多数据，但Cookie的访问性能与其他方法相比仍然值得关注。特别是因为Cookies是Web的重要基础功能，许多性能优化已经完成，甚至在这些日子里，仍然有进展，如Chromium的[共享内存版本控制][8]或异步的[CookieStore API][9]。 -### What is LocalStorage +### 什么是LocalStorage -The [localStorage API][10] was first proposed as part of the [WebStorage specification in 2009][11]. LocalStorage provides a simple API to store key-value pairs inside of a web browser. It has the methods `setItem`, `getItem`, `removeItem` and `clear` which is all you need from a key-value store. LocalStorage is only suitable for storing small amounts of data that need to persist across sessions and it is [limited by a 5MB storage cap][12]. Storing complex data is only possible by transforming it into a string for example with `JSON.stringify()`. The API is not asynchronous which means if fully blocks your JavaScript process while doing stuff. Therefore running heavy operations on it might block your UI from rendering. +[localStorage API][10]最早作为[WebStorage规范的一部分于2009年][11]提出。LocalStorage提供了一个简单的API，用于在Web浏览器中存储键值对。它具有`setItem`、`getItem`、`removeItem`和`clear`方法，这些方法是你从键值存储中所需的一切。LocalStorage仅适用于存储少量需要在会话之间持久化的数据，并且它[受到5MB存储上限的限制][12]。存储复杂数据只能通过将其转换为字符串来实现，例如使用`JSON.stringify()`。该API不是异步的，这意味着它在执行操作时会完全阻塞你的JavaScript进程。因此，在其上运行繁重的操作可能会阻止你的UI渲染。 -> There is also the **SessionStorage** API. The key difference is that localStorage data persists indefinitely until explicitly cleared, while sessionStorage data is cleared when the browser tab or window is closed. +> 还有**SessionStorage** API。关键区别在于localStorage数据会无限期持久化，直到显式清除，而sessionStorage数据在浏览器标签或窗口关闭时会被清除。 -### What is IndexedD +### 什么是IndexedDB -IndexedDB was first introduced as "Indexed Database API" [in 2015][13]. +IndexedDB最早作为"Indexed Database API"[于2015年][13]引入。 -[IndexedDB][14] is a low-level API for storing large amounts of structured JSON data. While the API is a bit hard to use, IndexedDB can utilize indexes and asynchronous operations. It lacks support for complex queries and only allows to iterate over the indexes which makes it more like a base layer for other libraries then a fully fledged database. +[IndexedDB][14]是一个用于存储大量结构化JSON数据的低级API。虽然该API有点难以使用，但IndexedDB可以利用索引和异步操作。它缺乏对复杂查询的支持，只允许在索引上进行迭代，这使得它更像是其他库的基础层，而不是一个完全成熟的数据库。 -In 2018, IndexedDB version 2.0 [was introduced][15]. This added some major improvements. Most noticeable the `getAll()` method which improves performance dramatically when fetching bulks of JSON documents. +2018年，IndexedDB 2.0版本[被引入][15]。这增加了一些重大改进。最显著的是`getAll()`方法，它在获取大量JSON文档时显著提高了性能。 -IndexedDB [version 3.0][16] is in the workings which contains many improvements. Most important the addition of `Promise` based calls that makes modern JS features like `async/await` more useful. +IndexedDB [3.0版本][16]正在开发中，包含了许多改进。最重要的是添加了基于`Promise`的调用，这使得现代JS功能如`async/await`更加有用。 -### What is OPFS +### 什么是OPFS -The [Origin Private File System][17] (OPFS) is a [relatively new][18] API that allows web applications to store large files directly in the browser. It is designed for data-intensive applications that want to write and read **binary data** in a simulated file system. +[Origin Private File System][17]（OPFS）是一个[相对较新][18]的API，允许Web应用程序直接在浏览器中存储大文件。它专为数据密集型应用程序设计，这些应用程序希望在模拟的文件系统中写入和读取**二进制数据**。 -OPFS can be used in two modes: +OPFS可以在两种模式下使用： -* Either asynchronous on the [main thread][19] -* Or in a WebWorker with the faster, asynchronous access with the `createSyncAccessHandle()` method. +* 在[主线程][19]上异步使用 +* 或者在WebWorker中使用更快的异步访问，通过`createSyncAccessHandle()`方法。 -Because only binary data can be processed, OPFS is made to be a base filesystem for library developers. You will unlikely directly want to use the OPFS in your code when you build a "normal" application because it is too complex. That would only make sense for storing plain files like images, not to store and query [JSON data][20] efficiently. I have build a [OPFS based storage][17] for RxDB with proper indexing and querying and it took me several months. +由于只能处理二进制数据，OPFS被设计为库开发者的基础文件系统。当你构建一个"普通"应用程序时，你不太可能直接使用OPFS，因为它太复杂了。这仅适用于存储纯文件（如图像），而不是高效地存储和查询[JSON数据][20]。我为RxDB构建了一个[基于OPFS的存储][17]，具有适当的索引和查询功能，这花了我几个月的时间。 -### What is WASM SQLite +### 什么是WASM SQLite ![Image 2: WASM SQLite][2] -[WebAssembly][21] (Wasm) is a binary format that allows high-performance code execution on the web. Wasm was added to major browsers over the course of 2017 which opened a wide range of opportunities on what to run inside of a browser. You can compile native libraries to WebAssembly and just run them on the client with just a few adjustments. WASM code can be shipped to browser apps and generally runs much faster compared to JavaScript, but still about [10% slower then native][22]. +[WebAssembly][21]（Wasm）是一种二进制格式，允许在Web上执行高性能代码。Wasm在2017年被添加到主要浏览器中，这为在浏览器中运行的内容开辟了广泛的机会。你可以将本地库编译为WebAssembly，只需稍作调整即可在客户端运行。WASM代码可以发送到浏览器应用程序，并且通常比JavaScript运行得更快，但仍然比本地代码[慢约10%][22]。 -Many people started to use compiled SQLite as a database inside of the browser which is why it makes sense to also compare this setup to the native APIs. +许多人开始在浏览器中使用编译后的SQLite作为数据库，这就是为什么将这种设置与本地API进行比较也是有意义的。 -The compiled byte code of SQLite has a size of [about 938.9 kB][23] which must be downloaded and parsed by the users on the first page load. WASM cannot directly access any persistent storage API in the browser. Instead it requires data to flow from WASM to the main-thread and then can be put into one of the browser APIs. This is done with so called [VFS (virtual file system) adapters][24] that handle data access from SQLite to anything else. +SQLite的编译字节码大小约为[938.9 kB][23]，必须在首次页面加载时由用户下载和解析。WASM不能直接访问浏览器中的任何持久存储API。相反，它需要数据从WASM流向主线程，然后可以放入浏览器API之一。这是通过所谓的[VFS（虚拟文件系统）适配器][24]完成的，这些适配器处理从SQLite到其他任何东西的数据访问。 -### What was WebSQL +### 什么是WebSQL -WebSQL **was** a web API [introduced in 2009][25] that allowed browsers to use SQL databases for client-side storage, based on SQLite. The idea was to give developers a way to store and query data using SQL on the client side, similar to server-side databases. WebSQL has been **removed from browsers** in the current years for multiple good reasons: +WebSQL**曾经**是一个Web API，[于2009年][25]引入，允许浏览器使用SQL数据库进行客户端存储，基于SQLite。其想法是让开发者能够使用SQL在客户端存储和查询数据，类似于服务器端数据库。WebSQL在近年已**从浏览器中移除**，原因如下： -* WebSQL was not standardized and having an API based on a single specific implementation in form of the SQLite source code is hard to ever make it to a standard. -* WebSQL required browsers to use a [specific version][26] of SQLite (version 3.6.19) which means whenever there would be any update or bugfix to SQLite, it would not be possible to add that to WebSQL without possible breaking the web. -* Major browsers like firefox never supported WebSQL. +* WebSQL没有标准化，并且基于SQLite源代码的单一特定实现的API很难成为标准。 +* WebSQL要求浏览器使用[特定版本][26]的SQLite（版本3.6.19），这意味着每当SQLite有任何更新或错误修复时，如果不破坏Web，就不可能将其添加到WebSQL中。 +* 像Firefox这样的主要浏览器从未支持WebSQL。 -Therefore in the following we will **just ignore WebSQL** even if it would be possible to run tests on in by setting specific browser flags or using old versions of chromium. +因此，在接下来的内容中，我们将**忽略WebSQL**，即使通过设置特定的浏览器标志或使用旧版本的Chromium，仍然可以对其进行测试。 * * * -Feature Comparison +功能比较 ------------------------------------------------------------------------------------------------------------------------------------------------------------- -Now that you know the basic concepts of the APIs, lets compare some specific features that have shown to be important for people using RxDB and browser based storages in general. +现在你已经了解了这些API的基本概念，让我们比较一些特定的功能，这些功能对于使用RxDB和基于浏览器的存储的人来说非常重要。 -### Storing complex JSON Documents +### 存储复杂的JSON文档 -When you store data in a web application, most often you want to store complex JSON documents and not only "normal" values like the `integers` and `strings` you store in a server side database. +当你在Web应用程序中存储数据时，大多数情况下你希望存储复杂的JSON文档，而不仅仅是存储在服务器端数据库中的`整数`和`字符串`等"普通"值。 -* Only IndexedDB works with JSON objects natively. -* With SQLite WASM you can [store JSON][27] in a `text` column since version 3.38.0 (2022-02-22) and even run deep queries on it and use single attributes as indexes. +* 只有IndexedDB原生支持JSON对象。 +* 使用SQLite WASM，你可以[存储JSON][27]在`text`列中，自版本3.38.0（2022-02-22）起，甚至可以在其上运行深度查询，并使用单个属性作为索引。 -Every of the other APIs can only store strings or binary data. Of course you can transform any JSON object to a string with `JSON.stringify()` but not having the JSON support in the API can make things complex when running queries and running `JSON.stringify()` many times can cause performance problems. +其他API只能存储字符串或二进制数据。当然，你可以使用`JSON.stringify()`将任何JSON对象转换为字符串，但在API中没有JSON支持会使查询变得复杂，并且多次运行`JSON.stringify()`可能会导致性能问题。 -### Multi-Tab Support +### 多标签支持 -A big difference when building a Web App compared to [Electron][28] or [React-Native][29], is that the user will open and close the app in **multiple browser tabs at the same time**. Therefore you have not only one JavaScript process running, but many of them can exist and might have to share state changes between each other to not show **outdated data** to the user. +构建Web应用程序与[Electron][28]或[React-Native][29]相比，一个很大的区别是用户会**同时打开和关闭多个浏览器标签**。因此，你不仅有一个JavaScript进程在运行，而且可能存在多个进程，并且它们可能需要在彼此之间共享状态更改，以避免向用户显示**过时的数据**。 -> If your users' muscle memory puts the left hand on the **F5** key while using your website, you did something wrong! +> 如果你的用户的肌肉记忆让他们在使用你的网站时将左手放在**F5**键上，那么你做错了什么！ -Not all storage APIs support a way to automatically share write events between tabs. +并非所有存储API都支持在标签之间自动共享写入事件。 -Only localstorage has a way to automatically share write events between tabs by the API itself with the [storage-event][30] which can be used to observe changes. +只有localStorage有一种通过API本身在标签之间自动共享写入事件的方式，即[storage-event][30]，它可以用于观察更改。 ``` -// localStorage can observe changes with the storage event.// This feature is missing in IndexedDB and othersaddEventListener("storage", (event) => {}); +// localStorage可以通过storage事件观察更改。// 这个功能在IndexedDB和其他API中缺失addEventListener("storage", (event) => {}); ``` -There was the [experimental IndexedDB observers API][31] for chrome, but the proposal repository has been archived. +曾经有[实验性的IndexedDB观察者API][31]用于Chrome，但提案仓库已被归档。 -To workaround this problem, there are two solutions: +为了解决这个问题，有两种解决方案： -* The first option is to use the [BroadcastChannel API][32] which can send messages across browser tabs. So whenever you do a write to the storage, you also send a notification to other tabs to inform them about these changes. This is the most common workaround which is also used by RxDB. Notice that there is also the [WebLocks API][33] which can be used to have mutexes across browser tabs. -* The other solution is to use the [SharedWorker][34] and do all writes inside of the worker. All browser tabs can then subscribe to messages from that **single** SharedWorker and know about changes. +* 第一种选择是使用[BroadcastChannel API][32]，它可以在浏览器标签之间发送消息。因此，每当你向存储写入数据时，你还可以向其他标签发送通知，告知这些更改。这是最常见的解决方法，RxDB也使用了这种方法。注意，还有[WebLocks API][33]，它可以用于在浏览器标签之间实现互斥锁。 +* 另一种解决方案是使用[SharedWorker][34]，并在该worker中执行所有写入操作。所有浏览器标签都可以订阅来自该**单一**SharedWorker的消息，并了解更改。 -### Indexing Support +### 索引支持 -The big difference between a database and storing data in a plain file, is that a database is writing data in a format that allows running operations over indexes to facilitate fast performant queries. From our list of technologies only **IndexedDB** and **WASM SQLite** support for indexing out of the box. In theory you can build indexes on top of any storage like localstorage or OPFS but you likely should not want to do that by yourself. +数据库与将数据存储在普通文件中的最大区别在于，数据库以允许通过索引运行操作的格式写入数据，从而支持快速查询。在我们的技术列表中，只有**IndexedDB**和**WASM SQLite**原生支持索引。理论上，你可以在任何存储（如localStorage或OPFS）上构建索引，但你可能不想自己动手。 -In IndexedDB for example, we can fetch a bulk of documents by a given index range: +例如，在IndexedDB中，我们可以通过给定的索引范围获取一批文档： ``` -// find all products with a price between 10 and 50const keyRange = IDBKeyRange.bound(10, 50);const transaction = db.transaction('products', 'readonly');const objectStore = transaction.objectStore('products');const index = objectStore.index('priceIndex');const request = index.getAll(keyRange);const result = await new Promise((res, rej) => { request.onsuccess = (event) => res(event.target.result); request.onerror = (event) => rej(event);}); +// 查找所有价格在10到50之间的产品const keyRange = IDBKeyRange.bound(10, 50);const transaction = db.transaction('products', 'readonly');const objectStore = transaction.objectStore('products');const index = objectStore.index('priceIndex');const request = index.getAll(keyRange);const result = await new Promise((res, rej) => { request.onsuccess = (event) => res(event.target.result); request.onerror = (event) => rej(event);}); ``` -Notice that IndexedDB has the limitation of [not having indexes on boolean values][35]. You can only index strings and numbers. To workaround that you have to transform boolean to numbers and backwards when storing the data. +注意，IndexedDB有一个限制，即[不支持布尔值的索引][35]。你只能索引字符串和数字。为了解决这个问题，你必须在存储数据时将布尔值转换为数字，并在读取时转换回来。 -### WebWorker Support +### WebWorker支持 -When running heavy data operations, you might want to move the processing away from the JavaScript main thread. This ensures that our app keeps being responsive and fast while the processing can run in parallel in the background. In a browser you can either use the [WebWorker][36], [SharedWorker][34] or the [ServiceWorker][37] API to do that. In RxDB you can use the [WebWorker][38] or [SharedWorker][39] plugins to move your storage inside of a worker. +当运行繁重的数据操作时，你可能希望将处理从JavaScript主线程中移出。这确保了我们的应用程序保持响应和快速，而处理可以在后台并行运行。在浏览器中，你可以使用[WebWorker][36]、[SharedWorker][34]或[ServiceWorker][37] API来实现这一点。在RxDB中，你可以使用[WebWorker][38]或[SharedWorker][39]插件将存储移动到worker中。 -The most common API for that use case is spawning a **WebWorker** and doing most work on that second JavaScript process. The worker is spawned from a separate JavaScript file (or base64 string) and communicates with the main thread by sending data with `postMessage()`. +最常见的用例是生成一个**WebWorker**，并在第二个JavaScript进程中执行大部分工作。worker是从一个单独的JavaScript文件（或base64字符串）生成的，并通过`postMessage()`与主线程通信。 -Unfortunately **LocalStorage** and **Cookies** [cannot be used in WebWorker or SharedWorker][40] because of the design and security constraints. WebWorkers run in a separate global context from the main browser thread and therefore cannot do stuff that might impact the main thread. They have no direct access to certain web APIs, like the DOM, localStorage, or cookies. +不幸的是，**LocalStorage**和**Cookies**[不能在WebWorker或SharedWorker中使用][40]，这是由于设计和安全限制。WebWorkers在与主浏览器线程分离的全局上下文中运行，因此不能执行可能影响主线程的操作。它们无法直接访问某些Web API，如DOM、localStorage或cookies。 -Everything else can be used from inside a WebWorker. The fast version of OPFS with the `createSyncAccessHandle` method can **only** [be used in a WebWorker][41], and **not on the main thread**. This is because all the operations of the returned `AccessHandle` are **not async** and therefore block the JavaScript process, so you do want to do that on the main thread and block everything. +其他所有内容都可以在WebWorker中使用。OPFS的快速版本`createSyncAccessHandle`方法**只能**[在WebWorker中使用][41]，而**不能在主线程中使用**。这是因为返回的`AccessHandle`的所有操作都是**非异步的**，因此会阻塞JavaScript进程，所以你肯定不希望在主线程中执行这些操作并阻塞一切。 * * * -Storage Size Limit +存储大小限制 ---------------------------------------------------------------------------------------------------------------------------------------------------------------- -* **Cookies** are limited to about `4 KB` of data in [RFC-6265][42]. Because the stored cookies are send to the server with every HTTP request, this limitation is reasonable. You can test your browsers cookie limits [here][43]. Notice that you should never fill up the full `4 KB` of your cookies because your websserver will not accept too long headers and reject the requests with `HTTP ERROR 431 - Request header fields too large`. Once you have reached that point you can not even serve updated JavaScript to your user to clean up the cookies and you will have locked out that user until the cookies get cleaned up manually. +* **Cookies** 在 [RFC-6265][42] 中被限制为大约 `4 KB` 的数据。由于存储的 Cookies 会随每个 HTTP 请求发送到服务器，这个限制是合理的。你可以在这里测试浏览器的 Cookies 限制[43]。需要注意的是，你永远不应该填满 `4 KB` 的 Cookies，因为你的 Web 服务器不会接受过长的请求头，并会以 `HTTP ERROR 431 - 请求头字段过大` 拒绝请求。一旦达到这个点，你甚至无法向用户提供更新的 JavaScript 来清理 Cookies，用户将被锁定，直到手动清理 Cookies。 -* **LocalStorage** has a storage size limitation that varies depending on the browser, but generally ranges from 4 MB to 10 MB per origin. You can test your localStorage size limit [here][44]. +* **LocalStorage** 的存储大小限制因浏览器而异，但通常在 4 MB 到 10 MB 之间。你可以在这里测试你的 localStorage 大小限制[44]。 - * Chrome/Chromium/Edge: 5 MB per domain - * Firefox: 10 MB per domain - * Safari: 4-5 MB per domain (varies slightly between versions) -* **IndexedDB** does not have a specific fixed size limitation like localStorage. The maximum storage size for IndexedDB depends on the browser implementation. The upper limit is typically based on the available disc space on the user's device. In chromium browsers it can use up to 80% of total disk space. You can get an estimation about the storage size limit by calling `await navigator.storage.estimate()`. Typically you can store gigabytes of data which can be tried out [here][45]. Notice that we have a full article about [storage max size limits of IndexedDB][46] that covers this topic. + * Chrome/Chromium/Edge: 每个域名 5 MB + * Firefox: 每个域名 10 MB + * Safari: 每个域名 4-5 MB（不同版本略有差异） +* **IndexedDB** 没有像 localStorage 那样的固定大小限制。IndexedDB 的最大存储大小取决于浏览器的实现。上限通常基于用户设备上的可用磁盘空间。在 Chromium 浏览器中，它可以使用高达 80% 的总磁盘空间。你可以通过调用 `await navigator.storage.estimate()` 来估算存储大小限制。通常你可以存储千兆字节的数据，可以在这里尝试[45]。请注意，我们有一篇关于 [IndexedDB 存储最大限制][46] 的完整文章，涵盖了该主题。 -* **OPFS** has the same storage size limitation as IndexedDB. Its limit depends on the available disc space. This can also be tested [here][45]. +* **OPFS** 的存储大小限制与 IndexedDB 相同。它的限制取决于可用磁盘空间。也可以在这里测试[45]。 * * * -Performance Comparison +性能比较 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- -Now that we've reviewed the features of each storage method, let's dive into performance comparisons, focusing on initialization times, read/write latencies, and bulk operations. +现在我们已经回顾了每种存储方法的功能，让我们深入探讨性能比较，重点关注初始化时间、读写延迟和批量操作。 -Notice that we only run simple tests and for your specific use case in your application the results might differ. Also we only compare performance in google chrome (version 128.0.6613.137). Firefox and Safari have similar **but not equal** performance patterns. You can run the test by yourself on your own machine from this [github repository][47]. For all tests we throttle the network to behave like the average german internet speed. (download: 135,900 kbit/s, upload: 28,400 kbit/s, latency: 125ms). Also all tests store an "average" JSON object that might be required to be stringified depending on the storage. We also only test the performance of storing documents by id because some of the technologies (cookies, OPFS and localstorage) do not support indexed range operations so it makes no sense to compare the performance of these. +请注意，我们只运行了简单的测试，对于你的应用程序中的特定用例，结果可能会有所不同。此外，我们只在 Google Chrome（版本 128.0.6613.137）中比较了性能。Firefox 和 Safari 有类似但**不完全相同**的性能模式。你可以在这个 [GitHub 仓库][47] 中在自己的机器上运行测试。对于所有测试，我们将网络限制为德国平均互联网速度。（下载：135,900 kbit/s，上传：28,400 kbit/s，延迟：125ms）。此外，所有测试都存储一个"平均"的 JSON 对象，可能需要根据存储方式将其字符串化。我们还只测试了按 ID 存储文档的性能，因为某些技术（Cookies、OPFS 和 localStorage）不支持索引范围操作，因此比较这些操作的性能没有意义。 -### Initialization Time +### 初始化时间 -Before you can store any data, many APIs require a setup process like creating databases, spawning WebAssembly processes or downloading additional stuff. To ensure your app starts fast, the initialization time is important. +在存储任何数据之前，许多 API 需要一个设置过程，比如创建数据库、启动 WebAssembly 进程或下载额外的内容。为了确保你的应用程序快速启动，初始化时间非常重要。 -The APIs of localStorage and Cookies do not have any setup process and can be directly used. IndexedDB requires to open a database and a store inside of it. WASM SQLite needs to download a WASM file and process it. OPFS needs to download and start a worker file and initialize the virtual file system directory. +localStorage 和 Cookies 的 API 没有任何设置过程，可以直接使用。IndexedDB 需要打开一个数据库并在其中创建一个存储。WASM SQLite 需要下载一个 WASM 文件并处理它。OPFS 需要下载并启动一个 worker 文件，并初始化虚拟文件系统目录。 -Here are the time measurements from how long it takes until the first bit of data can be stored: +以下是存储第一比特数据所需的时间测量结果： -| Technology | Time in Milliseconds | +| 技术 | 时间（毫秒） | | --- | --- | | IndexedDB | 46 | -| OPFS Main Thread | 23 | +| OPFS 主线程 | 23 | | OPFS WebWorker | 26.8 | -| WASM SQLite (memory) | 504 | -| WASM SQLite (IndexedDB) | 535 | +| WASM SQLite（内存） | 504 | +| WASM SQLite（IndexedDB） | 535 | -Here we can notice a few things: +这里我们可以注意到几点： -* Opening a new IndexedDB database with a single store takes surprisingly long -* The latency overhead of sending data from the main thread to a WebWorker OPFS is about 4 milliseconds. Here we only send minimal data to init the OPFS file handler. It will be interesting if that latency increases when more data is processed. -* Downloading and parsing WASM SQLite and creating a single table takes about half a second. Using also the IndexedDB VFS to store data persistently adds additional 31 milliseconds. Reloading the page with enabled caching and already prepared tables is a bit faster with 420 milliseconds (memory). +* 打开一个新的 IndexedDB 数据库并创建一个存储需要的时间出人意料地长 +* 将数据从主线程发送到 WebWorker OPFS 的延迟开销约为 4 毫秒。这里我们只发送了最少的数据来初始化 OPFS 文件句柄。如果处理更多数据，这个延迟是否会增加将是一个有趣的问题。 +* 下载并解析 WASM SQLite 并创建一个表大约需要半秒钟。使用 IndexedDB VFS 持久化存储数据还会额外增加 31 毫秒。在启用缓存并已准备好表的情况下重新加载页面会稍微快一些，为 420 毫秒（内存）。 -### Latency of small Writes +### 少量写操作的延迟 -Next lets test the latency of small writes. This is important when you do many small data changes that happen independent from each other. Like when you stream data from a websocket or persist pseudo randomly happening events like mouse movements. +接下来，我们测试少量写操作的延迟。这在你有许多独立发生的小数据更改时非常重要，比如从 WebSocket 流式传输数据或持久化伪随机发生的事件（如鼠标移动）。 -| Technology | Time in Milliseconds | +| 技术 | 时间（毫秒） | | --- | --- | | Cookies | 0.058 | | LocalStorage | 0.017 | | IndexedDB | 0.17 | -| OPFS Main Thread | 1.46 | +| OPFS 主线程 | 1.46 | | OPFS WebWorker | 1.54 | -| WASM SQLite (memory) | 0.17 | -| WASM SQLite (IndexedDB) | 3.17 | +| WASM SQLite（内存） | 0.17 | +| WASM SQLite（IndexedDB） | 3.17 | -Here we can notice a few things: +这里我们可以注意到几点： -* LocalStorage has the lowest write latency with only 0.017 milliseconds per write. -* IndexedDB writes are about 10 times slower compared to localStorage. -* Sending the data to the WASM SQLite process and letting it persist via IndexedDB is slow with over 3 milliseconds per write. +* LocalStorage 的写延迟最低，仅为 0.017 毫秒每次写操作。 +* IndexedDB 的写操作比 localStorage 慢约 10 倍。 +* 将数据发送到 WASM SQLite 进程并通过 IndexedDB 持久化存储的速度较慢，每次写操作超过 3 毫秒。 -The OPFS operations take about 1.5 milliseconds to write the JSON data into one document per file. We can see the sending the data to a webworker first is a bit slower which comes from the overhead of serializing and deserializing the data on both sides. If we would not create on OPFS file per document but instead append everything to a single file, the performance pattern changes significantly. Then the faster file handle from the `createSyncAccessHandle()` only takes about 1 millisecond per write. But this would require to somehow remember at which position the each document is stored. Therefore in our tests we will continue using one file per document. +OPFS 操作大约需要 1.5 毫秒将 JSON 数据写入每个文档的一个文件。我们可以看到，先将数据发送到 WebWorker 会稍微慢一些，这是由于在两端序列化和反序列化数据的开销。如果我们不为每个文档创建一个 OPFS 文件，而是将所有内容追加到单个文件中，性能模式会发生显著变化。然后，来自 `createSyncAccessHandle()` 的更快文件句柄每次写操作仅需约 1 毫秒。但这需要以某种方式记住每个文档的存储位置。因此，在我们的测试中，我们将继续使用每个文档一个文件的方式。 -### Latency of small Reads +### 少量读操作的延迟 -Now that we have stored some documents, lets measure how long it takes to read single documents by their `id`. +现在我们已经存储了一些文档，让我们测量按 `id` 读取单个文档所需的时间。 -| Technology | Time in Milliseconds | +| 技术 | 时间（毫秒） | | --- | --- | | Cookies | 0.132 | | LocalStorage | 0.0052 | | IndexedDB | 0.1 | -| OPFS Main Thread | 1.28 | +| OPFS 主线程 | 1.28 | | OPFS WebWorker | 1.41 | -| WASM SQLite (memory) | 0.45 | -| WASM SQLite (IndexedDB) | 2.93 | +| WASM SQLite（内存） | 0.45 | +| WASM SQLite（IndexedDB） | 2.93 | -Here we can notice a few things: +这里我们可以注意到几点： -* LocalStorage reads are **really really fast** with only 0.0052 milliseconds per read. -* The other technologies perform reads in a similar speed to their write latency. +* LocalStorage 的读取速度**非常非常快**，仅为 0.0052 毫秒每次读操作。 +* 其他技术的读取速度与其写延迟相似。 -### Big Bulk Writes +### 大批量写操作 -As next step, lets do some big bulk operations with 200 documents at once. +接下来，我们进行一些大批量操作，一次性写入 200 个文档。 -| Technology | Time in Milliseconds | +| 技术 | 时间（毫秒） | | --- | --- | | Cookies | 20.6 | | LocalStorage | 5.79 | | IndexedDB | 13.41 | -| OPFS Main Thread | 280 | +| OPFS 主线程 | 280 | | OPFS WebWorker | 104 | -| WASM SQLite (memory) | 19.1 | -| WASM SQLite (IndexedDB) | 37.12 | +| WASM SQLite（内存） | 19.1 | +| WASM SQLite（IndexedDB） | 37.12 | -Here we can notice a few things: +这里我们可以注意到几点： -* Sending the data to a WebWorker and running it via the faster OPFS API is about twice as fast. -* WASM SQLite performs better on bulk operations compared to its single write latency. This is because sending the data to WASM and backwards is faster if it is done all at once instead of once per document. +* 将数据发送到 WebWorker 并通过更快的 OPFS API 运行的速度大约快两倍。 +* WASM SQLite 在批量操作中的表现优于其单次写延迟。这是因为如果一次性发送数据到 WASM 并返回，而不是每次文档发送一次，速度会更快。 -### Big Bulk Reads +### 大批量读操作 -Now lets read 100 documents in a bulk request. +现在让我们一次性读取 100 个文档。 -| Technology | Time in Milliseconds | +| 技术 | 时间（毫秒） | | --- | --- | | Cookies | 6.34 | | LocalStorage | 0.39 | | IndexedDB | 4.99 | -| OPFS Main Thread | 54.79 | +| OPFS 主线程 | 54.79 | | OPFS WebWorker | 25.61 | -| WASM SQLite (memory) | 3.59 | -| WASM SQLite (IndexedDB) | 5.84 (35ms without cache) | +| WASM SQLite（内存） | 3.59 | +| WASM SQLite（IndexedDB） | 5.84（无缓存时为35毫秒） | -Here we can notice a few things: +这里我们可以注意到几点： -* Reading many files in the OPFS webworker is about **twice as fast** compared to the slower main thread mode. -* WASM SQLite is surprisingly fast. Further inspection has shown that the WASM SQLite process keeps the documents in memory cached which improves the latency when we do reads directly after writes on the same data. When the browser tab is reloaded between the writes and the reads, finding the 100 documents takes about **35 milliseconds** instead. +* 在 OPFS WebWorker 中读取多个文件的速度比主线程模式**快大约两倍**。 +* WASM SQLite 的速度出人意料地快。进一步检查发现，WASM SQLite 进程将文档缓存在内存中，这提高了在写入后直接读取相同数据时的延迟。如果在写入和读取之间重新加载浏览器标签，查找 100 个文档大约需要 **35 毫秒**。 -Performance Conclusions +性能结论 ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- -* LocalStorage is really fast but remember that is has some downsides: - * It blocks the main JavaScript process and therefore should not be used for big bulk operations. - * Only Key-Value assignments are possible, you cannot use it efficiently when you need to do index based range queries on your data. -* OPFS is way faster when used in the WebWorker with the `createSyncAccessHandle()` method compare to using it directly in the main thread. -* SQLite WASM can be fast but the you have to initially download the full binary and start it up which takes about half a second. This might not be relevant at all if your app is started up once and the used for a very long time. But for web-apps that are opened and closed in many browser tabs many times, this might be a problem. +* LocalStorage 确实非常快，但请记住它有一些缺点： + * 它会阻塞主 JavaScript 进程，因此不应用于大型批量操作。 + * 只能进行键值对操作，当你需要对数据进行基于索引的范围查询时，无法高效使用它。 +* OPFS 在 WebWorker 中使用 `createSyncAccessHandle()` 方法时，比直接在主线程中使用要快得多。 +* SQLite WASM 可以很快，但你必须先下载完整的二进制文件并启动它，这大约需要半秒钟。如果你的应用程序启动一次并长时间使用，这可能无关紧要。但对于在多个浏览器标签中多次打开和关闭的 Web 应用程序来说，这可能是一个问题。 * * * -Possible Improvements +可能的改进 ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- -There is a wide range of possible improvements and performance hacks to speed up the operations. +有许多可能的改进和性能优化技巧可以加速操作： -* For IndexedDB I have made a list of [performance hacks here][6]. For example you can do sharding between multiple database and webworkers or use a custom index strategy. -* OPFS is slow in writing one file per document. But you do not have to do that and instead you can store everything at a single file like a normal database would do. This improves performance dramatically like it was done with the RxDB [OPFS RxStorage][17]. -* You can mix up the technologies to optimize for multiple scenarios at once. For example in RxDB there is the [localstorage meta optimizer][48] which stores initial metadata in localstorage and "normal" documents inside of IndexedDB. This improves the initial startup time while still having the documents stored in a way to query them efficiently. -* There is the [memory-mapped][49] storage plugin in RxDB which maps data directly to memory. Using this in combination with a shared worker can improve pageloads and query time significantly. -* [Compressing][50] data before storing it might improve the performance for some of the storages. -* Splitting work up between [multiple WebWorkers][38] via [sharding][51] can improve performance by utilizing the whole capacity of your users device. +* 对于 IndexedDB，我在这里整理了一份[性能优化技巧列表][6]。例如，你可以在多个数据库和 WebWorker 之间进行分片，或者使用自定义的索引策略。 +* OPFS 在每份文档写入一个文件时速度较慢。但你不需要这样做，而是可以像普通数据库那样将所有内容存储在单个文件中。这可以显著提高性能，就像 RxDB 的 [OPFS RxStorage][17] 所做的那样。 +* 你可以混合使用多种技术来优化不同的场景。例如，在 RxDB 中有一个 [localstorage 元数据优化器][48]，它将初始元数据存储在 localStorage 中，而将"普通"文档存储在 IndexedDB 中。这提高了初始启动时间，同时仍然能够高效地查询文档。 +* RxDB 中有一个 [内存映射][49] 存储插件，它直接将数据映射到内存中。与 SharedWorker 结合使用可以显著提高页面加载和查询速度。 +* 在存储数据之前[压缩][50]数据可能会提高某些存储的性能。 +* 通过[分片][51]在[多个 WebWorker][38] 之间分配工作可以利用用户设备的全部性能，从而提高性能。 -Here you can see the [performance comparison][52] of various RxDB storage implementations which gives a better view of real world performance: +在这里你可以看到各种 RxDB 存储实现的[性能比较][52]，这为实际性能提供了更好的视角： -![Image 3: RxStorage performance - browser][3] +![图 3: RxStorage 性能 - 浏览器][3] -Future Improvements +未来的改进 ---------------------------------------------------------------------------------------------------------------------------------------------------------------- -You are reading this in 2024, but the web does not stand still. There is a good chance that browser get enhanced to allow faster and better data operations. +你正在 2024 年阅读这篇文章，但网络并没有停滞不前。浏览器很可能会增强以支持更快、更好的数据操作。 -* Currently there is no way to directly access a persistent storage from inside a WebAssembly process. If this changes in the future, running SQLite (or a similar database) in a browser might be the best option. -* Sending data between the main thread and a WebWorker is slow but might be improved in the future. There is a [good article][53] about why `postMessage()` is slow. -* IndexedDB lately [got support][54] for storage buckets (chrome only) which might improve performance. +* 目前还没有办法直接从 WebAssembly 进程中访问持久存储。如果未来这一点发生变化，在浏览器中运行 SQLite（或类似的数据库）可能是最佳选择。 +* 在主线程和 WebWorker 之间发送数据速度较慢，但未来可能会有所改进。这里有一篇[关于为什么 `postMessage()` 很慢的好文章][53]。 +* IndexedDB 最近[支持][54]了存储桶（仅限 Chrome），这可能会提高性能。 -Follow Up +后续 ---------------------------------------------------------------------------------------------------------------------------------- -* Share my [announcement tweet][55] --\> -* Reproduce the benchmarks at the [github repo][47] -* Learn how to use RxDB with the [RxDB Quickstart][56] -* Check out the [RxDB github repo][57] and leave a star ⭐ +* 分享我的[公告推文][55] --\> +* 在 [GitHub 仓库][47] 中复现基准测试 +* 学习如何使用 RxDB 的 [RxDB 快速入门][56] +* 查看 [RxDB GitHub 仓库][57] 并留下星星 ⭐ [1]: https://rxdb.info/files/logo/rxdb_javascript_database.svg [2]: https://rxdb.info/files/icons/sqlite.svg From e7dff68402d6f72a5e5710916c64ae8bac6c17a6 Mon Sep 17 00:00:00 2001 From: luojiyin Date: Mon, 3 Mar 2025 03:12:52 +0000 Subject: [PATCH 5/5] markdown lint --- ...indexeddb-cookies-opfs-sqlite-wasm.html.md | 112 +++++++++--------- 1 file changed, 56 insertions(+), 56 deletions(-) diff --git a/_drafts/Article/Translation/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html.md b/_drafts/Article/Translation/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html.md index e43fd440..96762d9d 100644 --- a/_drafts/Article/Translation/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html.md +++ b/_drafts/Article/Translation/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html.md @@ -7,136 +7,136 @@ translator: "" reviewer: "" --- -当你正在构建一个Web应用程序时，你可能希望**将数据存储在用户的浏览器中**。也许你只需要存储一些小标志，或者你甚至需要一个完整的数据库。 +当你正在构建一个 Web 应用程序时，你可能希望**将数据存储在用户的浏览器中**。也许你只需要存储一些小标志，或者你甚至需要一个完整的数据库。 -我们构建的Web应用程序类型已经发生了显著变化。在Web的早期，我们提供静态的HTML文件。然后我们提供动态渲染的HTML，后来我们构建了**单页应用程序**，这些应用程序在客户端运行大部分逻辑。而在未来几年，你可能希望构建所谓的[本地优先应用程序][4]，这些应用程序在客户端处理大量复杂的数据操作，甚至在离线时也能工作，这为你提供了构建**零延迟**用户交互的机会。 +我们构建的 Web 应用程序类型已经发生了显著变化。在 Web 的早期，我们提供静态的 HTML 文件。然后我们提供动态渲染的 HTML，后来我们构建了**单页应用程序**，这些应用程序在客户端运行大部分逻辑。而在未来几年，你可能希望构建所谓的[本地优先应用程序][4]，这些应用程序在客户端处理大量复杂的数据操作，甚至在离线时也能工作，这为你提供了构建**零延迟**用户交互的机会。 -在Web的早期，**Cookies**是存储小型键值对的唯一选择。但JavaScript和浏览器已经显著发展，并添加了更好的存储API，为更大、更复杂的数据操作铺平了道路。 +在 Web 的早期，**Cookies**是存储小型键值对的唯一选择。但 JavaScript 和浏览器已经显著发展，并添加了更好的存储 API，为更大、更复杂的数据操作铺平了道路。 -在本文中，我们将深入探讨在浏览器中存储和查询数据的各种技术。我们将探索传统方法，如**Cookies**、**localStorage**、**WebSQL**、**IndexedDB**，以及较新的解决方案，如**OPFS**和**通过WebAssembly的SQLite**。我们将比较这些技术的功能和限制，并通过性能测试揭示在Web应用程序中使用各种方法读写数据的速度。 +在本文中，我们将深入探讨在浏览器中存储和查询数据的各种技术。我们将探索传统方法，如**Cookies**、**localStorage**、**WebSQL**、**IndexedDB**，以及较新的解决方案，如**OPFS**和**通过 WebAssembly 的 SQLite**。我们将比较这些技术的功能和限制，并通过性能测试揭示在 Web 应用程序中使用各种方法读写数据的速度。 注意 -你正在[RxDB][5]文档中阅读本文。RxDB是一个JavaScript数据库，具有不同的存储适配器，可以利用不同的存储API。**自2017年以来**，我大部分时间都在与这些API打交道，进行性能测试，并构建[技巧][6]和插件，以达到浏览器数据库操作速度的极限。 +你正在[RxDB][5]文档中阅读本文。RxDB 是一个 JavaScript 数据库，具有不同的存储适配器，可以利用不同的存储 API。**自 2017 年以来**，我大部分时间都在与这些 API 打交道，进行性能测试，并构建[技巧][6]和插件，以达到浏览器数据库操作速度的极限。 [![Image 1: JavaScript Database][1]][5] -现代浏览器中可用的存储API +现代浏览器中可用的存储 API ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- -首先，让我们简要概述不同的API、它们的预期用例和历史： +首先，让我们简要概述不同的 API、它们的预期用例和历史： -### 什么是Cookies +### 什么是 Cookies -Cookies最早由[网景公司于1994年][7]引入。Cookies存储小型的键值数据，主要用于会话管理、个性化和跟踪。Cookies可以具有多个安全设置，如生存时间或`domain`属性，以在多个子域之间共享Cookies。 +Cookies 最早由[网景公司于 1994 年][7]引入。Cookies 存储小型的键值数据，主要用于会话管理、个性化和跟踪。Cookies 可以具有多个安全设置，如生存时间或`domain`属性，以在多个子域之间共享 Cookies。 -Cookies的值不仅存储在客户端，还会随**每个HTTP请求**发送到服务器。这意味着我们不能在Cookie中存储太多数据，但Cookie的访问性能与其他方法相比仍然值得关注。特别是因为Cookies是Web的重要基础功能，许多性能优化已经完成，甚至在这些日子里，仍然有进展，如Chromium的[共享内存版本控制][8]或异步的[CookieStore API][9]。 +Cookies 的值不仅存储在客户端，还会随**每个 HTTP 请求**发送到服务器。这意味着我们不能在 Cookie 中存储太多数据，但 Cookie 的访问性能与其他方法相比仍然值得关注。特别是因为 Cookies 是 Web 的重要基础功能，许多性能优化已经完成，甚至在这些日子里，仍然有进展，如 Chromium 的[共享内存版本控制][8]或异步的[CookieStore API][9]。 -### 什么是LocalStorage +### 什么是 LocalStorage -[localStorage API][10]最早作为[WebStorage规范的一部分于2009年][11]提出。LocalStorage提供了一个简单的API，用于在Web浏览器中存储键值对。它具有`setItem`、`getItem`、`removeItem`和`clear`方法，这些方法是你从键值存储中所需的一切。LocalStorage仅适用于存储少量需要在会话之间持久化的数据，并且它[受到5MB存储上限的限制][12]。存储复杂数据只能通过将其转换为字符串来实现，例如使用`JSON.stringify()`。该API不是异步的，这意味着它在执行操作时会完全阻塞你的JavaScript进程。因此，在其上运行繁重的操作可能会阻止你的UI渲染。 +[localStorage API][10]最早作为[WebStorage 规范的一部分于 2009 年][11]提出。LocalStorage 提供了一个简单的 API，用于在 Web 浏览器中存储键值对。它具有`setItem`、`getItem`、`removeItem`和`clear`方法，这些方法是你从键值存储中所需的一切。LocalStorage 仅适用于存储少量需要在会话之间持久化的数据，并且它[受到 5MB 存储上限的限制][12]。存储复杂数据只能通过将其转换为字符串来实现，例如使用`JSON.stringify()`。该 API 不是异步的，这意味着它在执行操作时会完全阻塞你的 JavaScript 进程。因此，在其上运行繁重的操作可能会阻止你的 UI 渲染。 -> 还有**SessionStorage** API。关键区别在于localStorage数据会无限期持久化，直到显式清除，而sessionStorage数据在浏览器标签或窗口关闭时会被清除。 +> 还有**SessionStorage** API。关键区别在于 localStorage 数据会无限期持久化，直到显式清除，而 sessionStorage 数据在浏览器标签或窗口关闭时会被清除。 -### 什么是IndexedDB +### 什么是 IndexedDB -IndexedDB最早作为"Indexed Database API"[于2015年][13]引入。 +IndexedDB 最早作为"Indexed Database API"[于 2015 年][13]引入。 -[IndexedDB][14]是一个用于存储大量结构化JSON数据的低级API。虽然该API有点难以使用，但IndexedDB可以利用索引和异步操作。它缺乏对复杂查询的支持，只允许在索引上进行迭代，这使得它更像是其他库的基础层，而不是一个完全成熟的数据库。 +[IndexedDB][14]是一个用于存储大量结构化 JSON 数据的低级 API。虽然该 API 有点难以使用，但 IndexedDB 可以利用索引和异步操作。它缺乏对复杂查询的支持，只允许在索引上进行迭代，这使得它更像是其他库的基础层，而不是一个完全成熟的数据库。 -2018年，IndexedDB 2.0版本[被引入][15]。这增加了一些重大改进。最显著的是`getAll()`方法，它在获取大量JSON文档时显著提高了性能。 +2018 年，IndexedDB 2.0 版本[被引入][15]。这增加了一些重大改进。最显著的是`getAll()`方法，它在获取大量 JSON 文档时显著提高了性能。 -IndexedDB [3.0版本][16]正在开发中，包含了许多改进。最重要的是添加了基于`Promise`的调用，这使得现代JS功能如`async/await`更加有用。 +IndexedDB [3.0 版本][16]正在开发中，包含了许多改进。最重要的是添加了基于`Promise`的调用，这使得现代 JS 功能如`async/await`更加有用。 -### 什么是OPFS +### 什么是 OPFS -[Origin Private File System][17]（OPFS）是一个[相对较新][18]的API，允许Web应用程序直接在浏览器中存储大文件。它专为数据密集型应用程序设计，这些应用程序希望在模拟的文件系统中写入和读取**二进制数据**。 +[Origin Private File System][17]（OPFS）是一个[相对较新][18]的 API，允许 Web 应用程序直接在浏览器中存储大文件。它专为数据密集型应用程序设计，这些应用程序希望在模拟的文件系统中写入和读取**二进制数据**。 -OPFS可以在两种模式下使用： +OPFS 可以在两种模式下使用： * 在[主线程][19]上异步使用 -* 或者在WebWorker中使用更快的异步访问，通过`createSyncAccessHandle()`方法。 +* 或者在 WebWorker 中使用更快的异步访问，通过`createSyncAccessHandle()`方法。 -由于只能处理二进制数据，OPFS被设计为库开发者的基础文件系统。当你构建一个"普通"应用程序时，你不太可能直接使用OPFS，因为它太复杂了。这仅适用于存储纯文件（如图像），而不是高效地存储和查询[JSON数据][20]。我为RxDB构建了一个[基于OPFS的存储][17]，具有适当的索引和查询功能，这花了我几个月的时间。 +由于只能处理二进制数据，OPFS 被设计为库开发者的基础文件系统。当你构建一个"普通"应用程序时，你不太可能直接使用 OPFS，因为它太复杂了。这仅适用于存储纯文件（如图像），而不是高效地存储和查询[JSON 数据][20]。我为 RxDB 构建了一个[基于 OPFS 的存储][17]，具有适当的索引和查询功能，这花了我几个月的时间。 -### 什么是WASM SQLite +### 什么是 WASM SQLite ![Image 2: WASM SQLite][2] -[WebAssembly][21]（Wasm）是一种二进制格式，允许在Web上执行高性能代码。Wasm在2017年被添加到主要浏览器中，这为在浏览器中运行的内容开辟了广泛的机会。你可以将本地库编译为WebAssembly，只需稍作调整即可在客户端运行。WASM代码可以发送到浏览器应用程序，并且通常比JavaScript运行得更快，但仍然比本地代码[慢约10%][22]。 +[WebAssembly][21]（Wasm）是一种二进制格式，允许在 Web 上执行高性能代码。Wasm 在 2017 年被添加到主要浏览器中，这为在浏览器中运行的内容开辟了广泛的机会。你可以将本地库编译为 WebAssembly，只需稍作调整即可在客户端运行。WASM 代码可以发送到浏览器应用程序，并且通常比 JavaScript 运行得更快，但仍然比本地代码[慢约 10%][22]。 -许多人开始在浏览器中使用编译后的SQLite作为数据库，这就是为什么将这种设置与本地API进行比较也是有意义的。 +许多人开始在浏览器中使用编译后的 SQLite 作为数据库，这就是为什么将这种设置与本地 API 进行比较也是有意义的。 -SQLite的编译字节码大小约为[938.9 kB][23]，必须在首次页面加载时由用户下载和解析。WASM不能直接访问浏览器中的任何持久存储API。相反，它需要数据从WASM流向主线程，然后可以放入浏览器API之一。这是通过所谓的[VFS（虚拟文件系统）适配器][24]完成的，这些适配器处理从SQLite到其他任何东西的数据访问。 +SQLite 的编译字节码大小约为[938.9 kB][23]，必须在首次页面加载时由用户下载和解析。WASM 不能直接访问浏览器中的任何持久存储 API。相反，它需要数据从 WASM 流向主线程，然后可以放入浏览器 API 之一。这是通过所谓的[VFS（虚拟文件系统）适配器][24]完成的，这些适配器处理从 SQLite 到其他任何东西的数据访问。 -### 什么是WebSQL +### 什么是 WebSQL -WebSQL**曾经**是一个Web API，[于2009年][25]引入，允许浏览器使用SQL数据库进行客户端存储，基于SQLite。其想法是让开发者能够使用SQL在客户端存储和查询数据，类似于服务器端数据库。WebSQL在近年已**从浏览器中移除**，原因如下： +WebSQL**曾经**是一个 Web API，[于 2009 年][25]引入，允许浏览器使用 SQL 数据库进行客户端存储，基于 SQLite。其想法是让开发者能够使用 SQL 在客户端存储和查询数据，类似于服务器端数据库。WebSQL 在近年已**从浏览器中移除**，原因如下： -* WebSQL没有标准化，并且基于SQLite源代码的单一特定实现的API很难成为标准。 -* WebSQL要求浏览器使用[特定版本][26]的SQLite（版本3.6.19），这意味着每当SQLite有任何更新或错误修复时，如果不破坏Web，就不可能将其添加到WebSQL中。 -* 像Firefox这样的主要浏览器从未支持WebSQL。 +* WebSQL 没有标准化，并且基于 SQLite 源代码的单一特定实现的 API 很难成为标准。 +* WebSQL 要求浏览器使用[特定版本][26]的 SQLite（版本 3.6.19），这意味着每当 SQLite 有任何更新或错误修复时，如果不破坏 Web，就不可能将其添加到 WebSQL 中。 +* 像 Firefox 这样的主要浏览器从未支持 WebSQL。 -因此，在接下来的内容中，我们将**忽略WebSQL**，即使通过设置特定的浏览器标志或使用旧版本的Chromium，仍然可以对其进行测试。 +因此，在接下来的内容中，我们将**忽略 WebSQL**，即使通过设置特定的浏览器标志或使用旧版本的 Chromium，仍然可以对其进行测试。 * * * 功能比较 ------------------------------------------------------------------------------------------------------------------------------------------------------------- -现在你已经了解了这些API的基本概念，让我们比较一些特定的功能，这些功能对于使用RxDB和基于浏览器的存储的人来说非常重要。 +现在你已经了解了这些 API 的基本概念，让我们比较一些特定的功能，这些功能对于使用 RxDB 和基于浏览器的存储的人来说非常重要。 -### 存储复杂的JSON文档 +### 存储复杂的 JSON 文档 -当你在Web应用程序中存储数据时，大多数情况下你希望存储复杂的JSON文档，而不仅仅是存储在服务器端数据库中的`整数`和`字符串`等"普通"值。 +当你在 Web 应用程序中存储数据时，大多数情况下你希望存储复杂的 JSON 文档，而不仅仅是存储在服务器端数据库中的`整数`和`字符串`等"普通"值。 -* 只有IndexedDB原生支持JSON对象。 -* 使用SQLite WASM，你可以[存储JSON][27]在`text`列中，自版本3.38.0（2022-02-22）起，甚至可以在其上运行深度查询，并使用单个属性作为索引。 +* 只有 IndexedDB 原生支持 JSON 对象。 +* 使用 SQLite WASM，你可以[存储 JSON][27]在`text`列中，自版本 3.38.0（2022-02-22）起，甚至可以在其上运行深度查询，并使用单个属性作为索引。 -其他API只能存储字符串或二进制数据。当然，你可以使用`JSON.stringify()`将任何JSON对象转换为字符串，但在API中没有JSON支持会使查询变得复杂，并且多次运行`JSON.stringify()`可能会导致性能问题。 +其他 API 只能存储字符串或二进制数据。当然，你可以使用`JSON.stringify()`将任何 JSON 对象转换为字符串，但在 API 中没有 JSON 支持会使查询变得复杂，并且多次运行`JSON.stringify()`可能会导致性能问题。 ### 多标签支持 -构建Web应用程序与[Electron][28]或[React-Native][29]相比，一个很大的区别是用户会**同时打开和关闭多个浏览器标签**。因此，你不仅有一个JavaScript进程在运行，而且可能存在多个进程，并且它们可能需要在彼此之间共享状态更改，以避免向用户显示**过时的数据**。 +构建 Web 应用程序与[Electron][28]或[React-Native][29]相比，一个很大的区别是用户会**同时打开和关闭多个浏览器标签**。因此，你不仅有一个 JavaScript 进程在运行，而且可能存在多个进程，并且它们可能需要在彼此之间共享状态更改，以避免向用户显示**过时的数据**。 > 如果你的用户的肌肉记忆让他们在使用你的网站时将左手放在**F5**键上，那么你做错了什么！ -并非所有存储API都支持在标签之间自动共享写入事件。 +并非所有存储 API 都支持在标签之间自动共享写入事件。 -只有localStorage有一种通过API本身在标签之间自动共享写入事件的方式，即[storage-event][30]，它可以用于观察更改。 +只有 localStorage 有一种通过 API 本身在标签之间自动共享写入事件的方式，即[storage-event][30]，它可以用于观察更改。 -``` +```plain // localStorage可以通过storage事件观察更改。// 这个功能在IndexedDB和其他API中缺失addEventListener("storage", (event) => {}); ``` -曾经有[实验性的IndexedDB观察者API][31]用于Chrome，但提案仓库已被归档。 +曾经有[实验性的 IndexedDB 观察者 API][31]用于 Chrome，但提案仓库已被归档。 为了解决这个问题，有两种解决方案： -* 第一种选择是使用[BroadcastChannel API][32]，它可以在浏览器标签之间发送消息。因此，每当你向存储写入数据时，你还可以向其他标签发送通知，告知这些更改。这是最常见的解决方法，RxDB也使用了这种方法。注意，还有[WebLocks API][33]，它可以用于在浏览器标签之间实现互斥锁。 -* 另一种解决方案是使用[SharedWorker][34]，并在该worker中执行所有写入操作。所有浏览器标签都可以订阅来自该**单一**SharedWorker的消息，并了解更改。 +* 第一种选择是使用[BroadcastChannel API][32]，它可以在浏览器标签之间发送消息。因此，每当你向存储写入数据时，你还可以向其他标签发送通知，告知这些更改。这是最常见的解决方法，RxDB 也使用了这种方法。注意，还有[WebLocks API][33]，它可以用于在浏览器标签之间实现互斥锁。 +* 另一种解决方案是使用[SharedWorker][34]，并在该 worker 中执行所有写入操作。所有浏览器标签都可以订阅来自该**单一**SharedWorker 的消息，并了解更改。 ### 索引支持 -数据库与将数据存储在普通文件中的最大区别在于，数据库以允许通过索引运行操作的格式写入数据，从而支持快速查询。在我们的技术列表中，只有**IndexedDB**和**WASM SQLite**原生支持索引。理论上，你可以在任何存储（如localStorage或OPFS）上构建索引，但你可能不想自己动手。 +数据库与将数据存储在普通文件中的最大区别在于，数据库以允许通过索引运行操作的格式写入数据，从而支持快速查询。在我们的技术列表中，只有**IndexedDB**和**WASM SQLite**原生支持索引。理论上，你可以在任何存储（如 localStorage 或 OPFS）上构建索引，但你可能不想自己动手。 -例如，在IndexedDB中，我们可以通过给定的索引范围获取一批文档： +例如，在 IndexedDB 中，我们可以通过给定的索引范围获取一批文档： -``` +```plain // 查找所有价格在10到50之间的产品const keyRange = IDBKeyRange.bound(10, 50);const transaction = db.transaction('products', 'readonly');const objectStore = transaction.objectStore('products');const index = objectStore.index('priceIndex');const request = index.getAll(keyRange);const result = await new Promise((res, rej) => { request.onsuccess = (event) => res(event.target.result); request.onerror = (event) => rej(event);}); ``` -注意，IndexedDB有一个限制，即[不支持布尔值的索引][35]。你只能索引字符串和数字。为了解决这个问题，你必须在存储数据时将布尔值转换为数字，并在读取时转换回来。 +注意，IndexedDB 有一个限制，即[不支持布尔值的索引][35]。你只能索引字符串和数字。为了解决这个问题，你必须在存储数据时将布尔值转换为数字，并在读取时转换回来。 -### WebWorker支持 +### WebWorker 支持 -当运行繁重的数据操作时，你可能希望将处理从JavaScript主线程中移出。这确保了我们的应用程序保持响应和快速，而处理可以在后台并行运行。在浏览器中，你可以使用[WebWorker][36]、[SharedWorker][34]或[ServiceWorker][37] API来实现这一点。在RxDB中，你可以使用[WebWorker][38]或[SharedWorker][39]插件将存储移动到worker中。 +当运行繁重的数据操作时，你可能希望将处理从 JavaScript 主线程中移出。这确保了我们的应用程序保持响应和快速，而处理可以在后台并行运行。在浏览器中，你可以使用[WebWorker][36]、[SharedWorker][34]或[ServiceWorker][37] API 来实现这一点。在 RxDB 中，你可以使用[WebWorker][38]或[SharedWorker][39]插件将存储移动到 worker 中。 -最常见的用例是生成一个**WebWorker**，并在第二个JavaScript进程中执行大部分工作。worker是从一个单独的JavaScript文件（或base64字符串）生成的，并通过`postMessage()`与主线程通信。 +最常见的用例是生成一个**WebWorker**，并在第二个 JavaScript 进程中执行大部分工作。worker 是从一个单独的 JavaScript 文件（或 base64 字符串）生成的，并通过`postMessage()`与主线程通信。 -不幸的是，**LocalStorage**和**Cookies**[不能在WebWorker或SharedWorker中使用][40]，这是由于设计和安全限制。WebWorkers在与主浏览器线程分离的全局上下文中运行，因此不能执行可能影响主线程的操作。它们无法直接访问某些Web API，如DOM、localStorage或cookies。 +不幸的是，**LocalStorage**和**Cookies**[不能在 WebWorker 或 SharedWorker 中使用][40]，这是由于设计和安全限制。WebWorkers 在与主浏览器线程分离的全局上下文中运行，因此不能执行可能影响主线程的操作。它们无法直接访问某些 Web API，如 DOM、localStorage 或 cookies。 -其他所有内容都可以在WebWorker中使用。OPFS的快速版本`createSyncAccessHandle`方法**只能**[在WebWorker中使用][41]，而**不能在主线程中使用**。这是因为返回的`AccessHandle`的所有操作都是**非异步的**，因此会阻塞JavaScript进程，所以你肯定不希望在主线程中执行这些操作并阻塞一切。 +其他所有内容都可以在 WebWorker 中使用。OPFS 的快速版本`createSyncAccessHandle`方法**只能**[在 WebWorker 中使用][41]，而**不能在主线程中使用**。这是因为返回的`AccessHandle`的所有操作都是**非异步的**，因此会阻塞 JavaScript 进程，所以你肯定不希望在主线程中执行这些操作并阻塞一切。 * * * @@ -258,7 +258,7 @@ OPFS 操作大约需要 1.5 毫秒将 JSON 数据写入每个文档的一个文 | OPFS 主线程 | 54.79 | | OPFS WebWorker | 25.61 | | WASM SQLite（内存） | 3.59 | -| WASM SQLite（IndexedDB） | 5.84（无缓存时为35毫秒） | +| WASM SQLite（IndexedDB） | 5.84（无缓存时为 35 毫秒） | 这里我们可以注意到几点：