Skip to content

Commit

Permalink
Update wording of resizing now that it is handled internally, and don…
Browse files Browse the repository at this point in the history
…'t bother warning about write maps that aren't used
  • Loading branch information
kriszyp committed May 16, 2021
1 parent fc0d16b commit 9f24a1e
Show file tree
Hide file tree
Showing 3 changed files with 2 additions and 9 deletions.
4 changes: 1 addition & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ LMDB supports multiple modes of transactions, including disabling of file sync'i

`lmdb-store` supports and encourages the use of conditional writes; this allows for atomic operations that are dependendent on previously read data, and most transactional types of operations can be written with an optimistic-locking based, atomic-conditional-write pattern. This allows `lmdb-store` to scale to handle concurrent execution across many processes or threads while maintaining data integrity.

When an `lmdb-store` is created, an LMDB environment/database is created, and starts with a default DB size of 256KB. LMDB itself uses a fixed size, but `lmdb-store` detects whenever the database goes beyond the current size, and automatically increases the size of DB, and re-executes the write operations after resizing. With this, you do not have to make any estimates of database size, the databases automatically grow as needed (as you would expect from a database!)
When an `lmdb-store` is created, an LMDB environment/database is created, and starts with a default DB size of 128KB. `lmdb-store` detects whenever the database goes beyond the current size, and automatically grows the size of database as needed (as you would expect from a database!)

`lmdb-store` provides optional compression using LZ4 that works in conjunction with the asynchronous writes by performing the compression in the same thread (off the main thread) that performs the writes in a transaction. LZ4 is extremely fast, and decompression can be performed at roughly 5GB/s, so excellent storage efficiency can be achieved with almost negligible performance impact.

Expand Down Expand Up @@ -106,8 +106,6 @@ This will run the provided callback in a transaction, asynchronously starting th

The callback function will be queued along with other `put` and `remove` operations, and run in the same transaction as other operations that have been queued in the current event turn, and will be executed in the order they were called. `transactionAsync` will return a promise that will resolve once its transaction has been committed. The promise will resolve to the value returned by the callback function.

It also important to remember that the transaction callback function may be executed more than once if the transaction needs to be restarted due to resizing the database. Also, any of the writes (`put` or `remove`) operations may throw a `MDB_MAP_FULL` error during the transaction. This is a normal part of the resizing process, and the transaction callback function should allow these errors to propagate (either don't catch the error, or rethrow it), and lmdb-store will automatically handle resizing and restarting the transaction and callbacks.

For example:
```
let products = open(...);
Expand Down
5 changes: 0 additions & 5 deletions index.js
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,6 @@ const ITERATOR_DONE = { done: true, value: undefined }
let env
let defaultCompression
let lastSize
let writeMapWarned
exports.open = open
exports.ABORT = ABORT
function open(path, options) {
Expand Down Expand Up @@ -89,10 +88,6 @@ function open(path, options) {
console.info('Removed', path)
}
let useWritemap = options.useWritemap
if (useWritemap && !writeMapWarned) {
console.warn('Using useWritemap flag is deprecated due to numerous complications and limitations')
writeMapWarned = true
}
try {
env.open(options)
} catch(error) {
Expand Down
2 changes: 1 addition & 1 deletion src/env.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -520,7 +520,7 @@ NAN_METHOD(EnvWrap::open) {
// Set MDB_NOTLS to enable multiple read-only transactions on the same thread (in this case, the nodejs main thread)
flags |= MDB_NOTLS;
#ifdef _WIN32
if ((flags & MDB_WRITEMAP) && !(flags & MDB_NOSYNC)) {
if ((flags & MDB_WRITEMAP) && !(flags & MDB_NOSYNC) && !(flags & MDB_REMAP_CHUNKS)) {
fprintf(stderr, "Writemaps are currently disabled on Windows doing to issues with syncing\n");
flags &= ~MDB_WRITEMAP;
}
Expand Down

0 comments on commit 9f24a1e

Please sign in to comment.