The next release of Nebari will fundamentally change what Nebari is. Today, Nebari uses one-file-per-tree to store data in an append-only format. The Roots type uses an additional file to store a transaction log to coordinate transactions.

In May 2022, I discovered fundamental flaws in my benchmarking which meant the assumptions I made about the performance of how transactions work were incorrect. No matter what, to get Nebari's speed faster, I needed to change the format to allow the tree and transaction files to be fsynced in parallel.

Nebari's append-only format has its own downsides, including the need to occasionally compact to reclaim disk space. Additionally, incorporating a write-ahead log can improve transaction speed significantly, especially when multiple trees are involved.

After a long exploration phase last year, Sediment and OkayWAL were created. Nebari v0.6 will be built atop Sediment.

Sediment is not an append-only format, and it does not offer a single-file storage mechanism. For some users of Nebari, this will be an unwelcome change. However, I feel like it moves this library forward, and if I were to fork this old format to its own library, I wouldn't be maintaining it moving forward. If you have any concerns about your use case, please leave a comment and I can try to address them.