Projects Being Developed

The page lists major projects being actively developed, or has been planned for future development.

RocksDB on Remote Storage

API between RocksDB and underlying storage

We recently completed a major refactoring of the rocksdb::Env class by separating the storage related interfaces into a class of its own, called rocksdb::FileSystem. In the long-term, the storage interfaces in Env will be deprecated and the main purpose of Env will be to abstract core OS functionality that RocksDB needs. The relevant PRs are https://github.com/facebook/rocksdb/pull/5761 and https://github.com/facebook/rocksdb/pull/6552.

Over time, we will implement new functionality enabled by this separation -

1. Richer error handling - A compliant FileSystem implementation can return information about an IO error, such as whether its transient/retryable, permanent data-loss, file scope or entire file system etc. in IOStatus, which will allow RocksDB to do more intelligent error handling.
2. Fail fast - For file systems that allow callers to provide a timeout for an IO, RocksDB can provide provide better SLAs for user reads by providing an option to specify a deadline, and failing a Get/MultiGet as soon as the deadline is exceeded. This is an ongoing project.

User Defined Timestamps

BlobDB

BlobDB is RocksDB's implementation of key-value separation, originally inspired by the WiscKey paper. Large values (blobs) are stored in separate blob files, and only references to them are stored in RocksDB's LSM tree. By separating value storage from the LSM tree, BlobDB provides an alternative way of reducing write amplification, instead of tuning compactions. BlobDB is used in production at Facebook.

Does this belong here? What's in development?

File Checksums

See https://github.com/facebook/rocksdb/wiki/Full-File-Checksum#the-next-step

Per Key/Value Checksum

Encryption at Rest

MultiGet()

See MultiGet Performance.

Bloom Filter Improvements

First phase complete, including

• Fixed flaws in old Bloom implementation for Block-based table with a new implementation (incl Issue 4120 and 5857), enabled using format_version=5. More detail on wiki and in the main PR.
• Allow non-integer bits / key settings for finer granularity in control
• Expose details like LSM level to custom FilterPolicy, for experimentation

Planned:

• Minimize memory internal fragmentation on generated filters (https://github.com/facebook/rocksdb/pull/6427)
• Investigate use of different bits/key for different levels (as in Monkey)
• Investigate use of alternative data structures, most likely based on perfect hashing static functions. See Xor filter, modified with "fuse graph" construction. Or even sgauss. We don't expect much difference in query times, but the primary trade-off to be between construction time and memory footprint for a given false positive rate. It's likely that L0 will continue to construct Bloom filters (fast memtable flushes) while compaction will spend more time to generate more compact structures.
• Re-vamp how filters are configured (based on above developments), probably moving away from bits/key as a proxy for accuracy.

Improving Testing

Adaptive Compaction

Improving RocksDB Backups

Including https://github.com/facebook/rocksdb/issues/6521

User-defined timestamp

https://docs.google.com/document/d/1FcDjOM8-pJzCajCa9waQkox6DKJIWZAHm36buK4wfqs/edit#heading=h.uxub5284i1ti