Evaluate redb as an alternative to rocksdb

[msk]

redb is a pure Rust implementation of a key-value database, which could bring several advantages to our project:

• Since redb is written entirely in Rust, it eliminates the long compile times and large disk space usage associated with rocksdb due to its C++ dependencies.
• Benchmark results show that redb outperforms RocksDB in most operations, except for bulk writing. Given that bulk writing isn't a critical operation for review-database, this trade-off might be acceptable.

I also found another crate called native_db that builds on top of redb. It offers features like migration, which could simplify our database management.

However, it’s important to note that neither redb nor native_db provides built-in backup and restore features. We will need to manually implement and test these features to ensure they meet our performance and reliability requirements.

Next Steps:

• Conduct a detailed evaluation of redb's performance in the context of review-database, focusing on its integration with our current workflows.
• Implement backup and restore functionality and evaluate its performance and reliability.
• Test the migration features provided by native_db and assess their utility.
• If the evaluation is positive, create a plan for transitioning from rocksdb to redb, including potential challenges and strategies for mitigation.

Update (on 2024-08-29):

• Added a note about the lack of built-in backup and restore features in redb and native_db.
• Included a task to manually implement and test backup and restore functionality as part of the evaluation process.

[msk]

Let's move forward with this evaluation by creating a dedicated branch for the transition to redb.

Proposed Plan:

1. Create a redb Branch:

   • We will create a new branch named redb, which will serve as the base for all changes related to the migration from rocksdb to redb.
   • The main branch will continue to incorporate any necessary updates to review-database, while the redb branch will focus on the migration effort.

2. Gradual Migration with native_db redb:

   • On the redb branch, we will start by gradually adding the code required to migrate data from rocksdb to redb.
   • We will leverage the native_db crate to facilitate this process, particularly focusing on its migration features.
   • Our approach will include implementing the necessary backup and restore functionalities, which redb currently lacks.

3. Testing and Performance Evaluation:

   • We will perform thorough testing using actual data sets of a reasonable size to evaluate performance and ensure that the migration does not introduce any regressions.
   • Testing will cover both the performance of redb in our typical workloads and the reliability of the custom backup and restore processes.

4. PR Management and Rebase Strategy:

   • Any pull requests (PRs) related to the redb migration will be targeted to the redb branch.
   • To keep the redb branch up-to-date with the latest changes in main, we will periodically rebase redb on top of main.

5. Final Transition:

   • Once the migration to redb is complete and the stability of the system has been thoroughly verified, we will merge the redb branch back into main.
   • This will be followed by the release of a new version of review-database that fully replaces rocksdb with redb.