One writer, many readers #111
-
Hello! Try to use that library for kv store in FastAPI app.
Reader:
So, when I change the values, readers don't see the changes without re-opening the database.
Read process:
What am I doing wrong? |
Beta Was this translation helpful? Give feedback.
Replies: 4 comments
-
Hi, I found the following from rocksdb doc: RocksDB database can be opened in read-write mode (aka. Primary Instance) or can be opened in read-only mode. RocksDB supports two variations of read-only mode: Read-only Instance - Opens the database in read-only mode. When the Read-only instance is created, it gets a static read-only view of the Primary Instance’s database contents https://github.com/facebook/rocksdb/wiki/Read-only-and-Secondary-instances So, the read only mode gets a static view. You should use the secondary instance. |
Beta Was this translation helpful? Give feedback.
-
Example from rocksdb doc: const std::string kDbPath = "/tmp/rocksdbtest";
...
// Assume we have already opened a regular RocksDB instance db_primary
// whose database directory is kDbPath.
assert(db_primary);
Options options;
options.max_open_files = -1;
// Secondary instance needs its own directory to store info logs (LOG)
const std::string kSecondaryPath = "/tmp/rocksdb_secondary/";
DB* db_secondary = nullptr;
Status s = DB::OpenAsSecondary(options, kDbPath, kSecondaryPath, &db_secondary);
assert(!s.ok() || db_secondary);
// Let secondary **try** to catch up with primary
s = db_secondary->TryCatchUpWithPrimary();
assert(s.ok());
// Read operations
std::string value;
s = db_secondary->Get(ReadOptions(), "foo", &value);
... for Python code use opt = Options(raw_mode=True)
opt.set_max_open_files(-1)
db = Rdict(db_path, options=opt, access_type=AccessType.Secondary(some_path_for_secondary_log))
# need to catch up at some frequency manually by calling:
db.try_catch_up_with_primary() |
Beta Was this translation helpful? Give feedback.
-
Yah, thanks you!
and try to enable wal, or try to undestand why not working good |
Beta Was this translation helpful? Give feedback.
-
So, for rocksdb, part of the data is written to those SST files, and part of them are not flushed (stored in memtable), but those data are in the WAL log. So, since the secondary or the readonly instance is another process, it does not have access to the memtable of the primary instance (which is in memory). So, it loads its view from SST files + WAL files. If WAL is not enabled, the view can be more partial than if it is enabled. |
Beta Was this translation helpful? Give feedback.
Hi, I found the following from rocksdb doc:
RocksDB database can be opened in read-write mode (aka. Primary Instance) or can be opened in read-only mode. RocksDB supports two variations of read-only mode:
Read-only Instance - Opens the database in read-only mode. When the Read-only instance is created, it gets a static read-only view of the Primary Instance’s database contents
Secondary Instance – Opens the database in read-only mode. Supports extra ability to dynamically catch-up with the Primary instance (through a manual call by the user – based on their delay/frequency requirements)
https://github.com/facebook/rocksdb/wiki/Read-only-and-Secondary-instances
So, the read only mode gets …