why RocksDict is not async?

[elrik75]

Hi!
Nowadays DB clients have an async version: asyncpg, aiokafka, etc...
Is there a reason for RocksDict to not be async? Is this just a design choice or there is another reason? Maybe it makes no sense?
With python 3.13 --disable-gil we can now have a multi-core process and locking 8 CPU to read RocksDB is obviously a bad idea. I guess it can be solve by using a ThreadPool but a native async client could be great.

[Congyuwang]

Like for async APIs you hardly know when something concretely happens. But with rocksdb, you are pretty sure that when a write returns, it actually is done. Also rocksddict is not async because rocksdb is not. It is an embedded db in the sense that you use its binary code directly. Not as Kafka or other DBs, which communicates through http requests.

I think the reason for this basically is that KV db is designed with a different purpose in mind. It is more like the storage engine for DBs.

[elrik75]

Even if there are no network IO, there are Disk IO.
This unmaintained lib is an async client for rocksDB: https://github.com/AYMENJD/rocksdb-python
The idea is: while rocksdb is writing on disk, you unlock the process to execute something else.

[Congyuwang]

I'm not sure about python3.13 yet. But for python 3.12 and prior versions, we actually let go of the GIL for some batch operations, and also operations like compactions. But for normal write and read, and especially iterators, because those are usually fast enough, the result for these operations, when lettings go of the GIL, is usually worse due to lock contention. That's because multiple python threads (<= py3.12) are competing for the GIL, and results in performance loss.

Maybe python 3.13 would be different after disable Gil. I'll certainly look into it when I got familiar with python 3.13. But welcome contribution and testing though.

[elrik75]

Do you think it makes sense for me to test a ThreadPoolExecutor on RocksDict and bench performances in a async context? I'll give you the results.

[Congyuwang]

I think it is good that you can proceed with the test. When we have python 3.13 support, we can then have a baseline for comparison. Thanks.

[elrik75]

Hi @Congyuwang. It appears that RocksDict is so fast that using async code is not necessary. In my test (python 3.12), I run a 4 CPUs process pull with a few computing operations and store the result in RocksDB. I test to use an async Executor or the vanilla RocksDict. There is no case where I can be faster in async code.