How to limit RAM usage

[webfrank]

Hi, I've come to KVRocks from Redis as I need to limit RAM usage on a Job Queue process. The main process will create thousands of keys with some JSON inside, once the job is done it will update the key, not much job and no latency issue, only memory allocation. On redis I reach 16GB of RAM which cannot be handle on the current environment. I would like to lower this figure to 1/2GB mantaining the data and this is why I reached kvrocks.

I exchanged redis for kvrocks and it worked as expected (apart from a corruption error for which I've opened an issue), now I would like to limit ram usage, I set --rocksdb.block_cache_size 128 on command line but the ram is increasing. Which flags I need to set to limit it? (128 is just a test to limit with very low usage). Now I'm using 2.10.1 as per the corruption issue with 2.11.0

[PragmaTwice]

You can refer to here: #2726

[webfrank]

Hi, I've followed the post but it seems it is not working, I setup 128MB of cache and ram went up to 480MB. I made a second test limiting docker ram to 256MB and it stopped there so it seems it uses RAM up to the limit.

[PragmaTwice]

Could you explain a bit about "128MB cache" here? Is it a sum of block cache and write buffer?

[webfrank]

I tried to set --rocksdb.block_cache_size 128 but without limiting docker image doesn't not limi ram usage.

The only way I found to limit memory usage is to limit container memory and nothing else.

[git-hulk]

@webfrank You could reduce the following configurations as well:

• rocksdb.write_buffer_size default value is 64MiB.
• rocksdb.max_write_buffer_number default value is 4.

You could reduce the write_buffer_size to 16MiB and max_write_buffer_numbner to 1 if you write traffic is low.

[git-hulk]

@webfrank Does this work well for you after reducing those two parameters?

[webfrank]

Hi, the ram reduced but not so much as expected, it continues to increase up to container limits.

Now I'm trying with 256MB block cache and 4x16MB write cache and a container limit of 512MB and after several batches seems stable around 480MB