Read vs Write Memory Cache Eviction

[victorstewart]

i assume reads and writes share equal priority in evicting items from the in memory cache?

an interesting knob to consider might be tuning that priority... either globally or preferably per operation (which would then associate that property # the key).

imagine a scenario where you have 3 global datacenters and replicate all writes generated in one to the others. this provides resiliency / safety, but the data access patterns happens to be, data read or written is 5-10x more likely to be reaccessed within the same datacenter, rather than another. aka the data has high geographical relevance, with clients connected to a datacenter mostly transacting with each other's data.

so now those foreign writes arrive, evicting locally relevant data, and might consume 2/3 of your local instance's in memory cache, increasing drive accesses by 3x.

[smoyerx]

Writes are cached in heap before being written O_DIRECT to media. Reads are cached in the page cache. So there is no direct impact of writes on read caching, except for general DRAM pressure.

This is actually a bit of an oversimplification, but it's the right mental model to work with.

It is a goal in upcoming releases to reduce overall memory consumption to the degree practical, which will definitely help in scenarios like you describe.

[victorstewart]

oh fantastic.

are there any plans to explore io_uring usage? especially with registered buffers for O_DIRECT, avoiding kernel page pinning.

[smoyerx]

Yes, io_uring is on the roadmap. We could use io_uring for new data writes (ingest) and maintenance reads and writes (compaction). We plan to continue using mmap for (most) normal query reads (gets or cursors).

Note that io_uring is not yet supported in all major distros.