It is possible cache the bloom filter to NVMe? (support filters larger than 2^38 bytes)

[hskun]

libfilter is very fast.
I have about 2^36 items to add in bf, it is possible cache the bloom filter to NVMe?

[jbapple]

You could use mmap in the C interface to allocate a libfilter_region's block pointer, I suppose. If you just accessed the NVMe space needed on each read or write, I suppose the OS might cache it anyway.

Thoughts?

[hskun]

You could use mmap in the C interface to allocate a libfilter_region's block pointer, I suppose. If you just accessed the NVMe space needed on each read or write, I suppose the OS might cache it anyway.

Thoughts?

I will have a try.
Thank you!

[jbapple]

Looking more carefully at the code, I don't think it supports filters larger than 2^32 bytes. I'll take this as a feature request.

[jbapple]

When I run the following, the correct amount of memory is allocated on my machine:

BlockFilter::CreateWithNdvFpp(1ul << 34, 0.0001)

How much memory is allocated on your machine if you just call this factory method and then return from main? What tool are you using to measure the amount of memory allocated?