memory-mapped faiss indices

[alexanderguzhva]

Summary

A prototype of memory-mapped files that contain Faiss Indices (IndexFlatCodes and HNSW at this moment).

zilliztech/knowhere#996

@mdouze Could you please give your comments on the approach, just in case I've missed some important corner cases? Despite the link is from the Faiss fork, it should be pretty straightforward to understand the changes. Basically, I replace std::vector<> with a custom container that supports mmap-ping.

The PR for Faiss will be implemented as soon as we agree on the details.
Thanks.

[mdouze]

Yes that seems very good.
I started a similar effort but never got a chance to finalize it, see . So please go ahead if you want to commit it.
For ref, this is shipit-facebookresearch-faiss-D56685773.patch what I was working on.

[navneet1v]

@alexanderguzhva this is very promising if we can replace the std::vector<> with the container. I think beyond mmap it has a lot of usecases. are you working on it? If not I can pick this up.

@mdouze what are your thoughts on how we should move forward in the contribution.

[alexanderguzhva]

@navneet1v I will publish a PR next week, I'm waiting for some extra things to be verified, such as using data from some 3-rd party preallocated memory area as well

[navneet1v]

@navneet1v I will publish a PR next week, I'm waiting for some extra things to be verified, such as using data from some 3-rd party preallocated memory area as well

thank you for the update. Excited to see this feature coming out.

[alexanderguzhva]

#4199