How to set max_open_files to avoid Too many open files?

[sevenjay]

We have exceeded the maximum number of open files, 65535, and have temporarily increased the system to 1048576 . In rocksdb, it can be set the max_open_files to limit it.
In Qdrant where to set the max_open_files or how to limit the number of opening files?

[timvisee]

In Qdrant where to set the max_open_files or how to limit the number of opening files?

For Qdrant, it is currently not possible to limit the number of open files.

If you want to limit this number and are using a lot of collections, the right approach is very likely to move everything into a single collection. Please see our recommendations at: https://qdrant.tech/documentation/guides/multiple-partitions/

[sevenjay]

We only have 1262 collections, but more than 77428 files have been opened. Is this normal?
thanks.

[timvisee]

Yes, that can be expected.

1262 collections

That is a lot. Please see the recommendations in: https://qdrant.tech/documentation/guides/multiple-partitions/

[sevenjay]

I understand. that actually answered my question. However, there are some more that I would like to discuss under this topic.

Pinecone has three ways to multitenancy: https://docs.pinecone.io/docs/multitenancy

Namespaces: A Pinecone-specific way
Metadata filtering: This should be similar to the link recommendation method you provided
Separate indexes: for different pods, this should be the same as using different collections

Will Qdrant offer a similar Namespaces way in the future?

[timvisee]

Will Qdrant offer a similar Namespaces way in the future?

Yes. In fact, we've released it about one hour ago! 🎉

We call it user-defined sharding (or custom sharding). You can read more about it here: https://qdrant.tech/documentation/guides/distributed_deployment/#user-defined-sharding

In fact, user-defined sharding is more featureful, because unlike Pinecone namespaces, Qdrant supports operations over multiple user-defined shards together.