Skip to content

Commit

Permalink
Merge pull request #879 from qdrant/q-and-a-update
Browse files Browse the repository at this point in the history
FAQ Update
  • Loading branch information
davidmyriel authored Jul 18, 2024
2 parents ad7b380 + 41222be commit c62a1b0
Show file tree
Hide file tree
Showing 2 changed files with 68 additions and 24 deletions.
Original file line number Diff line number Diff line change
@@ -1,28 +1,28 @@
---
title: Database Optimization
weight: 3
weight: 2
---

## Database Optimization Strategies
# Frequently Asked Questions: Database Optimization

### How do I reduce memory usage?

The primary source of memory usage vector data. There are several ways to address that:
The primary source of memory usage is vector data. There are several ways to address that:

- Configure [Quantization](../../guides/quantization/) to reduce the memory usage of vectors.
- Configure on-disk vector storage

The choice of the approach depends on your requirements.
Read more about [configuring the optimal](../../tutorials/optimize/) use of Qdrant.
The choice of the approach depends on your requirements.
Read more about [configuring the optimal](../../tutorials/optimize/) use of Qdrant.

### How do you choose machine configuration?
### How do you choose the machine configuration?

There are two main scenarios of Qdrant usage in terms of resource consumption:

- **Performance-optimized** -- when you need to serve vector search as fast (many) as possible. In this case, you need to have as much vector data in RAM as possible. Use our [calculator](https://cloud.qdrant.io/calculator) to estimate the required RAM.
- **Storage-optimized** -- when you need to store many vectors and minimize costs by compromising some search speed. In this case, pay attention to the disk speed instead. More about it in the article about [Memory Consumption](../../../articles/memory-consumption/).

### I configured on-disk vector storage, but memory usage is still high. Why?
### I configured on-disk vector storage, but memory usage is still high. Why?

Firstly, memory usage metrics as reported by `top` or `htop` may be misleading. They are not showing the minimal amount of memory required to run the service.
If the RSS memory usage is 10 GB, it doesn't mean that it won't work on a machine with 8 GB of RAM.
Expand All @@ -34,11 +34,10 @@ As a result, the Qdrant process might use more memory than the minimum required
If you want to limit the memory usage of the service, we recommend using [limits in Docker](https://docs.docker.com/config/containers/resource_constraints/#memory) or Kubernetes.


### My requests are very slow or time out. What should I do?

There are several possible reasons for that:

- **Using filters without payload index** -- If you're performing a search with a filter but you don't have a payload index, Qdrant will have to load whole payload data from disk to check the filtering condition. Ensure you have adequately configured [payload indexes](../../concepts/indexing/#payload-index).
- **Usage of on-disk vector storage with slow disks** -- If you're using on-disk vector storage, ensure you have fast enough disks. We recommend using local SSDs with at least 50k IOPS. Read more about the influence of the disk speed on the search latency in the article about [Memory Consumption](../../../articles/memory-consumption/).
- **Large limit or non-optimal query parameters** -- A large limit or offset might lead to significant performance degradation. Please pay close attention to the query/collection parameters that significantly diverge from the defaults. They might be the reason for the performance issues.
- **Large limit or non-optimal query parameters** -- A large limit or offset might lead to significant performance degradation. Please pay close attention to the query/collection parameters that significantly diverge from the defaults. They might be the reason for the performance issues.
75 changes: 60 additions & 15 deletions qdrant-landing/content/documentation/faq/qdrant-fundamentals.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,48 @@
---
title: Fundamentals
title: Qdrant Fundamentals
weight: 1
---

## Qdrant Fundamentals
# Frequently Asked Questions: General Topics
||||||
|-|-|-|-|-|
|[Vectors](/documentation/faq/qdrant-fundamentals/#vectors)|[Search](/documentation/faq/qdrant-fundamentals/#search)|[Collections](/documentation/faq/qdrant-fundamentals/#collections)|[Compatibility](/documentation/faq/qdrant-fundamentals/#compatibility)|[Cloud](/documentation/faq/qdrant-fundamentals/#cloud)|

### How many collections can I create?
## Vectors

As much as you want, but be aware that each collection requires additional resources.
It is _highly_ recommended not to create many small collections, as it will lead to significant resource consumption overhead.
### What is the maximum vector dimension supported by Qdrant?

We consider creating a collection for each user/dialog/document as an antipattern.
Qdrant supports up to 65,535 dimensions by default, but this can be configured to support higher dimensions.

Please read more about collections, isolation, and multiple users in our [Multitenancy](../../tutorials/multiple-partitions/) tutorial.
### What is the maximum size of vector metadata that can be stored?

There is no inherent limitation on metadata size, but it should be [optimized for performance and resource usage](/documentation/guides/optimize/). Users can set upper limits in the configuration.

### Can the same similarity search query yield different results on different machines?

Yes, due to differences in hardware configurations and parallel processing, results may vary slightly.

### What to do with documents with small chunks using a fixed chunk strategy?

For documents with small chunks, consider merging chunks or using variable chunk sizes to optimize vector representation and search performance.

### How do I choose the right vector embeddings for my use case?

This depends on the nature of your data and the specific application. Consider factors like dimensionality, domain-specific models, and the performance characteristics of different embeddings.

### How does Qdrant handle different vector embeddings from various providers in the same collection?

Qdrant natively [supports multiple vectors per data point](/documentation/concepts/vectors/#multivectors), allowing different embeddings from various providers to coexist within the same collection.

### Can I migrate my embeddings from another vector store to Qdrant?

Yes, Qdrant supports migration of embeddings from other vector stores, facilitating easy transitions and adoption of Qdrant’s features.

## Search

### How does Qdrant handle real-time data updates and search?

Qdrant supports live updates for vector data, with newly inserted, updated and deleted vectors available for immediate search. The system uses full-scan search on unindexed segments during background index updates.

### My search results contain vectors with null values. Why?

Expand Down Expand Up @@ -48,6 +78,17 @@ What Qdrant doesn't plan to support:
Of course, you can always combine Qdrant with any specialized tool you need, including full-text search engines.
Read more about [our approach](../../../articles/hybrid-search/) to hybrid search.

## Collections

### How many collections can I create?

As many as you want, but be aware that each collection requires additional resources.
It is _highly_ recommended not to create many small collections, as it will lead to significant resource consumption overhead.

We consider creating a collection for each user/dialog/document as an antipattern.

Please read more about collections, isolation, and multiple users in our [Multitenancy](../../tutorials/multiple-partitions/) tutorial.

### How do I upload a large number of vectors into a Qdrant collection?

Read about our recommendations in the [bulk upload](../../tutorials/bulk-upload/) tutorial.
Expand All @@ -56,14 +97,16 @@ Read about our recommendations in the [bulk upload](../../tutorials/bulk-upload/

No, Qdrant requires full precision vectors for operations like reindexing, rescoring, etc.

## Qdrant Cloud
## Compatibility

### Is it possible to scale down a Qdrant Cloud cluster?
### Is Qdrant compatible with CPUs or GPUs for vector computation?

In general, no. There's no way to scale down the underlying disk storage.
But in some cases, we might be able to help you with that through manual intervention, but it's not guaranteed.
Qdrant primarily relies on CPU acceleration for scalability and efficiency, with no current support for GPU acceleration.

### Do you guarantee compatibility across versions?

## Versioning
In case your version is older, we only guarantee compatibility between two consecutive minor versions. This also applies to client versions. Ensure your client version is never more than one minor version away from your cluster version.
While we will assist with break/fix troubleshooting of issues and errors specific to our products, Qdrant is not accountable for reviewing, writing (or rewriting), or debugging custom code.

### Do you support downgrades?

Expand All @@ -74,7 +117,9 @@ data is automatically migrated to the newer storage format. This migration is no

We only guarantee compatibility if you update between consecutive versions. You would need to upgrade versions one at a time: `1.1 -> 1.2`, then `1.2 -> 1.3`, then `1.3 -> 1.4`.

### Do you guarantee compatibility across versions?
## Cloud

In case your version is older, we only guarantee compatibility between two consecutive minor versions. This also applies to client versions. Ensure your client version is never more than one minor version away from your cluster version.
While we will assist with break/fix troubleshooting of issues and errors specific to our products, Qdrant is not accountable for reviewing, writing (or rewriting), or debugging custom code.
### Is it possible to scale down a Qdrant Cloud cluster?

In general, no. There's no way to scale down the underlying disk storage.
But in some cases, we might be able to help you with that through manual intervention, but it's not guaranteed.

0 comments on commit c62a1b0

Please sign in to comment.