From 0dd3c0381d59739b67dec4b4fa33e7bb8b6a4cb5 Mon Sep 17 00:00:00 2001 From: daveatweaviate Date: Thu, 26 Sep 2024 09:52:47 -0400 Subject: [PATCH 1/5] indexing --- developers/wcs/platform/manage-costs.md | 30 +++++++++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100644 developers/wcs/platform/manage-costs.md diff --git a/developers/wcs/platform/manage-costs.md b/developers/wcs/platform/manage-costs.md new file mode 100644 index 0000000000..8144586a06 --- /dev/null +++ b/developers/wcs/platform/manage-costs.md @@ -0,0 +1,30 @@ +--- +title: Manage resource costs +sidebar_position: 20 +image: og/wcs/user_guides.jpg +--- + +Weaviate Cloud provides flexible resource management features that let you balance system performance and system resource costs. This page discusses configuration strategies to help you to get the most value from Weaviate Cloud. + +## Indexes + +Weaviate is a vector database. Individual vectors can have thousands of dimensions. Most objects in Weaviate collections have one or more vectors. Collections can have millions of objects. Weaviate uses vector indexes to search collections efficiently. A lot of space is required to store the indexes, and the underlying vectors. + +Disk based storage is relatively cheap. RAM memory is expensive. Loading an index and its vectors into RAM costs more than storing them on disk, but there are other considerations too. Besides storage costs, speed and accuracy are perhaps the most prominent considerations. + + +The default vector index in Weaviate Cloud is a [Hierarchical Navigable Small World (HNSW) index](/developers/weaviate/concepts/vector-index#hierarchical-navigable-small-world-hnsw-index). + +## Compression + +## Tenant states + +## Support + +import SupportAndTrouble from '/_includes/wcs/support-and-troubleshoot.mdx'; + + + +import CustomScriptLoader from '/src/components/scriptSwitch'; + + From b99743eac3d76d57858534779c7bf7b5bf8fde8f Mon Sep 17 00:00:00 2001 From: daveatweaviate Date: Thu, 26 Sep 2024 15:37:13 -0400 Subject: [PATCH 2/5] indexes --- developers/wcs/platform/manage-costs.md | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/developers/wcs/platform/manage-costs.md b/developers/wcs/platform/manage-costs.md index 8144586a06..290bf3cb24 100644 --- a/developers/wcs/platform/manage-costs.md +++ b/developers/wcs/platform/manage-costs.md @@ -8,12 +8,23 @@ Weaviate Cloud provides flexible resource management features that let you balan ## Indexes -Weaviate is a vector database. Individual vectors can have thousands of dimensions. Most objects in Weaviate collections have one or more vectors. Collections can have millions of objects. Weaviate uses vector indexes to search collections efficiently. A lot of space is required to store the indexes, and the underlying vectors. +Weaviate is a vector database. Its purpose is to manage collections of vectors, potentially very large collections of vectors. Most objects in Weaviate collections have one or more vectors; collections can have millions of objects; individual vectors can have thousands of dimensions. This adds up to a lot of storage. -Disk based storage is relatively cheap. RAM memory is expensive. Loading an index and its vectors into RAM costs more than storing them on disk, but there are other considerations too. Besides storage costs, speed and accuracy are perhaps the most prominent considerations. +Weaviate uses vector indexes to search collections efficiently. Indexes dramatically improve search, but they also increase storage requirements. +The default vector index in Weaviate Cloud is a [Hierarchical Navigable Small World (HNSW) index](/developers/weaviate/concepts/vector-index#hierarchical-navigable-small-world-hnsw-index). HNSW is an in memory index. The index is held in RAM memory along with the underlying vectors in the collection. The memory costs for a large collection can add up quickly. -The default vector index in Weaviate Cloud is a [Hierarchical Navigable Small World (HNSW) index](/developers/weaviate/concepts/vector-index#hierarchical-navigable-small-world-hnsw-index). +### Index alternatives + +Weaviate offers these index alternatives: + +- [HNSW indexes](/developers/weaviate/starter-guides/managing-resources/indexing#hnsw-indexes). HNSW indexes are in-memory indexes that enable fast searching even with very large data sets. +- [Flat indexes](/developers/weaviate/starter-guides/managing-resources/indexing#flat-indexes). Flat indexes are disk-based indexes that work best with small data sets. +- [Dynamic indexes](/developers/weaviate/starter-guides/managing-resources/indexing#dynamic-indexes). Dynamic indexes are a configuration option that converts a flat index to an HNSW index at a threshold object count. + +Weaviate Cloud costs are based on usage. More expensive resources, like RAM, incur more cost than cheaper resources, like disk or offline storage. If you don't need the precision and [control of an HNSW index](/developers/weaviate/config-refs/schema/vector-index#hnsw-indexes), consider using a flat index to shift resource costs. + +If your application is multi-tenanted, consider [configuring a dynamic index](/developers/weaviate/manage-data/collections#set-vector-index-type) for your tenants. Smaller tenant indexes are flat, and stored on disk. Larger tenants are stored in RAM as HNSW indexes, but the overall cost is lower. ## Compression From a00fa4f1dcaa8a1595577583f8022d82f0697c6c Mon Sep 17 00:00:00 2001 From: daveatweaviate Date: Thu, 26 Sep 2024 17:34:18 -0400 Subject: [PATCH 3/5] review --- developers/wcs/platform/manage-costs.md | 31 +++++++++++++++++++++++-- 1 file changed, 29 insertions(+), 2 deletions(-) diff --git a/developers/wcs/platform/manage-costs.md b/developers/wcs/platform/manage-costs.md index 290bf3cb24..97b43a504c 100644 --- a/developers/wcs/platform/manage-costs.md +++ b/developers/wcs/platform/manage-costs.md @@ -14,9 +14,9 @@ Weaviate uses vector indexes to search collections efficiently. Indexes dramatic The default vector index in Weaviate Cloud is a [Hierarchical Navigable Small World (HNSW) index](/developers/weaviate/concepts/vector-index#hierarchical-navigable-small-world-hnsw-index). HNSW is an in memory index. The index is held in RAM memory along with the underlying vectors in the collection. The memory costs for a large collection can add up quickly. -### Index alternatives +### Vector index alternatives -Weaviate offers these index alternatives: +Weaviate offers these vector index alternatives: - [HNSW indexes](/developers/weaviate/starter-guides/managing-resources/indexing#hnsw-indexes). HNSW indexes are in-memory indexes that enable fast searching even with very large data sets. - [Flat indexes](/developers/weaviate/starter-guides/managing-resources/indexing#flat-indexes). Flat indexes are disk-based indexes that work best with small data sets. @@ -26,10 +26,37 @@ Weaviate Cloud costs are based on usage. More expensive resources, like RAM, inc If your application is multi-tenanted, consider [configuring a dynamic index](/developers/weaviate/manage-data/collections#set-vector-index-type) for your tenants. Smaller tenant indexes are flat, and stored on disk. Larger tenants are stored in RAM as HNSW indexes, but the overall cost is lower. +### Inverted indexes + +Weaviate uses inverted indexes for filtering and keyword searches. Compared to vector indexes, inverted indexes aren't a major component of Cloud resource costs. However, inverted indexes do use some resources. [Turn off](/developers/weaviate/manage-data/collections#property-level-settings) unused indexes to save disk space. + +- If you never search or filter on a property, turn off `invertedFilterable` and `invertedSearchable` indexing. +- If you only filter on a property occasionally, turn off `invertedFilterable` indexing. The property is still filterable, just less efficiently. + ## Compression +Compression is the most effective way to reduce costs in Weaviate Cloud. Quantization algorithms are methods that make different trade-offs to reduce the size of vectors. Weaviate supports these methods: + +- [PQ](/developers/weaviate/configuration/compression/pq-compression) is the most configurable method. It gives you the most control to adjust compression rates and recall. PQ compressed vectors typically use significantly less memory than uncompressed vectors. +- [SQ](/developers/weaviate/configuration/compression/sq-compression) uses a bucketing approach to normalize data. SQ compressed vectors use 75% less memory than uncompressed vectors. +- [BQ](/developers/weaviate/configuration/compression/bq-compression) reduces vector dimension float values to bits. BQ compressed vectors use 97% less memory than uncompressed vectors. + +These compression algorithms have different functional tradeoffs, but they all help to control costs the same way. They reduce the size of the vectors so the indexes are smaller. Smaller indexes need less hosting resources so you spend less money. + +HNSW indexes store vectors in RAM and on disk. Compressed indexes use much less RAM when they are loaded into memory. Weaviate stores the uncompressed vector and the compressed vector index on disk, so the cost of disk storage goes up slightly. However, since the cost of RAM is orders of magnitude higher than the cost of disk, the overall cost to use a compressed index is much lower than the cost of using an uncompressed index. + +Flat indexes use an [in-memory cache](/developers/weaviate/config-refs/schema/vector-index#flat-indexes) to improve response time for common queries. Enable BQ to shrink the size of the objects in the cache and lower RAM usage. + +### Price calculator + +The Weaviate Cloud [pricing calculator](https://weaviate.io/pricing) demonstrates the savings with compressed indexes. + +To use the calculator, enter the number of objects in your collection and the dimensionality of each object. Under `Storage Type`, select "Compression". Compare the costs for "Performance" and for "Compression." + ## Tenant states +Multi-tenant applications often have active and inactive tenants. Inactive tenants continue to use system resources. If you are an enterprise customer, it is possible to offload inactive tenants to low-cost, Amazon S3 storage. For pricing and configuration details, [contact support](mailto:support@weaviate.io). + ## Support import SupportAndTrouble from '/_includes/wcs/support-and-troubleshoot.mdx'; From 17eb65aefeb995d1a7eb99ef493683898cd3d4f7 Mon Sep 17 00:00:00 2001 From: daveatweaviate Date: Thu, 26 Sep 2024 17:52:43 -0400 Subject: [PATCH 4/5] ha --- developers/wcs/platform/manage-costs.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/developers/wcs/platform/manage-costs.md b/developers/wcs/platform/manage-costs.md index 97b43a504c..e9fb7aa102 100644 --- a/developers/wcs/platform/manage-costs.md +++ b/developers/wcs/platform/manage-costs.md @@ -57,6 +57,14 @@ To use the calculator, enter the number of objects in your collection and the di Multi-tenant applications often have active and inactive tenants. Inactive tenants continue to use system resources. If you are an enterprise customer, it is possible to offload inactive tenants to low-cost, Amazon S3 storage. For pricing and configuration details, [contact support](mailto:support@weaviate.io). +## High availability + +Production systems have stringent uptime and resiliency requirements. [High availability (HA)](/developers/weaviate/concepts/replication-architecture/motivation#high-availability-redundancy) clusters help you meet these requirements. + +In Weaviate Cloud highly available clusters are three times as expensive as standalone clusters. The extra cost is due to replication. HA clusters use more resources so they cost correspondingly more. + +To lower the cost of an HA cluster, choose an appropriate index for your workload and enable compression. An HA system with compression enabled is about 1/3 the price of a similar standalone system that doesn't use compression. + ## Support import SupportAndTrouble from '/_includes/wcs/support-and-troubleshoot.mdx'; From c4ede727b93afe041b4301dfad7dd2539b11026a Mon Sep 17 00:00:00 2001 From: daveatweaviate Date: Thu, 26 Sep 2024 18:19:33 -0400 Subject: [PATCH 5/5] staging --- developers/wcs/platform/manage-costs.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/developers/wcs/platform/manage-costs.md b/developers/wcs/platform/manage-costs.md index e9fb7aa102..a1e3765bb5 100644 --- a/developers/wcs/platform/manage-costs.md +++ b/developers/wcs/platform/manage-costs.md @@ -16,22 +16,24 @@ The default vector index in Weaviate Cloud is a [Hierarchical Navigable Small Wo ### Vector index alternatives -Weaviate offers these vector index alternatives: +Weaviate Cloud costs are based on usage. More expensive resources, like RAM, incur more cost than cheaper resources such as disk or offline storage. Weaviate offers vector index types that can shift resource usage between RAM and disk: - [HNSW indexes](/developers/weaviate/starter-guides/managing-resources/indexing#hnsw-indexes). HNSW indexes are in-memory indexes that enable fast searching even with very large data sets. - [Flat indexes](/developers/weaviate/starter-guides/managing-resources/indexing#flat-indexes). Flat indexes are disk-based indexes that work best with small data sets. - [Dynamic indexes](/developers/weaviate/starter-guides/managing-resources/indexing#dynamic-indexes). Dynamic indexes are a configuration option that converts a flat index to an HNSW index at a threshold object count. -Weaviate Cloud costs are based on usage. More expensive resources, like RAM, incur more cost than cheaper resources, like disk or offline storage. If you don't need the precision and [control of an HNSW index](/developers/weaviate/config-refs/schema/vector-index#hnsw-indexes), consider using a flat index to shift resource costs. +If you don't need the precision and [control of an HNSW index](/developers/weaviate/config-refs/schema/vector-index#hnsw-indexes), consider using a flat index to shift resource costs. -If your application is multi-tenanted, consider [configuring a dynamic index](/developers/weaviate/manage-data/collections#set-vector-index-type) for your tenants. Smaller tenant indexes are flat, and stored on disk. Larger tenants are stored in RAM as HNSW indexes, but the overall cost is lower. +If might need an HNSW index and your application is multi-tenanted, consider [configuring a dynamic index](/developers/weaviate/manage-data/collections#set-vector-index-type) for your tenants. + +When a dynamic index is configured, smaller tenants use flat indexes that are stored on disk. Larger tenants use HNSW indexes that are stored in RAM. Even though some tenants use expensive RAM, the overall costs are lower because some of the tenants are able to use cheaper resources. ### Inverted indexes Weaviate uses inverted indexes for filtering and keyword searches. Compared to vector indexes, inverted indexes aren't a major component of Cloud resource costs. However, inverted indexes do use some resources. [Turn off](/developers/weaviate/manage-data/collections#property-level-settings) unused indexes to save disk space. -- If you never search or filter on a property, turn off `invertedFilterable` and `invertedSearchable` indexing. -- If you only filter on a property occasionally, turn off `invertedFilterable` indexing. The property is still filterable, just less efficiently. +- If you never search or filter on a property, turn off `invertedFilterable` and `invertedSearchable` indexing. Properties are not searchable if `invertedSearchable` is not enabled. +- If you only filter on a property occasionally, turn off `invertedFilterable` indexing. Properties are still filterable if `invertedFilterable` is not enabled. The filter is just less efficient. ## Compression