diff --git a/articles/cosmos-db/nosql/how-to-dotnet-vector-index-query.md b/articles/cosmos-db/nosql/how-to-dotnet-vector-index-query.md index bc2faaeced..93e1b5ec17 100644 --- a/articles/cosmos-db/nosql/how-to-dotnet-vector-index-query.md +++ b/articles/cosmos-db/nosql/how-to-dotnet-vector-index-query.md @@ -7,27 +7,29 @@ ms.author: jacodel ms.service: azure-cosmos-db ms.subservice: nosql ms.topic: how-to -ms.date: 08/01/2023 +ms.date: 12/03/2024 ms.custom: query-reference, devx-track-dotnet, build-2024, ignite-2024 ms.collection: - ce-skilling-ai-copilot +appliesto: + - ✅ NoSQL --- # Index and query vectors in Azure Cosmos DB for NoSQL in .NET -[!INCLUDE[NoSQL](../includes/appliesto-nosql.md)] Before you use vector indexing and search, you must first enable the feature. This article covers the following steps: 1. Enabling the Vector Search in Azure Cosmos DB for NoSQL feature. -2. Setting up the Azure Cosmos DB container for vector search -3. Authoring vector embedding policy -4. Adding vector indexes to the container indexing policy -5. Creating a container with vector indexes and vector embedding policy -6. Performing a vector search on the stored data +1. Setting up the Azure Cosmos DB container for vector search +1. Authoring vector embedding policy +1. Adding vector indexes to the container indexing policy +1. Creating a container with vector indexes and vector embedding policy +1. Performing a vector search on the stored data This guide walks through the process of creating vector data, indexing the data, and then querying the data in a container. ## Prerequisites + - An existing Azure Cosmos DB for NoSQL account. - If you don't have an Azure subscription, [Try Azure Cosmos DB for NoSQL free](https://cosmos.azure.com/try/). - If you have an existing Azure subscription, [create a new Azure Cosmos DB for NoSQL account](how-to-create-account.md). @@ -38,10 +40,10 @@ This guide walks through the process of creating vector data, indexing the data, Vector search for Azure Cosmos DB for NoSQL requires enabling the feature by completing the following steps: 1. Navigate to your Azure Cosmos DB for NoSQL resource page. -2. Select the "Features" pane under the "Settings" menu item. -3. Select for "Vector Search in Azure Cosmos DB for NoSQL." -4. Read the description of the feature to confirm you want to enable it. -5. Select "Enable" to turn on vector search in Azure Cosmos DB for NoSQL. +1. Select the "Features" pane under the "Settings" menu item. +1. Select for "Vector Search in Azure Cosmos DB for NoSQL." +1. Read the description of the feature to confirm you want to enable it. +1. Select "Enable" to turn on vector search in Azure Cosmos DB for NoSQL. > [!TIP] > Alternatively, use the Azure CLI to update the capabilities of your account to support NoSQL vector search. @@ -53,20 +55,18 @@ Vector search for Azure Cosmos DB for NoSQL requires enabling the feature by com > --capabilities EnableNoSQLVectorSearch > ``` - > [!NOTE] > The registration request will be autoapproved; however, it may take 15 minutes to take effect. -## Understanding the steps involved in vector search - +## Understanding the steps involved in vector search -Let’s take an example of creating a database for an internet-based bookstore and you're storing Title, Author, ISBN, and Description for each book. We also define two properties to contain vector embeddings. The first is the “contentVector” property, which contains [text embeddings](/azure/ai-services/openai/concepts/models#embeddings ) generated from the text content of the book (for example, concatenating the “title” “author” “isbn” and “description” properties before creating the embedding). The second is “coverImageVector”, which is generated from [images of the book’s cover](/azure/ai-services/computer-vision/concept-image-retrieval). +Let’s take an example of creating a database for an internet-based bookstore and you're storing Title, Author, ISBN, and Description for each book. We also define two properties to contain vector embeddings. The first is the “contentVector” property, which contains [text embeddings](/azure/ai-services/openai/concepts/models#embeddings ) generated from the text content of the book (for example, concatenating the “title” “author” “isbn” and “description” properties before creating the embedding). The second is “coverImageVector”, which is generated from [images of the book’s cover](/azure/ai-services/computer-vision/concept-image-retrieval). -1. Create and store vector embeddings for the fields on which you want to perform vector search. -2. Specify the vector embedding paths in the vector embedding policy. -3. Include any desired vector indexes in the indexing policy for the container. +1. Create and store vector embeddings for the fields on which you want to perform vector search. +1. Specify the vector embedding paths in the vector embedding policy. +1. Include any desired vector indexes in the indexing policy for the container. -For subsequent sections of this article, we consider the below structure for the items stored in our container: +For subsequent sections of this article, we consider the below structure for the items stored in our container: ```json { @@ -79,18 +79,19 @@ For subsequent sections of this article, we consider the below structure for the } ``` -## Creating a vector embedding policy for your container. +## Creating a vector embedding policy for your container + Next, you need to define a container vector policy. This policy provides information that is used to inform the Azure Cosmos DB query engine how to handle vector properties in the VectorDistance system functions. This policy also informs the vector indexing policy of necessary information, should you choose to specify one. The following information is included in the contained vector policy: - * “path”: The property path that contains vectors  - * “datatype”: The type of the elements of the vector (default Float32)  - * “dimensions”: The length of each vector in the path (default 1536)  - * “distanceFunction”: The metric used to compute distance/similarity (default Cosine)  +- “path”: The property path that contains vectors  +- “datatype”: The type of the elements of the vector (default Float32)  +- “dimensions”: The length of each vector in the path (default 1536)  +- “distanceFunction”: The metric used to compute distance/similarity (default Cosine)  -For our example with book details, the vector policy can look like the example JSON below: +For our example with book details, the vector policy can look like the example JSON below: -```csharp +```csharp Database db = await client.CreateDatabaseIfNotExistsAsync("vector-benchmarking"); List embeddings = new List() { @@ -109,13 +110,13 @@ For our example with book details, the vector policy can look like the example J Dimensions = 10, } }; -``` +``` +## Creating a vector index in the indexing policy -## Creating a vector index in the indexing policy -Once the vector embedding paths are decided, vector indexes need to be added to the indexing policy. Currently, the vector search feature for Azure Cosmos DB for NoSQL is supported only on new containers so you need to apply the vector policy during the time of container creation and it can’t be modified later. For this example, the indexing policy would look something like this: +Once the vector embedding paths are decided, vector indexes need to be added to the indexing policy. Currently, the vector search feature for Azure Cosmos DB for NoSQL is supported only on new containers so you need to apply the vector policy during the time of container creation and it can’t be modified later. For this example, the indexing policy would look something like this: -```csharp +```csharp Collection collection = new Collection(embeddings); ContainerProperties properties = new ContainerProperties(id: "vector-container", partitionKeyPath: "/id") { @@ -134,13 +135,14 @@ Once the vector embedding paths are decided, vector indexes need to be added to }; properties.IndexingPolicy.IncludedPaths.Add(new IncludedPath { Path = "/*" }); properties.IndexingPolicy.ExcludedPaths.Add(new ExcludedPath { Path = "/vector/*" }); -``` ->[!IMPORTANT] +``` + +> [!IMPORTANT] > The vector path added to the "excludedPaths" section of the indexing policy to ensure optimized performance for insertion. Not adding the vector path to "excludedPaths" will result in higher RU charge and latency for vector insertions. -## Running vector similarity search query +## Running vector similarity search query -Once you create a container with the desired vector policy, and insert vector data into the container, you can conduct a vector search using the [Vector Distance](query/vectordistance.md) system function in a query. Suppose you want to search for books about food recipes by looking at the description, you first need to get the embeddings for your query text. In this case, you might want to generate embeddings for the query text – “food recipe”. Once you have the embedding for your search query, you can use it in the VectorDistance function in the vector search query and get all the items that are similar to your query as shown here: +Once you create a container with the desired vector policy, and insert vector data into the container, you can conduct a vector search using the [Vector Distance](query/vectordistance.md) system function in a query. Suppose you want to search for books about food recipes by looking at the description, you first need to get the embeddings for your query text. In this case, you might want to generate embeddings for the query text – “food recipe”. Once you have the embedding for your search query, you can use it in the VectorDistance function in the vector search query and get all the items that are similar to your query as shown here: ```sql SELECT TOP 10 c.title, VectorDistance(c.contentVector, [1,2,3,4,5,6,7,8,9,10]) AS SimilarityScore   @@ -150,7 +152,7 @@ ORDER BY VectorDistance(c.contentVector, [1,2,3,4,5,6,7,8,9,10])   This query retrieves the book titles along with similarity scores with respect to your query. Here's an example in .NET: -```csharp +```csharp float[] embedding = {1f,2f,3f,4f,5f,6f,7f,8f,9f,10f}; var queryDef = new QueryDefinition( query: $"SELECT c.title, VectorDistance(c.contentVector,@embedding) AS SimilarityScore FROM c ORDER BY VectorDistance(c.contentVector,@embedding)" @@ -166,9 +168,10 @@ This query retrieves the book titles along with similarity scores with respect t Console.WriteLine($"Found item:\t{item}"); } } -``` +``` + +## Related content -## Next steps - [VectorDistance system function](query/vectordistance.md) - [Vector indexing](../index-policy.md) - [Setup Azure Cosmos DB for NoSQL for vector search](../vector-search.md). diff --git a/articles/cosmos-db/nosql/how-to-java-vector-index-query.md b/articles/cosmos-db/nosql/how-to-java-vector-index-query.md index 79d83c7e5d..de951a5973 100644 --- a/articles/cosmos-db/nosql/how-to-java-vector-index-query.md +++ b/articles/cosmos-db/nosql/how-to-java-vector-index-query.md @@ -7,39 +7,43 @@ ms.author: jacodel ms.service: azure-cosmos-db ms.subservice: nosql ms.topic: how-to -ms.date: 08/01/2023 +ms.date: 12/03/2024 ms.custom: query-reference, devx-track-java, build-2024, devx-track-extended-java, ignite-2024 ms.collection: - ce-skilling-ai-copilot +appliesto: + - ✅ NoSQL --- -# Index and query vectors in Azure Cosmos DB for NoSQL in Java +# Index and query vectors in Azure Cosmos DB for NoSQL in Java Before you use vector indexing and search, you must first enable the feature. This article covers the following steps: 1. Enabling the Vector Search in Azure Cosmos DB for NoSQL feature -2. Setting up the Azure Cosmos DB container for vector search -3. Authoring vector embedding policy -4. Adding vector indexes to the container indexing policy -5. Creating a container with vector indexes and vector embedding policy -6. Performing a vector search on the stored data +1. Setting up the Azure Cosmos DB container for vector search +1. Authoring vector embedding policy +1. Adding vector indexes to the container indexing policy +1. Creating a container with vector indexes and vector embedding policy +1. Performing a vector search on the stored data This guide walks through the process of creating vector data, indexing the data, and then querying the data in a container. ## Prerequisites + - An existing Azure Cosmos DB for NoSQL account. - If you don't have an Azure subscription, [Try Azure Cosmos DB for NoSQL free](https://cosmos.azure.com/try/). - If you have an existing Azure subscription, [create a new Azure Cosmos DB for NoSQL account](how-to-create-account.md). - Latest version of the Azure Cosmos DB [Java](sdk-java-v4.md) SDK. ## Enable the feature + Vector search for Azure Cosmos DB for NoSQL requires feature enablement. Follow the below steps to register: 1. Navigate to your Azure Cosmos DB for NoSQL resource page. -2. Select the "Features" pane under the "Settings" menu item. -3. Select for "Vector Search in Azure Cosmos DB for NoSQL." -4. Read the description of the feature to confirm you want to enable it. -5. Select "Enable" to turn on vector search in Azure Cosmos DB for NoSQL. +1. Select the "Features" pane under the "Settings" menu item. +1. Select for "Vector Search in Azure Cosmos DB for NoSQL." +1. Read the description of the feature to confirm you want to enable it. +1. Select "Enable" to turn on vector search in Azure Cosmos DB for NoSQL. > [!TIP] > Alternatively, use the Azure CLI to update the capabilities of your account to support NoSQL vector search. @@ -54,42 +58,44 @@ Vector search for Azure Cosmos DB for NoSQL requires feature enablement. Follow > [!NOTE] > The registration request will be autoapproved; however, it may take 15 minutes to take effect. -## Understanding the steps involved in vector search +## Understanding the steps involved in vector search -The following steps assume that you know how to [setup a Cosmos DB NoSQL account and create a database](quickstart-portal.md). The vector search feature is currently not supported on the existing containers, so you need to create a new container and specify the container-level vector embedding policy, and the vector indexing policy at the time of container creation. +The following steps assume that you know how to [setup a Cosmos DB NoSQL account and create a database](quickstart-portal.md). The vector search feature is currently not supported on the existing containers, so you need to create a new container and specify the container-level vector embedding policy, and the vector indexing policy at the time of container creation. -Let’s take an example of creating a database for an internet-based bookstore and you're storing Title, Author, ISBN, and Description for each book. We also define two properties to contain vector embeddings. The first is the “contentVector” property, which contains [text embeddings](/azure/ai-services/openai/concepts/models#embeddings ) generated from the text content of the book (for example, concatenating the “title” “author” “isbn” and “description” properties before creating the embedding). The second is “coverImageVector”, which is generated from [images of the book’s cover](/azure/ai-services/computer-vision/concept-image-retrieval). +Let’s take an example of creating a database for an internet-based bookstore and you're storing Title, Author, ISBN, and Description for each book. We also define two properties to contain vector embeddings. The first is the “contentVector” property, which contains [text embeddings](/azure/ai-services/openai/concepts/models#embeddings ) generated from the text content of the book (for example, concatenating the “title” “author” “isbn” and “description” properties before creating the embedding). The second is “coverImageVector”, which is generated from [images of the book’s cover](/azure/ai-services/computer-vision/concept-image-retrieval). -1. Create and store vector embeddings for the fields on which you want to perform vector search. -2. Specify the vector embedding paths in the vector embedding policy. -3. Include any desired vector indexes in the indexing policy for the container. +1. Create and store vector embeddings for the fields on which you want to perform vector search. +2. Specify the vector embedding paths in the vector embedding policy. +3. Include any desired vector indexes in the indexing policy for the container. -For subsequent sections of this article, we consider the below structure for the items stored in our container: +For subsequent sections of this article, we consider the below structure for the items stored in our container: ```json { -"title": "book-title", -"author": "book-author", -"isbn": "book-isbn", -"description": "book-description", -"contentVector": [2, -1, 4, 3, 5, -2, 5, -7, 3, 1], -"coverImageVector": [0.33, -0.52, 0.45, -0.67, 0.89, -0.34, 0.86, -0.78] + "title": "book-title", + "author": "book-author", + "isbn": "book-isbn", + "description": "book-description", + "contentVector": [2, -1, 4, 3, 5, -2, 5, -7, 3, 1], + "coverImageVector": [0.33, -0.52, 0.45, -0.67, 0.89, -0.34, 0.86, -0.78] } ``` First, create the `CosmosContainerProperties` object. + ```java CosmosContainerProperties collectionDefinition = new CosmosContainerProperties(UUID.randomUUID().toString(), "Partition_Key_Def"); ``` ## Creating a vector embedding policy for your container + Next, you need to define a container vector policy. This policy provides information that is used to inform the Azure Cosmos DB query engine how to handle vector properties in the VectorDistance system functions. This also informs the vector indexing policy of necessary information, should you choose to specify one. The following information is included in the contained vector policy: - * “path”: The property path that contains vectors  - * “datatype”: The type of the elements of the vector (default Float32)  - * “dimensions”: The length of each vector in the path (default 1536)  - * “distanceFunction”: The metric used to compute distance/similarity (default Cosine)  +- “path”: The property path that contains vectors  +- “datatype”: The type of the elements of the vector (default Float32)  +- “dimensions”: The length of each vector in the path (default 1536)  +- “distanceFunction”: The metric used to compute distance/similarity (default Cosine)  For our example with book details, the vector policy may look like the example JSON: @@ -114,11 +120,11 @@ cosmosVectorEmbeddingPolicy.setCosmosVectorEmbeddings(Arrays.asList(embedding1, collectionDefinition.setVectorEmbeddingPolicy(cosmosVectorEmbeddingPolicy); ``` +## Creating a vector index in the indexing policy -## Creating a vector index in the indexing policy -Once the vector embedding paths are decided, vector indexes need to be added to the indexing policy. Currently, the vector search feature for Azure Cosmos DB for NoSQL is supported only on new containers so you need to apply the vector policy during the time of container creation and it can’t be modified later. For this example, the indexing policy would look something like below: +Once the vector embedding paths are decided, vector indexes need to be added to the indexing policy. Currently, the vector search feature for Azure Cosmos DB for NoSQL is supported only on new containers so you need to apply the vector policy during the time of container creation and it can’t be modified later. For this example, the indexing policy would look something like below: -```java +```java IndexingPolicy indexingPolicy = new IndexingPolicy(); indexingPolicy.setIndexingMode(IndexingMode.CONSISTENT); ExcludedPath excludedPath1 = new ExcludedPath("/coverImageVector/*"); @@ -140,22 +146,20 @@ cosmosVectorIndexSpec2.setType(CosmosVectorIndexType.DISK_ANN.toString()); indexingPolicy.setVectorIndexes(Arrays.asList(cosmosVectorIndexSpec1, cosmosVectorIndexSpec2, cosmosVectorIndexSpec3)); collectionDefinition.setIndexingPolicy(indexingPolicy); -``` +``` Finally, create the container with the container index policy and the vector index policy. -```java +```java database.createContainer(collectionDefinition).block(); ``` - - ->[!IMPORTANT] +> [!IMPORTANT] > The vector path added to the "excludedPaths" section of the indexing policy to ensure optimized performance for insertion. Not adding the vector path to "excludedPaths" will result in higher RU charge and latency for vector insertions. -## Running vector similarity search query +## Running vector similarity search query -Once you have created a container with the desired vector policy, and inserted vector data into the container, you can conduct a vector search using the [Vector Distance](query/vectordistance.md) system function in a query. Suppose you want to search for books about food recipes by looking at the description, you first need to get the embeddings for your query text. In this case, you might want to generate embeddings for the query text – “food recipe”. Once you have the embedding for your search query, you can use it in the VectorDistance function in the vector search query and get all the items that are similar to your query as shown here: +Once you have created a container with the desired vector policy, and inserted vector data into the container, you can conduct a vector search using the [Vector Distance](query/vectordistance.md) system function in a query. Suppose you want to search for books about food recipes by looking at the description, you first need to get the embeddings for your query text. In this case, you might want to generate embeddings for the query text – “food recipe”. Once you have the embedding for your search query, you can use it in the VectorDistance function in the vector search query and get all the items that are similar to your query as shown here: ```sql SELECT TOP 10 c.title, VectorDistance(c.contentVector, [1,2,3,4,5,6,7,8,9,10]) AS SimilarityScore   @@ -181,7 +185,8 @@ ArrayList paramList = new ArrayList(); } ``` -## Next steps +## Related content + - [VectorDistance system function](query/vectordistance.md) - [Vector indexing](../index-policy.md) - [Setup Azure Cosmos DB for NoSQL for vector search](../vector-search.md). diff --git a/articles/cosmos-db/nosql/how-to-javascript-vector-index-query.md b/articles/cosmos-db/nosql/how-to-javascript-vector-index-query.md index c5d9731e4e..9d5c37bfff 100644 --- a/articles/cosmos-db/nosql/how-to-javascript-vector-index-query.md +++ b/articles/cosmos-db/nosql/how-to-javascript-vector-index-query.md @@ -7,29 +7,24 @@ ms.author: jacodel ms.service: azure-cosmos-db ms.subservice: nosql ms.topic: how-to -ms.date: 08/08/2024 +ms.date: 12/03/2024 ms.custom: query-reference, build-2024, devx-track-js, ignite-2024 ms.collection: - ce-skilling-ai-copilot +appliesto: + - ✅ NoSQL --- # Index and query vectors in Azure Cosmos DB for NoSQL in JavaScript -[!INCLUDE[NoSQL](../includes/appliesto-nosql.md)] - Before you use Vector Indexing and Search, you must first enable the feature. This article covers the following steps: 1. Enabling the Vector Search in Azure Cosmos DB for NoSQL feature - -2. Setting up the Azure Cosmos DB container for vector search - -3. Authoring vector embedding policy - -4. Adding vector indexes to the container indexing policy - -5. Creating a container with vector indexes and vector embedding policy - -6. Performing a vector search on the stored data +1. Setting up the Azure Cosmos DB container for vector search +1. Authoring vector embedding policy +1. Adding vector indexes to the container indexing policy +1. Creating a container with vector indexes and vector embedding policy +1. Performing a vector search on the stored data This guide walks through the process of creating vector data, indexing the data, and then querying the data in a container. @@ -45,24 +40,20 @@ This guide walks through the process of creating vector data, indexing the data, Vector search for Azure Cosmos DB for NoSQL requires enabling the feature by completing the following steps: 1. Navigate to your Azure Cosmos DB for NoSQL resource page. - 1. Select the "Features" pane under the "Settings" menu item. - 1. Select for "Vector Search in Azure Cosmos DB for NoSQL." - -2. Read the description of the feature to confirm you want to enable it. - -3. Select "Enable" to turn on vector search in Azure Cosmos DB for NoSQL. - -> [!TIP] -> Alternatively, use the Azure CLI to update the capabilities of your account to support NoSQL vector search. -> -> ```azurecli -> az cosmosdb update \ -> --resource-group \ -> --name \ -> --capabilities EnableNoSQLVectorSearch -> ``` +1. Read the description of the feature to confirm you want to enable it. +1. Select "Enable" to turn on vector search in Azure Cosmos DB for NoSQL. + + > [!TIP] + > Alternatively, use the Azure CLI to update the capabilities of your account to support NoSQL vector search. + > + > ```azurecli + > az cosmosdb update \ + > --resource-group \ + > --name \ + > --capabilities EnableNoSQLVectorSearch + > ``` > [!NOTE] > The registration request will be autoapproved; however, it may take 15 minutes to take effect. @@ -74,8 +65,8 @@ The following steps assume that you know how to [setup a Cosmos DB NoSQL account Let’s take an example of creating a database for an internet-based bookstore and you're storing Title, Author, ISBN, and Description for each book. We also define two properties to contain vector embeddings. The first is the "contentVector" property, which contains [text embeddings](/azure/ai-services/openai/concepts/models#embeddings ) generated from the text content of the book (for example, concatenating the "title" "author" "isbn" and "description" properties before creating the embedding). The second is "coverImageVector," which is generated from [images of the book’s cover](/azure/ai-services/computer-vision/concept-image-retrieval). 1. Create and store vector embeddings for the fields on which you want to perform vector search. -2. Specify the vector embedding paths in the vector embedding policy. -3. Include any desired vector indexes in the indexing policy for the container. +1. Specify the vector embedding paths in the vector embedding policy. +1. Include any desired vector indexes in the indexing policy for the container. For subsequent sections of this article, we consider this structure for the items stored in our container: diff --git a/articles/cosmos-db/nosql/how-to-manage-indexing-policy.md b/articles/cosmos-db/nosql/how-to-manage-indexing-policy.md index a7cee6c9f4..8fa26eb959 100644 --- a/articles/cosmos-db/nosql/how-to-manage-indexing-policy.md +++ b/articles/cosmos-db/nosql/how-to-manage-indexing-policy.md @@ -5,15 +5,16 @@ author: markjbrown ms.service: azure-cosmos-db ms.subservice: nosql ms.topic: how-to -ms.date: 03/08/2023 +ms.date: 12/03/2024 ms.author: mjbrown ms.custom: devx-track-csharp, build-2024, ignite-2024 ms.collection: - ce-skilling-ai-copilot +appliesto: + - ✅ NoSQL --- # Manage indexing policies in Azure Cosmos DB -[!INCLUDE[NoSQL](../includes/appliesto-nosql.md)] In Azure Cosmos DB, data is indexed following [indexing policies](../index-policy.md) that are defined for each container. The default indexing policy for newly created containers enforces range indexes for any string or number. You can override this policy with your own custom indexing policy. @@ -24,7 +25,7 @@ In Azure Cosmos DB, data is indexed following [indexing policies](../index-polic Here are some examples of indexing policies shown in [their JSON format](../index-policy.md). They're exposed on the Azure portal in JSON format. The same parameters can be set through the Azure CLI or any SDK. -### Opt-out policy to selectively exclude some property paths +### Opt-out policy to selectively exclude some property paths ```json { @@ -69,7 +70,7 @@ Here are some examples of indexing policies shown in [their JSON format](../inde > [!NOTE] > We generally recommend that you use an *opt-out* indexing policy. Azure Cosmos DB proactively indexes any new property that might be added to your data model. -### Using a spatial index on a specific property path only +### Using a spatial index on a specific property path only ```json { @@ -100,10 +101,11 @@ Here are some examples of indexing policies shown in [their JSON format](../inde ``` ## Vector indexing policy examples -In addition to including or excluding paths for individual properties, you can also specify a [vector index](../index-policy.md#vector-indexes). In general, vector indexes should be specified whenever the `VectorDistance` system function is used to measure similarity between a query vector and a vector property. + +In addition to including or excluding paths for individual properties, you can also specify a [vector index](../index-policy.md#vector-indexes). In general, vector indexes should be specified whenever the `VectorDistance` system function is used to measure similarity between a query vector and a vector property. > [!NOTE] -> Before proceeding, you must enable the [Azure Cosmos DB NoSQL Vector Indexing and Search](vector-search.md#enable-the-vector-indexing-and-search-feature). +> Before proceeding, you must enable the [Azure Cosmos DB NoSQL Vector Indexing and Search](vector-search.md#enable-the-vector-indexing-and-search-feature). >[!IMPORTANT] > A vector indexing policy must be on the same path defined in the container's vector policy. [Learn more about container vector policies](vector-search.md#container-vector-policies). @@ -134,7 +136,7 @@ In addition to including or excluding paths for individual properties, you can a } ``` ->[!IMPORTANT] +> [!IMPORTANT] > The vector path added to the "excludedPaths" section of the indexing policy to ensure optimized performance for insertion. Not adding the vector path to "excludedPaths" will result in higher RU charge and latency for vector insertions. > [!IMPORTANT] @@ -154,9 +156,10 @@ The `quantizedFlat` index stores quantized or compressed vectors on the index. V The `diskANN` index is a separate index defined specifically for vectors leveraging [DiskANN](https://www.microsoft.com/research/publication/diskann-fast-accurate-billion-point-nearest-neighbor-search-on-a-single-node/), a suite of highly performant vector indexing algorithms developed by Microsoft Research. DiskANN indexes can offer some of the lowest latency, highest query-per-second (QPS), and lowest RU cost queries at high accuracy. However, since DiskANN is an approximate nearest neighbors (ANN) index, the accuracy may be lower than `quantizedFlat` or `flat`. -The `diskANN` and `quantizedFlat` indexes can take optional index build parameters that can be used to tune the accuracy vs latency trade off that applies to every Approximate Nearest Neighbors vector index. -* `quantizationByteSize`: Sets the size (in bytes) for product quantization. Min=1, Default=dynamic (system decides), Max=512. Setting this larger may result in higher accuracy vector searches at expense of higher RU cost and higher latency. This applies to both `quantizedFlat` and `DiskANN` index types. -* `indexingSearchListSize`: Sets how many vectors to search over during index build construction. Min=10, Default=100, Max=500. Setting this larger may result in higher accuracy vector searches at the expense of longer index build times and higher vector ingest latencies. This applies to `DiskANN` indexes only. +The `diskANN` and `quantizedFlat` indexes can take optional index build parameters that can be used to tune the accuracy vs latency trade-off that applies to every Approximate Nearest Neighbors vector index. + +- `quantizationByteSize`: Sets the size (in bytes) for product quantization. Min=1, Default=dynamic (system decides), Max=512. Setting this larger may result in higher accuracy vector searches at expense of higher RU cost and higher latency. This applies to both `quantizedFlat` and `DiskANN` index types. +- `indexingSearchListSize`: Sets how many vectors to search over during index build construction. Min=10, Default=100, Max=500. Setting this larger may result in higher accuracy vector searches at the expense of longer index build times and higher vector ingest latencies. This applies to `DiskANN` indexes only. ### Tuple indexing policy examples @@ -185,8 +188,7 @@ WHERE WHERE ev.name = ‘M&M’ AND ev.category = ‘Candy’) ``` - -## Composite indexing policy examples +## Composite indexing policy examples In addition to including or excluding paths for individual properties, you can also specify a composite index. To perform a query that has an `ORDER BY` clause for multiple properties, a [composite index](../index-policy.md#composite-indexes) is required on those properties. If the query includes filters along with sorting on multiple properties, you may need more than one composite index. @@ -361,11 +363,10 @@ In Azure Cosmos DB, the indexing policy can be updated using any of the followin An [indexing policy update](../index-policy.md#modifying-the-indexing-policy) triggers an index transformation. The progress of this transformation can also be tracked from the SDKs. > [!NOTE] -> When you update indexing policy, writes to Azure Cosmos DB are uninterrupted. Learn more about [indexing transformations](../index-policy.md#modifying-the-indexing-policy) - -> [!IMPORTANT] -> Removing an index takes effect immediately, whereas adding a new index takes some time as it requires an indexing transformation. When replacing one index with another (for example, replacing a single property index with a composite-index) make sure to add the new index first and then wait for the index transformation to complete **before** you remove the previous index from the indexing policy. Otherwise this will negatively affect your ability to query the previous index and may break any active workloads that reference the previous index. +> When you update indexing policy, writes to Azure Cosmos DB are uninterrupted. Learn more about [indexing transformations](../index-policy.md#modifying-the-indexing-policy). +> [!IMPORTANT] +> Removing an index takes effect immediately, whereas adding a new index takes some time as it requires an indexing transformation. When replacing one index with another (for example, replacing a single property index with a composite-index) make sure to add the new index first and then wait for the index transformation to complete **before** you remove the previous index from the indexing policy. Otherwise this will negatively affect your ability to query the previous index and may break any active workloads that reference the previous index. ### Use the Azure portal @@ -393,9 +394,9 @@ To create a container with a custom indexing policy, see [Create a container wit To create a container with a custom indexing policy, see [Create a container with a custom index policy using PowerShell](manage-with-powershell.md#create-container-custom-index). -### Use the .NET SDK +### Use the .NET SDK -# [.NET SDK V3](#tab/dotnetv3) +#### [.NET SDK V3](#tab/dotnetv3) The `ContainerProperties` object from the [.NET SDK v3](https://www.nuget.org/packages/Microsoft.Azure.Cosmos/) exposes an `IndexingPolicy` property that lets you change the `IndexingMode` and add or remove `IncludedPaths` and `ExcludedPaths`. For more information, see [Quickstart: Azure Cosmos DB for NoSQL client library for .NET](quickstart-dotnet.md). @@ -452,7 +453,7 @@ await client.GetDatabase("database").DefineContainer(name: "container", partitio .CreateIfNotExistsAsync(); ``` -# [.NET SDK V2](#tab/dotnetv2) +#### [.NET SDK V2](#tab/dotnetv2) The `DocumentCollection` object from the [.NET SDK v2](https://www.nuget.org/packages/Microsoft.Azure.DocumentDB/) exposes an `IndexingPolicy` property that lets you change the `IndexingMode` and add or remove `IncludedPaths` and `ExcludedPaths`. @@ -635,6 +636,7 @@ const containerResponse = await client.database('database').container('container // retrieve the index transformation progress from the response headers const indexTransformationProgress = replaceResponse.headers['x-ms-documentdb-collection-index-transformation-progress']; ``` + Add a composite index: ```javascript @@ -669,7 +671,7 @@ Add a composite index: ### Use the Python SDK -# [Python SDK V3](#tab/pythonv3) +#### [Python SDK V3](#tab/pythonv3) When you use the [Python SDK V3](https://pypi.org/project/azure-cosmos/), the container configuration is managed as a dictionary. From this dictionary, you can access the indexing policy and all its attributes. For more information, see [Quickstart: Azure Cosmos DB for NoSQL client library for Python](quickstart-python.md). @@ -733,7 +735,7 @@ Update the container with changes: response = client.ReplaceContainer(containerPath, container) ``` -# [Python SDK V4](#tab/pythonv4) +#### [Python SDK V4](#tab/pythonv4) When you use the [Python SDK V4](https://pypi.org/project/azure-cosmos/), the container configuration is managed as a dictionary. From this dictionary, you can access the indexing policy and all its attributes. @@ -808,9 +810,7 @@ container_client.read(populate_quota_info = True, --- -## Next steps - -Read more about the indexing in the following articles: +## Related content - [Indexing overview](../index-overview.md) - [Indexing policy](../index-policy.md) diff --git a/articles/cosmos-db/nosql/how-to-python-vector-index-query.md b/articles/cosmos-db/nosql/how-to-python-vector-index-query.md index 430ba92e6c..3bb037026f 100644 --- a/articles/cosmos-db/nosql/how-to-python-vector-index-query.md +++ b/articles/cosmos-db/nosql/how-to-python-vector-index-query.md @@ -7,90 +7,93 @@ ms.author: jacodel ms.service: azure-cosmos-db ms.subservice: nosql ms.topic: how-to -ms.date: 08/01/2023 +ms.date: 12/03/2024 ms.custom: query-reference, devx-track-python, build-2024, ignite-2024 ms.collection: - ce-skilling-ai-copilot +appliesto: + - ✅ NoSQL --- # Index and query vectors in Azure Cosmos DB for NoSQL in Python -[!INCLUDE[NoSQL](../includes/appliesto-nosql.md)] Before you use vector indexing and search, you must first enable the feature. This article covers the following steps: 1. Enabling the Vector Search in Azure Cosmos DB for NoSQL feature -2. Setting up the Azure Cosmos DB container for vector search -3. Authoring vector embedding policy -4. Adding vector indexes to the container indexing policy -5. Creating a container with vector indexes and vector embedding policy -6. Performing a vector search on the stored data +1. Setting up the Azure Cosmos DB container for vector search +1. Authoring vector embedding policy +1. Adding vector indexes to the container indexing policy +1. Creating a container with vector indexes and vector embedding policy +1. Performing a vector search on the stored data This guide walks through the process of creating vector data, indexing the data, and then querying the data in a container. ## Prerequisites + - An existing Azure Cosmos DB for NoSQL account. - If you don't have an Azure subscription, [Try Azure Cosmos DB for NoSQL free](https://cosmos.azure.com/try/). - If you have an existing Azure subscription, [create a new Azure Cosmos DB for NoSQL account](how-to-create-account.md). - Latest version of the Azure Cosmos DB [Python](sdk-python.md) SDK. ## Enable the feature + Vector search for Azure Cosmos DB for NoSQL requires enabling the feature by completing the following steps: 1. Navigate to your Azure Cosmos DB for NoSQL resource page. -2. Select the "Features" pane under the "Settings" menu item. -3. Select for "Vector Search in Azure Cosmos DB for NoSQL." -4. Read the description of the feature to confirm you want to enable it. -5. Select "Enable" to turn on vector search in Azure Cosmos DB for NoSQL. - -> [!TIP] -> Alternatively, use the Azure CLI to update the capabilities of your account to support NoSQL vector search. -> -> ```azurecli -> az cosmosdb update \ -> --resource-group \ -> --name \ -> --capabilities EnableNoSQLVectorSearch -> ``` - +1. Select the "Features" pane under the "Settings" menu item. +1. Select for "Vector Search in Azure Cosmos DB for NoSQL." +1. Read the description of the feature to confirm you want to enable it. +1. Select "Enable" to turn on vector search in Azure Cosmos DB for NoSQL. + + > [!TIP] + > Alternatively, use the Azure CLI to update the capabilities of your account to support NoSQL vector search. + > + > ```azurecli + > az cosmosdb update \ + > --resource-group \ + > --name \ + > --capabilities EnableNoSQLVectorSearch + > ``` > [!NOTE] > The registration request will be autoapproved; however, it may take 15 minutes to take effect. -## Understanding the steps involved in vector search +## Understanding the steps involved in vector search -The following steps assume that you know how to [setup a Cosmos DB NoSQL account and create a database](quickstart-portal.md). The vector search feature is currently not supported on the existing containers, so you need to create a new container and specify the container-level vector embedding policy, and the vector indexing policy at the time of container creation. +The following steps assume that you know how to [setup a Cosmos DB NoSQL account and create a database](quickstart-portal.md). The vector search feature is currently not supported on the existing containers, so you need to create a new container and specify the container-level vector embedding policy, and the vector indexing policy at the time of container creation. -Let’s take an example of creating a database for an internet-based bookstore and you're storing Title, Author, ISBN, and Description for each book. We’ll also define two properties to contain vector embeddings. The first is the “contentVector” property, which contains [text embeddings](/azure/ai-services/openai/concepts/models#embeddings ) generated from the text content of the book (for example, concatenating the “title” “author” “isbn” and “description” properties before creating the embedding). The second is “coverImageVector”, which is generated from [images of the book’s cover](/azure/ai-services/computer-vision/concept-image-retrieval). +Let’s take an example of creating a database for an internet-based bookstore and you're storing Title, Author, ISBN, and Description for each book. We’ll also define two properties to contain vector embeddings. The first is the “contentVector” property, which contains [text embeddings](/azure/ai-services/openai/concepts/models#embeddings ) generated from the text content of the book (for example, concatenating the “title” “author” “isbn” and “description” properties before creating the embedding). The second is “coverImageVector”, which is generated from [images of the book’s cover](/azure/ai-services/computer-vision/concept-image-retrieval). -1. Create and store vector embeddings for the fields on which you want to perform vector search. -2. Specify the vector embedding paths in the vector embedding policy. -3. Include any desired vector indexes in the indexing policy for the container. +1. Create and store vector embeddings for the fields on which you want to perform vector search. +1. Specify the vector embedding paths in the vector embedding policy. +1. Include any desired vector indexes in the indexing policy for the container. -For subsequent sections of this article, we consider the below structure for the items stored in our container: +For subsequent sections of this article, we consider the below structure for the items stored in our container: ```json { -"title": "book-title", -"author": "book-author", -"isbn": "book-isbn", -"description": "book-description", -"contentVector": [2, -1, 4, 3, 5, -2, 5, -7, 3, 1], -"coverImageVector": [0.33, -0.52, 0.45, -0.67, 0.89, -0.34, 0.86, -0.78] -} + "title": "book-title", + "author": "book-author", + "isbn": "book-isbn", + "description": "book-description", + "contentVector": [2, -1, 4, 3, 5, -2, 5, -7, 3, 1], + "coverImageVector": [0.33, -0.52, 0.45, -0.67, 0.89, -0.34, 0.86, -0.78] +} ``` ## Creating a vector embedding policy for your container + Next, you need to define a container vector policy. This policy provides information that is used to inform the Azure Cosmos DB query engine how to handle vector properties in the VectorDistance system functions. This also informs the vector indexing policy of necessary information, should you choose to specify one. The following information is included in the contained vector policy: - * “path”: The property path that contains vectors  - * “datatype”: The type of the elements of the vector (default Float32)  - * “dimensions”: The length of each vector in the path (default 1536)  - * “distanceFunction”: The metric used to compute distance/similarity (default Cosine)  +- “path”: The property path that contains vectors  +- “datatype”: The type of the elements of the vector (default Float32)  +- “dimensions”: The length of each vector in the path (default 1536)  +- “distanceFunction”: The metric used to compute distance/similarity (default Cosine)  For our example with book details, the vector policy may look like the example JSON: -```python +```python vector_embedding_policy = {     "vectorEmbeddings": [         { @@ -107,14 +110,13 @@ vector_embedding_policy = {         } ] } -``` - +``` +## Creating a vector index in the indexing policy -## Creating a vector index in the indexing policy -Once the vector embedding paths are decided, vector indexes need to be added to the indexing policy. For this example, the indexing policy would look something like this: +Once the vector embedding paths are decided, vector indexes need to be added to the indexing policy. For this example, the indexing policy would look something like this: -```python +```python indexing_policy = {     "includedPaths": [         { @@ -138,21 +140,19 @@ indexing_policy = {         }     ] } -``` +``` ->[!IMPORTANT] +> [!IMPORTANT] > The vector path added to the "excludedPaths" section of the indexing policy to ensure optimized performance for insertion. Not adding the vector path to "excludedPaths" will result in higher RU charge and latency for vector insertions. - > [!IMPORTANT] -> Currently vector search in Azure Cosmos DB for NoSQL is supported on new containers only. You need to set both the container vector policy and any vector indexing policy during the time of container creation as it can’t be modified later. Both policies will be modifiable in a future improvement to the preview feature. - +> Currently vector search in Azure Cosmos DB for NoSQL is supported on new containers only. You need to set both the container vector policy and any vector indexing policy during the time of container creation as it can’t be modified later. -## Create container with vector policy +## Create container with vector policy Currently the vector search feature for Azure Cosmos DB for NoSQL is supported only on new containers so you need to apply the vector policy during the time of container creation and it can’t be modified later. -```python +```python try:         container = db.create_container_if_not_exists(                     id=CONTAINER_NAME, @@ -163,11 +163,11 @@ try:     except exceptions.CosmosHttpResponseError: raise -``` +``` -## Running vector similarity search query +## Running vector similarity search query -Once you create a container with the desired vector policy, and insert vector data into the container, you can conduct a vector search using the [Vector Distance](query/vectordistance.md) system function in a query. Suppose you want to search for books about food recipes by looking at the description, you first need to get the embeddings for your query text. In this case, you might want to generate embeddings for the query text – “food recipe”. Once you have the embedding for your search query, you can use it in the VectorDistance function in the vector search query and get all the items that are similar to your query as shown here: +Once you create a container with the desired vector policy, and insert vector data into the container, you can conduct a vector search using the [Vector Distance](query/vectordistance.md) system function in a query. Suppose you want to search for books about food recipes by looking at the description, you first need to get the embeddings for your query text. In this case, you might want to generate embeddings for the query text – “food recipe”. Once you have the embedding for your search query, you can use it in the VectorDistance function in the vector search query and get all the items that are similar to your query as shown here: ```sql SELECT TOP 10 c.title, VectorDistance(c.contentVector, [1,2,3,4,5,6,7,8,9,10]) AS SimilarityScore   @@ -177,7 +177,7 @@ ORDER BY VectorDistance(c.contentVector, [1,2,3,4,5,6,7,8,9,10])   This query retrieves the book titles along with similarity scores with respect to your query. Here's an example in Python: -```python +```python query_embedding = [1,2,3,4,5,6,7,8,9,10] # Query for items for item in container.query_items( @@ -187,7 +187,8 @@ for item in container.query_items(             ],             enable_cross_partition_query=True):     print(json.dumps(item, indent=True)) -``` +``` + ## Related content - [VectorDistance system function](query/vectordistance.md) diff --git a/articles/cosmos-db/nosql/multi-tenancy-vector-search.md b/articles/cosmos-db/nosql/multi-tenancy-vector-search.md index d9c922286d..d40cba437c 100644 --- a/articles/cosmos-db/nosql/multi-tenancy-vector-search.md +++ b/articles/cosmos-db/nosql/multi-tenancy-vector-search.md @@ -1,27 +1,30 @@ --- - title: Multitenancy in Azure Cosmos DB -description: Learn concepts for building multitenant gen-ai apps in Azure Cosmos DB +description: Review critical concepts required to build multitenant generative AI applications in Azure Cosmos DB. author: TheovanKraay ms.service: azure-cosmos-db ms.subservice: nosql -ms.topic: conceptual -ms.date: 06/26/2024 +ms.topic: concept-article +ms.date: 12/03/2024 ms.author: thvankra ms.collection: - ce-skilling-ai-copilot +appliesto: + - ✅ NoSQL + - ✅ MongoDB vCore + - ✅ PostgreSQL --- # Multitenancy for vector search in Azure Cosmos DB -> "OpenAI relies on Cosmos DB to dynamically scale their ChatGPT service – one of the fastest-growing consumer apps ever – enabling high reliability and low maintenance." +> "OpenAI relies on Cosmos DB to dynamically scale their ChatGPT service – one of the fastest-growing consumer apps ever – enabling high reliability and low maintenance." > — Satya Nadella -Azure Cosmos DB stands out as the world's first full-featured serverless operational database with vector search, offering unparalleled scalability and performance. By using Azure Cosmos DB, users can enhance their vector search capabilities, ensuring high reliability and low maintenance for multitenant applications. +Azure Cosmos DB stands out as the world's first full-featured serverless operational database with vector search, offering unparalleled scalability and performance. By using Azure Cosmos DB, users can enhance their vector search capabilities, ensuring high reliability and low maintenance for multitenant applications. Multitenancy enables a single instance of a database to serve multiple customers, or tenants, simultaneously. This approach efficiently shares infrastructure and operational overhead, resulting in cost savings and simplified management. It's a crucial design consideration for SaaS applications and some internal enterprise solutions. -Multitenancy introduces complexity. Your system must scale efficiently to maintain high performance across all tenants, who may have unique workloads, requirements, and service-level agreements (SLAs). +Multitenancy introduces complexity. Your system must scale efficiently to maintain high performance across all tenants, who may have unique workloads, requirements, and service-level agreements (SLAs). Imagine a fictional AI-assisted research platform called ResearchHub. Serving thousands of companies and individual researchers, ResearchHub manages varying user bases, data scales, and SLAs. Ensuring low query latency and high performance is vital for sustaining an excellent user experience. @@ -39,7 +42,7 @@ For a higher density of tenants and lower isolation, the partition key-per-tenan - **Cost Efficiency:** Sharing a single Cosmos DB account across multiple tenants reduces overhead. - **Scalability:** Can manage a large number of tenants, each isolated within their partition key. - **Simplified Management:** Fewer Cosmos DB accounts to manage. -- **Hierarchical Partition Keys (HPK):** Optimizes data organization and query performance in multi-tenant apps with a high number of tenants. +- **Hierarchical Partition Keys (HPK):** Optimizes data organization and query performance in multitenant apps with a high number of tenants. **Drawbacks:** - **Resource Contention:** Shared resources can lead to contention during peak usage. @@ -157,16 +160,16 @@ When designing a multitenant system with Cosmos DB, consider these factors: Azure Cosmos DB's support for DiskANN vector index capability makes it an excellent choice for applications that require fast, high-dimensional searches, such as AI-assisted research platforms like ResearchHub. Here’s how you can leverage these capabilities: **Efficient Storage and Retrieval:** - - **Vector Indexing:** Use the DiskANN vector index to efficiently store and retrieve high-dimensional vectors. This is useful for applications that involve similarity searches in large datasets, such as image recognition or document similarity. - - **Performance Optimization:** DiskANN’s vector search capabilities enable quick, accurate searches, ensuring low latency and high performance, which is critical for maintaining a good user experience. +- **Vector Indexing:** Use the DiskANN vector index to efficiently store and retrieve high-dimensional vectors. This is useful for applications that involve similarity searches in large datasets, such as image recognition or document similarity. +- **Performance Optimization:** DiskANN’s vector search capabilities enable quick, accurate searches, ensuring low latency and high performance, which is critical for maintaining a good user experience. **Scaling Across Tenants:** - - **Partition Key-Per-Tenant:** Utilize partition keys to logically isolate tenant data while benefiting from Cosmos DB’s scalable infrastructure. - - **Hierarchical Partitioning:** Implement hierarchical partitioning to further segment data within each tenant’s partition, improving query performance and resource distribution. +- **Partition Key-Per-Tenant:** Utilize partition keys to logically isolate tenant data while benefiting from Cosmos DB’s scalable infrastructure. +- **Hierarchical Partitioning:** Implement hierarchical partitioning to further segment data within each tenant’s partition, improving query performance and resource distribution. **Security and Compliance:** - - **Customer-Managed Keys:** Implement customer-managed keys for data encryption at rest, ensuring each tenant’s data is securely isolated. - - **Regular Key Rotation:** Enhance security by regularly rotating encryption keys stored in Azure Key Vault. +- **Customer-Managed Keys:** Implement customer-managed keys for data encryption at rest, ensuring each tenant’s data is securely isolated. +- **Regular Key Rotation:** Enhance security by regularly rotating encryption keys stored in Azure Key Vault. ### Real-world example: implementing ResearchHub @@ -190,16 +193,18 @@ Multi-tenancy in Azure Cosmos DB, especially with its DiskANN vector index capab Azure Cosmos DB provides the tools necessary to build a robust, secure, and scalable multitenant environment. With the power of DiskANN vector indexing, you can deliver fast, high-dimensional searches that drive your AI applications. -### Next steps +## Vector database solutions -[30-day Free Trial without Azure subscription](https://azure.microsoft.com/try/cosmosdb/) +[Azure PostgreSQL Server pgvector Extension](../../postgresql/flexible-server/how-to-use-pgvector.md) -[Multitenancy and Azure Cosmos DB](https://aka.ms/CosmosMultitenancy) +:::image type="content" source="../media/vector-search/azure-databases-and-ai-search.png" lightbox="../media/vector-search/azure-databases-and-ai-search.png" alt-text="Diagram of Vector indexing services for Azure PostgreSQL."::: -> [!div class="nextstepaction"] -> [Use the Azure Cosmos DB lifetime free tier](../free-tier.md) +## Related content -## More vector database solutions -- [Azure PostgreSQL Server pgvector Extension](../../postgresql/flexible-server/how-to-use-pgvector.md) +- [30-day Free Trial without Azure subscription](https://azure.microsoft.com/try/cosmosdb/) +- [Multitenancy and Azure Cosmos DB](https://aka.ms/CosmosMultitenancy) -:::image type="content" source="../media/vector-search/azure-databases-and-ai-search.png" lightbox="../media/vector-search/azure-databases-and-ai-search.png" alt-text="Diagram of Vector indexing services."::: \ No newline at end of file +## Next step + +> [!div class="nextstepaction"] +> [Use the Azure Cosmos DB lifetime free tier](../free-tier.md) diff --git a/articles/cosmos-db/nosql/query/how-to-enable-use-copilot.md b/articles/cosmos-db/nosql/query/how-to-enable-use-copilot.md index ac1a0aefc2..389d28b55f 100644 --- a/articles/cosmos-db/nosql/query/how-to-enable-use-copilot.md +++ b/articles/cosmos-db/nosql/query/how-to-enable-use-copilot.md @@ -1,5 +1,5 @@ --- -title: Query NoSQL with Microsoft Copilot for Azure (preview) +title: Queries with Microsoft Copilot for Azure (preview) titleSuffix: Azure Cosmos DB for NoSQL description: Generate suggestions from natural language prompts to write NoSQL queries using Microsoft Copilot for Azure in Cosmos DB (preview). author: seesharprun @@ -10,16 +10,16 @@ ms.custom: - ignite-2023 ms.topic: how-to ms.devlang: nosql -ms.date: 08/22/2024 +ms.date: 12/03/2024 ms.collection: - ce-skilling-ai-copilot # CustomerIntent: As a developer, I want to use Copilot so that I can write queries faster and easier. +appliesto: + - ✅ NoSQL --- # Generate NoSQL queries with Microsoft Copilot for Azure in Cosmos DB (preview) -[!INCLUDE[NoSQL](../../includes/appliesto-nosql.md)] - Microsoft Copilot for Azure in Cosmos DB (preview) can assist with authoring Azure Cosmos DB for NoSQL queries by generating queries based on your natural English-language prompts. Copilot is available to use in the API for NoSQL's query editor within the Data Explorer. With Copilot in the API for NoSQL, you can: - Ask questions about your data as you would in text or conversation to generate a NoSQL query. @@ -28,7 +28,6 @@ Microsoft Copilot for Azure in Cosmos DB (preview) can assist with authoring Azu > [!NOTE] > You may see the database `CopilotSampleDb` appear in Data Explorer. This is a completely separate database managed by Microsoft and access is provided to you (at no cost) as a testbed to become familiar with Microsoft Copilot for Azure in Cosmos DB. This database consists of 100% synthetic data created by Microsoft and has has no interaction or relationships to any data you may have in Azure Cosmos DB. - > [!WARNING] > Copilot is a preview feature that is powered by large language models (LLMs). Output produced by Copilot may contain inaccuracies, biases, or other unintended content. This occurs because the model powering Copilot was trained on information from the internet and other sources. As with any generative AI model, humans should review the output produced by Copilot before use. @@ -38,11 +37,12 @@ Microsoft Copilot for Azure in Cosmos DB (preview) can assist with authoring Azu - If you don't have an Azure subscription, [create an account for free](https://azure.microsoft.com/free). - Once you have an existing Azure subscription, [create a new Azure Cosmos DB for NoSQL account](../quickstart-portal.md). - Enroll your Azure subscription, in the Microsoft Copilot for Azure in Cosmos DB [preview feature](/azure/azure-resource-manager/management/preview-features). - + > [!IMPORTANT] > Review these [preview terms](https://azure.microsoft.com/support/legal/preview-supplemental-terms/#AzureOpenAI-PoweredPreviews) before using query Copilot for NoSQL. ## Access the feature + As a preview feature, you'll have to add Microsoft Copilot for Azure in Cosmos DB preview to your Azure subscription. Once enrolled, you can find Microsoft Copilot for Azure integrated with the Data Explorer’s query editor. 1. Navigate to any Azure Cosmos DB NoSQL resource. @@ -53,7 +53,7 @@ As a preview feature, you'll have to add Microsoft Copilot for Azure in Cosmos D 1. Next, open the query editor experience from one of two ways: - - Select the **Query faster with Copilot** card on the Data Explorer's welcome screen. This option will take you to the a query editor targeting the `CopilotSampleDb` database and `SampleContainer` container, which contains sample data for you to use with Copilot. This database is managed by Microsoft and does not interact or connect to your other databases. `CopoilotSampleDb` is free for all Azure Cosmos DB NoSQL customers. + - Select the **Query faster with Copilot** card on the Data Explorer's welcome screen. This option will take you to the query editor targeting the `CopilotSampleDb` database and `SampleContainer` container, which contains sample data for you to use with Copilot. This database is managed by Microsoft and does not interact or connect to your other databases. `CopoilotSampleDb` is free for all Azure Cosmos DB NoSQL customers. - Select an existing API for NoSQL database and container. Then, select **New SQL Query** from the menu bar. diff --git a/articles/cosmos-db/nosql/vector-search.md b/articles/cosmos-db/nosql/vector-search.md index 050268964a..639ecb0f9e 100644 --- a/articles/cosmos-db/nosql/vector-search.md +++ b/articles/cosmos-db/nosql/vector-search.md @@ -1,26 +1,25 @@ --- -title: Vector store +title: Integrated vector store titleSuffix: Azure Cosmos DB for NoSQL -description: Use vector store in Azure Cosmos DB for NoSQL to enhance AI-based applications. +description: Enhance AI-based applications using the integrated vector store functionality in Azure Cosmos DB for NoSQL author: jcodella ms.author: jacodel ms.service: azure-cosmos-db ms.subservice: nosql ms.custom: - - Build 2024 - build-2024 - ignite-2024 -ms.topic: conceptual -ms.date: 5/7/2024 +ms.topic: concept-article +ms.date: 12/03/2024 ms.collection: - ce-skilling-ai-copilot +appliesto: + - ✅ NoSQL --- # Vector Search in Azure Cosmos DB for NoSQL -[!INCLUDE[NoSQL](../includes/appliesto-nosql.md)] - -Azure Cosmos DB for NoSQL now offers efficient vector indexing and search. This feature is designed to handle high-dimensional vectors, enabling efficient and accurate vector search at any scale. You can now store vectors directly in the documents alongside your data. Each document in your database can contain not only traditional schema-free data, but also high-dimensional vectors as other properties of the documents. This colocation of data and vectors allows for efficient indexing and searching, as the vectors are stored in the same logical unit as the data they represent. Keeping vectors and data together simplifies data management, AI application architectures, and the efficiency of vector-based operations. +Azure Cosmos DB for NoSQL now offers efficient vector indexing and search. This feature is designed to handle high-dimensional vectors, enabling efficient and accurate vector search at any scale. You can now store vectors directly in the documents alongside your data. Each document in your database can contain not only traditional schema-free data, but also high-dimensional vectors as other properties of the documents. This colocation of data and vectors allows for efficient indexing and searching, as the vectors are stored in the same logical unit as the data they represent. Keeping vectors and data together simplifies data management, AI application architectures, and the efficiency of vector-based operations. Azure Cosmos DB for NoSQL offers the flexibility it offers in choosing the vector indexing method: - A "flat" or k-nearest neighbors exact search (sometimes called brute-force) can provide 100% retrieval recall for smaller, focused vector searches. especially when combined with query filters and partition-keys. @@ -29,14 +28,13 @@ Azure Cosmos DB for NoSQL offers the flexibility it offers in choosing the vecto [Learn more about vector indexing here](../index-policy.md#vector-indexes) - Vector search in Azure Cosmos DB can be combined with all other supported Azure Cosmos DB NoSQL query filters and indexes using `WHERE` clauses. This enables your vector searches to be the most relevant data to your applications. This feature enhances the core capabilities of Azure Cosmos DB, making it more versatile for handling vector data and search requirements in AI applications. ## What is a vector store? -A vector store or [vector database](../vector-database.md) is a database designed to store and manage vector embeddings, which are mathematical representations of data in a high-dimensional space. In this space, each dimension corresponds to a feature of the data, and tens of thousands of dimensions might be used to represent sophisticated data. A vector's position in this space represents its characteristics. Words, phrases, or entire documents, and images, audio, and other types of data can all be vectorized. +A vector store or [vector database](../vector-database.md) is a database designed to store and manage vector embeddings, which are mathematical representations of data in a high-dimensional space. In this space, each dimension corresponds to a feature of the data, and tens of thousands of dimensions might be used to represent sophisticated data. A vector's position in this space represents its characteristics. Words, phrases, or entire documents, and images, audio, and other types of data can all be vectorized. ## How does a vector store work? @@ -45,50 +43,48 @@ In a vector store, vector search algorithms are used to index and query embeddin In the Integrated Vector Database in Azure Cosmos DB for NoSQL, embeddings can be stored, indexed, and queried alongside the original data. This approach eliminates the extra cost of replicating data in a separate pure vector database. Moreover, this architecture keeps the vector embeddings and original data together, which better facilitates multi-modal data operations, and enables greater data consistency, scale, and performance. ## Enable the vector indexing and search feature -Vector indexing and search in Azure Cosmos DB for NoSQL requires enabling on the Features page of your Azure Cosmos DB. Follow the below steps to register: + +Vector indexing and search in Azure Cosmos DB for NoSQL requires enabling on the Features page of your Azure Cosmos DB. Follow the below steps to register: 1. Navigate to your Azure Cosmos DB for NoSQL resource page. -2. Select the "Features" pane under the "Settings" menu item. -3. Select the “Vector Search in Azure Cosmos DB for NoSQL” feature. -4. Read the description of the feature to confirm you want to enable it. -5. Select "Enable" to turn on the vector indexing and search capability. +1. Select the "Features" pane under the "Settings" menu item. +1. Select the “Vector Search in Azure Cosmos DB for NoSQL” feature. +1. Read the description of the feature to confirm you want to enable it. +1. Select "Enable" to turn on the vector indexing and search capability. + + > [!TIP] + > Alternatively, use the Azure CLI to update the capabilities of your account to support NoSQL vector search. + > + > ```azurecli + > az cosmosdb update \ + > --resource-group \ + > --name \ + > --capabilities EnableNoSQLVectorSearch + > ``` + > > [!NOTE] > The registration request will be autoapproved; however, it may take 15 minutes to fully activate on the account. -> [!TIP] -> Alternatively, use the Azure CLI to update the capabilities of your account to support NoSQL vector search. -> -> ```azurecli -> az cosmosdb update \ -> --resource-group \ -> --name \ -> --capabilities EnableNoSQLVectorSearch -> ``` -> - -> [!div class="nextstepaction"] -> [Use the Azure Cosmos DB lifetime free tier](../free-tier.md) - ## Container Vector Policies -Performing vector search with Azure Cosmos DB for NoSQL requires you to define a vector policy for the container. This provides essential information for the database engine to conduct efficient similarity search for vectors found in the container's documents. This also informs the vector indexing policy of necessary information, should you choose to specify one. The following information is included in the contained vector policy: - * “path”: the property containing the vector (required). - * “datatype”: the data type of the vector property (default Float32).  - * “dimensions”: The dimensionality or length of each vector in the path. All vectors in a path should have the same number of dimensions. (default 1536). - * “distanceFunction”: The metric used to compute distance/similarity. Supported metrics are: - * [cosine](https://en.wikipedia.org/wiki/Cosine_similarity), which has values from -1 (least similar) to +1 (most similar). - * [dot product](https://en.wikipedia.org/wiki/Dot_product), which has values from -inf (least similar) to +inf (most similar). - * [euclidean](https://en.wikipedia.org/wiki/Euclidean_distance), which has values from 0 (most similar) to +inf) (least similar). +Performing vector search with Azure Cosmos DB for NoSQL requires you to define a vector policy for the container. This provides essential information for the database engine to conduct efficient similarity search for vectors found in the container's documents. This also informs the vector indexing policy of necessary information, should you choose to specify one. The following information is included in the contained vector policy: +- “path”: the property containing the vector (required). +- “datatype”: the data type of the vector property (default Float32).  +- “dimensions”: The dimensionality or length of each vector in the path. All vectors in a path should have the same number of dimensions. (default 1536). +- “distanceFunction”: The metric used to compute distance/similarity. Supported metrics are: + - [cosine](https://en.wikipedia.org/wiki/Cosine_similarity), which has values from -1 (least similar) to +1 (most similar). + - [dot product](https://en.wikipedia.org/wiki/Dot_product), which has values from -inf (least similar) to +inf (most similar). + - [euclidean](https://en.wikipedia.org/wiki/Euclidean_distance), which has values from 0 (most similar) to +inf) (least similar). - > [!NOTE] > Each unique path can have at most one policy. However, multiple policies can be specified provided that they all target a different path. - + The container vector policy can be described as JSON objects. Here are two examples of valid container vector policies: -**A policy with a single vector path** +### A policy with a single vector path + ```json { "vectorEmbeddings": [ @@ -102,7 +98,8 @@ The container vector policy can be described as JSON objects. Here are two examp } ``` -**A policy with two vector paths** +### A policy with two vector paths + ```json { "vectorEmbeddings": [ @@ -136,11 +133,12 @@ The container vector policy can be described as JSON objects. Here are two examp > The `quantizedFlat` and `diskANN` indexes requires that at least 1,000 vectors to be inserted. This is to ensure accuracy of the quantization process. If there are fewer than 1,000 vectors, a full scan is executed instead and will lead to higher RU charges for a vector search query. A few points to note: - - The `flat` and `quantizedFlat` index types uses Azure Cosmos DB's index to store and read each vector when performing a vector search. Vector searches with a `flat` index are brute-force searches and produce 100% accuracy or recall. That is, it's guaranteed to find the most similar vectors in the dataset. However, there's a limitation of `505` dimensions for vectors on a flat index. - - The `quantizedFlat` index stores quantized (compressed) vectors on the index. Vector searches with `quantizedFlat` index are also brute-force searches, however their accuracy might be slightly less than 100% since the vectors are quantized before adding to the index. However, vector searches with `quantized flat` should have lower latency, higher throughput, and lower RU cost than vector searches on a `flat` index. This is a good option for smaller scenarios, or scenarios where you're using query filters to narrow down the vector search to a relatively small set of vectors. `quantizedFlat` is recommended when the number of vectors to be indexed is somewhere around 50,000 or fewer per physical partition. However, this is just a general guideline and actual performance should be tested as each scenario can be different. +- The `flat` and `quantizedFlat` index types uses Azure Cosmos DB's index to store and read each vector when performing a vector search. Vector searches with a `flat` index are brute-force searches and produce 100% accuracy or recall. That is, it's guaranteed to find the most similar vectors in the dataset. However, there's a limitation of `505` dimensions for vectors on a flat index. + +- The `quantizedFlat` index stores quantized (compressed) vectors on the index. Vector searches with `quantizedFlat` index are also brute-force searches, however their accuracy might be slightly less than 100% since the vectors are quantized before adding to the index. However, vector searches with `quantized flat` should have lower latency, higher throughput, and lower RU cost than vector searches on a `flat` index. This is a good option for smaller scenarios, or scenarios where you're using query filters to narrow down the vector search to a relatively small set of vectors. `quantizedFlat` is recommended when the number of vectors to be indexed is somewhere around 50,000 or fewer per physical partition. However, this is just a general guideline and actual performance should be tested as each scenario can be different. - - The `diskANN` index is a separate index defined specifically for vectors using [DiskANN](https://www.microsoft.com/research/publication/diskann-fast-accurate-billion-point-nearest-neighbor-search-on-a-single-node/), a suite of high performance vector indexing algorithms developed by Microsoft Research. DiskANN indexes can offer some of the lowest latency, highest throughput, and lowest RU cost queries, while still maintaining high accuracy. In general, DiskANN is the most performant of all index types if there are more than 50,000 vectors per physical partition. +- The `diskANN` index is a separate index defined specifically for vectors using [DiskANN](https://www.microsoft.com/research/publication/diskann-fast-accurate-billion-point-nearest-neighbor-search-on-a-single-node/), a suite of high performance vector indexing algorithms developed by Microsoft Research. DiskANN indexes can offer some of the lowest latency, highest throughput, and lowest RU cost queries, while still maintaining high accuracy. In general, DiskANN is the most performant of all index types if there are more than 50,000 vectors per physical partition. Here are examples of valid vector index policies: @@ -203,7 +201,7 @@ Here are examples of valid vector index policies: } ``` ->[!IMPORTANT] +> [!IMPORTANT] > The vector path added to the "excludedPaths" section of the indexing policy to ensure optimized performance for insertion. Not adding the vector path to "excludedPaths" will result in higher RU charge and latency for vector insertions. > [!IMPORTANT] @@ -219,11 +217,11 @@ FROM c  ORDER BY VectorDistance(c.contentVector, [1,2,3])   ``` ->[!IMPORTANT] +> [!IMPORTANT] > Always use a `TOP N` clause in the `SELECT` statement of a query. Otherwise the vector search will try to return many more results and the query will cost more RUs and have higher latency than necessary. - ## Current limitations + Vector indexing and search in Azure Cosmos DB for NoSQL has some limitations. - `quantizedFlat` and `diskANN` indexes require at least 1,000 vectors to be indexed to ensure that the quantization is accurate. If fewer than 1,000 vectors are indexed, then a full-scan is used instead and RU charges may be higher. - Vectors indexed with the `flat` index type can be at most 505 dimensions. Vectors indexed with the `quantizedFlat` or `DiskANN` index type can be at most 4,096 dimensions. @@ -233,7 +231,8 @@ Vector indexing and search in Azure Cosmos DB for NoSQL has some limitations. - At this time, vector indexing and search is not supported on accounts with Analytical Store (and Synapse Link) and Shared Throughput. - Once vector indexing and search is enabled on a container, it cannot be disabled. -## Next step +## Related content + - [DiskANN + Azure Cosmos DB - Microsoft Mechanics Video](https://www.youtube.com/watch?v=MlMPIYONvfQ) - [.NET - How-to Index and query vector data](how-to-dotnet-vector-index-query.md) - [Python - How-to Index and query vector data](how-to-python-vector-index-query.md) @@ -246,3 +245,8 @@ Vector indexing and search in Azure Cosmos DB for NoSQL has some limitations. - [LangChain, Python](https://python.langchain.com/v0.2/docs/integrations/vectorstores/azure_cosmos_db_no_sql/) - [Semantic Kernel, .NET](https://github.com/microsoft/semantic-kernel/tree/main/dotnet/src/Connectors/Connectors.Memory.AzureCosmosDBNoSQL) - [Semantic Kernel, Python](https://github.com/microsoft/semantic-kernel/tree/main/python/semantic_kernel/connectors/memory/azure_cosmosdb_no_sql) + +## Next step + +> [!div class="nextstepaction"] +> [Use the Azure Cosmos DB lifetime free tier](../free-tier.md)