diff --git a/.env.example b/.env.example index efedda9..3b632bb 100644 --- a/.env.example +++ b/.env.example @@ -1,12 +1,10 @@ -OPENAI_API_TYPE = "Set this to "azure" for API key authentication or "azure_ad" for Azure AD authentication>" -OPENAI_API_KEY = "" -OPENAI_API_BASE = "" -OPENAI_API_VERSION = "2023-05-15" -OPENAI_COMPLETION_MODEL = "" -AZURE_TENANT_ID = "" +OPENAI_API_VERSION = "2023-09-01-preview" +AZURE_OPENAI_API_KEY = "" +AZURE_OPENAI_ENDPOINT = "" +AZURE_OPENAI_COMPLETION_MODEL = "" AZURE_OPENAI_COMPLETION_DEPLOYMENT_NAME = "" AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME = "" -AZURE_COGNITIVE_SEARCH_SERVICE_NAME = "" -AZURE_COGNITIVE_SEARCH_ENDPOINT_NAME = "`** to make the name unique." + "**NOTE:** Before running the commands, replace the **``** with your own initials or some random characters, as we need to provide a unique name for the Azure AI Search service." ] }, { @@ -27,9 +28,9 @@ "metadata": {}, "outputs": [], "source": [ - "RESOURCE_GROUP=\"azure-cognitive-search-rg\"\n", + "RESOURCE_GROUP=\"azure-ai-search-rg\"\n", "LOCATION=\"westeurope\"\n", - "NAME=\"acs-vectorstore-\"\n", + "NAME=\"ai-vectorstore-\"\n", "!az group create --name $RESOURCE_GROUP --location $LOCATION\n", "!az search service create -g $RESOURCE_GROUP -n $NAME -l $LOCATION --sku Basic --partition-count 1 --replica-count 1" ] @@ -38,13 +39,13 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Next, we need to find and update the following values in the `.env` file with the Azure Cognitive Search **endpoint**, **admin key**, and **index name** values. Use the Azure Portal or CLI.\n", + "Next, we need to find and update the following values in the `.env` file with the Azure AI Search **name**, **endpoint** and **admin key** values, which you can get from the Azure portal. You also need to provide an **index name** value. The index will be created during this lab, so you can use any name you like.\n", "\n", "```\n", - "AZURE_COGNITIVE_SEARCH_SERVICE_NAME = \"\"\n", - "AZURE_COGNITIVE_SEARCH_ENDPOINT_NAME = \"\"\n", - "AZURE_COGNITIVE_SEARCH_API_KEY = \"\"\n", + "AZURE_AI_SEARCH_SERVICE_NAME = \"\"\n", + "AZURE_AI_SEARCH_ENDPOINT = \"\"\n", + "AZURE_AI_SEARCH_API_KEY = \"\"\n", "```" ] }, @@ -52,11 +53,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Setup Azure OpenAI\n", - "\n", - "We'll start as usual by defining our Azure OpenAI service API key and endpoint details, specifying the model deployment we want to use and then we'll initiate a connection to the Azure OpenAI service.\n", - "\n", - "**NOTE**: As with previous labs, we'll use the values from the `.env` file in the root of this repository." + "## Load environment variable values\n", + "As with previous labs, we'll use the values from the `.env` file in the root of this repository." ] }, { @@ -70,20 +68,21 @@ "\n", "# Load environment variables\n", "if load_dotenv():\n", - " print(\"Found OpenAPI Base Endpoint: \" + os.getenv(\"OPENAI_API_BASE\"))\n", + " print(\"This lab exercise will use the following values:\")\n", + " print(\"Azure OpenAI Endpoint: \" + os.getenv(\"AZURE_OPENAI_ENDPOINT\"))\n", + " print(\"Azure AI Search: \" + os.getenv(\"AZURE_AI_SEARCH_SERVICE_NAME\"))\n", "else: \n", " print(\"No file .env found\")\n", "\n", - "openai_api_type = os.getenv(\"OPENAI_API_TYPE\")\n", - "openai_api_key = os.getenv(\"OPENAI_API_KEY\")\n", - "openai_api_base = os.getenv(\"OPENAI_API_BASE\")\n", + "azure_openai_api_key = os.getenv(\"AZURE_OPENAI_API_KEY\")\n", + "azure_openai_endpoint = os.getenv(\"AZURE_OPENAI_ENDPOINT\")\n", "openai_api_version = os.getenv(\"OPENAI_API_VERSION\")\n", - "deployment_name = os.getenv(\"AZURE_OPENAI_COMPLETION_DEPLOYMENT_NAME\")\n", - "embedding_name = os.getenv(\"AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME\")\n", - "acs_service_name = os.getenv(\"AZURE_COGNITIVE_SEARCH_SERVICE_NAME\")\n", - "acs_endpoint_name = os.getenv(\"AZURE_COGNITIVE_SEARCH_ENDPOINT_NAME\")\n", - "acs_index_name = os.getenv(\"AZURE_COGNITIVE_SEARCH_INDEX_NAME\")\n", - "acs_api_key = os.getenv(\"AZURE_COGNITIVE_SEARCH_API_KEY\")" + "azure_openai_completion_deployment_name = os.getenv(\"AZURE_OPENAI_COMPLETION_DEPLOYMENT_NAME\")\n", + "azure_openai_embedding_deployment_name = os.getenv(\"AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME\")\n", + "azure_ai_search_name = os.getenv(\"AZURE_AI_SEARCH_SERVICE_NAME\")\n", + "azure_ai_search_endpoint = os.getenv(\"AZURE_AI_SEARCH_ENDPOINT\")\n", + "azure_ai_search_index_name = os.getenv(\"AZURE_AI_SEARCH_INDEX_NAME\")\n", + "azure_ai_search_api_key = os.getenv(\"AZURE_AI_SEARCH_API_KEY\")" ] }, { @@ -91,7 +90,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "First, we will load the data from the movies.csv file using the Langchain CSV document loader." + "First, we will load the data from the movies.csv file and then extract a subset to load into the Azure AI Search index. We do this to help avoid the Azure OpenAI embedding limits and long loading times when inserting data into the index. We use a Langchain document loader to do this." ] }, { @@ -102,12 +101,15 @@ "source": [ "from langchain.document_loaders.csv_loader import CSVLoader\n", "\n", - "# Movie Fields in CSV\n", - "# id,original_language,original_title,popularity,release_date,vote_average,vote_count,genre,overview,revenue,runtime,tagline\n", "loader = CSVLoader(file_path='./movies.csv', source_column='original_title', encoding='utf-8', csv_args={'delimiter':',', 'fieldnames': ['id', 'original_language', 'original_title', 'popularity', 'release_date', 'vote_average', 'vote_count', 'genre', 'overview', 'revenue', 'runtime', 'tagline']})\n", "data = loader.load()\n", - "data = data[1:51] # reduce dataset if you want\n", - "print('Loaded %s movies' % len(data))" + "\n", + "# Rather than load all 500 movies into Azure AI search, we will use a\n", + "# smaller subset of movie data to make things quicker. The more movies you load,\n", + "# the more time it will take for embeddings to be generated.\n", + "\n", + "data = data[1:51]\n", + "print('Loaded %s movies.' % len(data))" ] }, { @@ -115,7 +117,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Next, we will create an Azure OpenAI embedding and completion deployments in order to create the vector representation of the movies so we can start asking our questions." + "During this lab, we will need to work with embeddings. We use embeddings to create a vector representation of a piece of text. We will need to create embeddings for the documents we want to store in our Azure AI Search index and also for the queries we want to use to search the index. We will create an Azure OpenAI client to do this." ] }, { @@ -124,45 +126,20 @@ "metadata": {}, "outputs": [], "source": [ - "from langchain.embeddings.openai import OpenAIEmbeddings\n", - "from langchain.chat_models import AzureChatOpenAI\n", - "\n", - "# Create an Embeddings Instance of Azure OpenAI\n", - "embeddings = OpenAIEmbeddings(\n", - " model=\"text-embedding-ada-002\",\n", - " deployment=embedding_name,\n", - " openai_api_type = openai_api_type,\n", - " openai_api_version = openai_api_version,\n", - " openai_api_base = openai_api_base,\n", - " openai_api_key = openai_api_key,\n", - " embedding_ctx_length=8191,\n", - " chunk_size=1000,\n", - " max_retries=6\n", - ")\n", + "from langchain_openai import AzureOpenAIEmbeddings\n", "\n", - "# Create a Completion Instance of Azure OpenAI\n", - "llm = AzureChatOpenAI(\n", - " model=\"gpt-3.5-turbo\",\n", - " deployment_name = deployment_name,\n", - " openai_api_type = openai_api_type,\n", - " openai_api_version = openai_api_version,\n", - " openai_api_base = openai_api_base,\n", - " openai_api_key = openai_api_key,\n", - " temperature=0.7,\n", - " max_retries=6,\n", - " max_tokens=4000\n", - ")\n", - "\n", - "print('Completed creation of embedding and completion instances.')" + "azure_openai_embeddings = AzureOpenAIEmbeddings(\n", + " azure_deployment = os.getenv(\"AZURE_OPENAI_EMBEDDINGS_DEPLOYMENT_NAME\")\n", + ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## Load Movies into Azure Cognitive Search\n", + "## Create an Azure AI Search index and load movie data\n", "\n", - "Next, we'll create the Azure Cognitive Search index, embed the loaded movies from the CSV file, and upload the data into the newly created index. Depending on the number of movies loaded and rate limiting, this might take a while to do the embeddings so be patient." + "Next, we'll step through the process of configuring an Azure AI Search index to store our movie data and then loading the data into the index. " ] }, { @@ -172,84 +149,139 @@ "outputs": [], "source": [ "from azure.core.credentials import AzureKeyCredential\n", + "from azure.search.documents import SearchClient\n", "from azure.search.documents.indexes import SearchIndexClient\n", "from azure.search.documents.indexes.models import (\n", - " SearchIndex,\n", - " SearchField,\n", - " SearchFieldDataType,\n", + " VectorSearch,\n", + " VectorSearchProfile,\n", + " HnswAlgorithmConfiguration,\n", + " SemanticPrioritizedFields,\n", + " SemanticSearch,\n", + " SemanticField,\n", + " SemanticConfiguration,\n", " SimpleField,\n", " SearchableField,\n", - " SearchIndex,\n", - " SemanticConfiguration,\n", - " PrioritizedFields,\n", - " SemanticField,\n", " SearchField,\n", - " SemanticSettings,\n", - " VectorSearch,\n", - " HnswVectorSearchAlgorithmConfiguration,\n", + " SearchFieldDataType,\n", + " SearchIndex\n", ")\n", + "from azure.search.documents.models import (\n", + " VectorizedQuery\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "When configuring an Azure AI Search index, we need to specify the fields we want to store in the index and the data types for each field. These match the fields in the movie data, containing values such as the movie title, genre, year of release and so on.\n", "\n", - "# Let's Create the Azure Cognitive Search Index\n", - "index_client = SearchIndexClient(\n", - " acs_endpoint_name,\n", - " AzureKeyCredential(acs_api_key)\n", - ")\n", - "# Movie Fields in CSV\n", - "# id,original_language,original_title,popularity,release_date,vote_average,vote_count,genre,overview,revenue,runtime,tagline\n", + "To use Azure AI Search as a vector store, we will also need to define a field to hold the vector representaion of the movie data. We indicate to Azure AI Search that this field will contain vector data by providing details of the vector dimensions and a profile. We'll also define the vector search configuration and profile with default values.\n", + "\n", + "**NOTE:** It is possible just to use Azure AI Search as a vector store only, in which case we probably wouldn't need to define all of the index fields below. However, in this lab, we're also going to demonstrate Hybrid Search, a feature which makes use of both traditional keyword based search in combination with vector search." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ "fields = [\n", " SimpleField(name=\"id\", type=SearchFieldDataType.String, key=True, sortable=True, filterable=True, facetable=True),\n", " SearchableField(name=\"title\", type=SearchFieldDataType.String),\n", + " SearchableField(name=\"overview\", type=SearchFieldDataType.String),\n", + " SearchableField(name=\"genre\", type=SearchFieldDataType.String),\n", " SearchableField(name=\"tagline\", type=SearchFieldDataType.String),\n", + " SearchableField(name=\"release_date\", type=SearchFieldDataType.DateTimeOffset, sortable=True),\n", " SearchableField(name=\"popularity\", type=SearchFieldDataType.Double, sortable=True),\n", - " SearchableField(name=\"content\", type=SearchFieldDataType.String),\n", - " SearchField(name=\"content_vector\", type=SearchFieldDataType.Collection(SearchFieldDataType.Single), searchable=True, vector_search_dimensions=1536, vector_search_configuration=\"my-vector-config\"),\n", + " SearchableField(name=\"vote_average\", type=SearchFieldDataType.Double, sortable=True),\n", + " SearchableField(name=\"vote_count\", type=SearchFieldDataType.Int32, sortable=True),\n", + " SearchableField(name=\"runtime\", type=SearchFieldDataType.Int32, sortable=True),\n", + " SearchableField(name=\"revenue\", type=SearchFieldDataType.Int64, sortable=True),\n", + " SearchableField(name=\"original_language\", type=SearchFieldDataType.String),\n", + " SearchField(name=\"vector\", type=SearchFieldDataType.Collection(SearchFieldDataType.Single), searchable=True, vector_search_dimensions=1536, vector_search_profile_name=\"movies-vector-profile\"),\n", "]\n", "\n", - "# Configure Vector Search Configuration\n", "vector_search = VectorSearch(\n", - " algorithm_configurations=[\n", - " HnswVectorSearchAlgorithmConfiguration(\n", - " name=\"my-vector-config\",\n", - " kind=\"hnsw\",\n", - " parameters={\n", - " \"m\": 4,\n", - " \"efConstruction\": 400,\n", - " \"efSearch\": 500,\n", - " \"metric\": \"cosine\"\n", - " }\n", - " )\n", - " ]\n", - ")\n", - "\n", - "# Configure Semantic Configuration\n", + " profiles=[VectorSearchProfile(name=\"movies-vector-profile\", algorithm_configuration_name=\"movies-vector-config\")],\n", + " algorithms=[HnswAlgorithmConfiguration(name=\"movies-vector-config\")],\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We're going to be using Semantic Ranking, a feature of Azure AI Search that improves search results by using language understanding to rerank the search results. We provide a Semantic Search Configuration to help the ranking model understand the movie data, by telling it which fields contain the movie title, which fields contain keywords and which fields contain general free text content." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ "semantic_config = SemanticConfiguration(\n", - " name=\"my-semantic-config\",\n", - " prioritized_fields=PrioritizedFields(\n", + " name=\"movies-semantic-config\",\n", + " prioritized_fields=SemanticPrioritizedFields(\n", " title_field=SemanticField(field_name=\"title\"),\n", - " prioritized_keywords_fields=[SemanticField(field_name=\"title\"), SemanticField(field_name=\"tagline\")],\n", - " prioritized_content_fields=[SemanticField(field_name=\"content\")]\n", + " keywords_fields=[SemanticField(field_name=\"genre\")],\n", + " content_fields=[SemanticField(field_name=\"title\"),\n", + " SemanticField(field_name=\"overview\"),\n", + " SemanticField(field_name=\"tagline\"),\n", + " SemanticField(field_name=\"genre\"),\n", + " SemanticField(field_name=\"release_date\"),\n", + " SemanticField(field_name=\"popularity\"),\n", + " SemanticField(field_name=\"vote_average\"),\n", + " SemanticField(field_name=\"vote_count\"),\n", + " SemanticField(field_name=\"runtime\"),\n", + " SemanticField(field_name=\"revenue\"),\n", + " SemanticField(field_name=\"original_language\")],\n", " )\n", ")\n", "\n", - "# Create the semantic settings with the configuration\n", - "semantic_settings = SemanticSettings(configurations=[semantic_config])\n", - "\n", + "semantic_search = SemanticSearch(configurations=[semantic_config])" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Finally, we'll go ahead and create the index by creating an instance of the `SearchIndex` class and adding the keyword and vectors fields and the semantic search profile." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ "# Create the search index with the desired vector search and semantic configurations\n", "index = SearchIndex(\n", - " name=acs_index_name,\n", + " name=azure_ai_search_index_name,\n", " fields=fields,\n", " vector_search=vector_search,\n", - " semantic_settings=semantic_settings\n", + " semantic_search=semantic_search\n", ")\n", + "\n", + "index_client = SearchIndexClient(\n", + " azure_ai_search_endpoint,\n", + " AzureKeyCredential(azure_ai_search_api_key)\n", + ")\n", + "\n", "result = index_client.create_or_update_index(index)\n", - "print(f'The {result.name} index was created.')" + "\n", + "print(f'Index {result.name} created.')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "Next we will create the document structure needed to upload the data into the Azure Cognitive Search index." + "The index is now ready, so next we need to prepare the movie data to load into the index.\n", + "\n", + "**NOTE**: During this phase, we send the data for each movie to an Azure OpenAI embeddings model to create the vector data. This may take some time due to rate limiting in the API." ] }, { @@ -258,32 +290,51 @@ "metadata": {}, "outputs": [], "source": [ - "# Now that the index is created, let's load the documents into it.\n", - "\n", - "import uuid\n", + "# Loop through all of the movies and create a new item for each one.\n", "\n", - "# Let's take a quick look at the data structure of the CSVLoader\n", - "print(data[0])\n", - "print(data[0].metadata['source'])\n", - "print(\"----------\")\n", - "\n", - "# Generate Document Embeddings for page_content field in the movies CSVLoader dataset using Azure OpenAI\n", "items = []\n", "for movie in data:\n", " content = movie.page_content\n", - " items.append(dict([(\"id\", str(uuid.uuid4())), (\"title\", movie.metadata['source']), (\"content\", content), (\"content_vector\", embeddings.embed_query(content))]))\n", - "\n", - "# Print out a sample item to validate the updated data structure.\n", - "# It should have the id, content, and content_vector values.\n", - "print(items[0])\n", - "print(f\"Movie Count: {len(items)}\")" + " fields = movie.page_content.split('\\n')\n", + " movieId = (fields[0].split(': ')[1])[:-2]\n", + " movieTitle = (fields[2].split(': ')[1])\n", + " movieOverview = (fields[8].split(': ')[1])\n", + " movieGenre = (fields[7].split(': ')[1])[1:-1]\n", + " movieTagline = (fields[11].split(': ')[1])\n", + " movieReleaseDate = (fields[4].split(': ')[1])\n", + " moviePopularity = (fields[3].split(': ')[1])\n", + " movieVoteAverage = (fields[5].split(': ')[1])\n", + " movieVoteCount = (fields[6].split(': ')[1])\n", + " movieRuntime = (fields[10].split(': ')[1])\n", + " movieRevenue = (fields[9].split(': ')[1])\n", + " movieOriginalLanguage = (fields[1].split(': ')[1])\n", + "\n", + " items.append(dict([\n", + " (\"id\", movieId), \n", + " (\"title\", movieTitle),\n", + " (\"overview\", movieOverview),\n", + " (\"genre\", movieGenre),\n", + " (\"tagline\", movieTagline),\n", + " (\"release_date\", movieReleaseDate),\n", + " (\"popularity\", moviePopularity),\n", + " (\"vote_average\", movieVoteAverage),\n", + " (\"vote_count\", movieVoteCount),\n", + " (\"runtime\", movieRuntime),\n", + " (\"revenue\", movieRevenue),\n", + " (\"original_language\", movieOriginalLanguage),\n", + " (\"vector\", azure_openai_embeddings.embed_query(content))\n", + " ]))\n", + "\n", + " print(f\"Movie {movieTitle} added.\")\n", + "\n", + "print(f\"New items structure with embeddings created for {len(items)} movies.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "Next we will upload the movie documents in the newly created structure to the Azure Cognitive Search index." + "We can write out the contents of one of the documents to see what it looks like. You can see that it contains the movie data at the top and then a long array containing the vector data." ] }, { @@ -292,28 +343,14 @@ "metadata": {}, "outputs": [], "source": [ - "# Upload movies to Azure Cognitive Search index.\n", - "from azure.search.documents.models import Vector\n", - "from azure.search.documents import SearchClient\n", - "\n", - "# Insert Text and Embeddings into the Azure Cognitive Search index created.\n", - "search_client = SearchClient(\n", - " acs_endpoint_name,\n", - " acs_index_name,\n", - " AzureKeyCredential(acs_api_key)\n", - ")\n", - "result = search_client.upload_documents(items)\n", - "print(\"Successfully added documents to Azure Cognitive Search index.\")\n", - "print(f\"Uploaded {len(data)} documents\")" + "print(items[0])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## Vector Store Searching using Azure Cognitive Search\n", - "\n", - "Now that we have the movies loaded into Azure Cognitive Search, let's do some different types of searches using the Azure Cognitive Search SDK." + "Now we have the movie data stored in the correct format, so let's load it into the Azure AI Search index we created earlier." ] }, { @@ -322,30 +359,28 @@ "metadata": {}, "outputs": [], "source": [ - "# First, let's do a plain vanilla text search, no vectors or embeddings.\n", - "query = \"What are the best 80s movies I should look at?\"\n", + "from azure.search.documents import SearchClient\n", "\n", "search_client = SearchClient(\n", - " acs_endpoint_name,\n", - " acs_index_name,\n", - " AzureKeyCredential(acs_api_key)\n", + " azure_ai_search_endpoint,\n", + " azure_ai_search_index_name,\n", + " AzureKeyCredential(azure_ai_search_api_key)\n", ")\n", "\n", - "# Execute the search\n", - "results = list(search_client.search(\n", - " search_text=query,\n", - " include_total_count=True,\n", - " top=5\n", - "))\n", + "result = search_client.upload_documents(items)\n", "\n", - "# Print count of total results.\n", - "print(f\"Returned {len(results)} results using only text-based search.\")\n", - "print(\"----------\")\n", - "# Iterate over Results\n", - "# Index Fields - id, content, content_vector\n", - "for result in results:\n", - " print(\"Movie: {}\".format(result[\"content\"]))\n", - " print(\"----------\")" + "print(f\"Successfully loaded {len(data)} movies into Azure AI Search index.\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Vector store searching using Azure AI Search\n", + "\n", + "We've loaded the movies into Azure AI Search, so now let's experiment with some of the different types of searches you can perform.\n", + "\n", + "First we'll just perform a simple keyword search." ] }, { @@ -354,36 +389,18 @@ "metadata": {}, "outputs": [], "source": [ - "# Now let's do a vector search that uses the embeddings we created and inserted into content_vector field in the index.\n", - "query = \"What are the best 80s movies I should look at?\"\n", + "query = \"hero\"\n", "\n", - "search_client = SearchClient(\n", - " acs_endpoint_name,\n", - " acs_index_name,\n", - " AzureKeyCredential(acs_api_key)\n", - ")\n", - "\n", - "# You can see here that we are getting the embedding representation of the query.\n", - "vector = Vector(\n", - " value=embeddings.embed_query(query),\n", - " k=5,\n", - " fields=\"content_vector\"\n", - ")\n", - "\n", - "# Execute the search\n", "results = list(search_client.search(\n", - " search_text=\"\",\n", + " search_text=query,\n", + " query_type=\"simple\",\n", " include_total_count=True,\n", - " vectors=[vector],\n", - " select=[\"id\", \"content\", \"title\"],\n", + " top=5\n", "))\n", "\n", - "# Print count of total results.\n", - "print(f\"Returned {len(results)} results using only vector-based search.\")\n", - "print(\"----------\")\n", - "# Iterate over results and print out the content.\n", "for result in results:\n", - " print(result[\"title\"])\n", + " print(\"Movie: {}\".format(result[\"title\"]))\n", + " print(\"Genre: {}\".format(result[\"genre\"]))\n", " print(\"----------\")" ] }, @@ -391,9 +408,9 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Did that return what you expected? Probably not, let's dig deeper to see why.\n", + "We get some results, but they're not necessarily movies about heroes. It could be that there is some text in the index for these results that relates to the word \"hero\". For example, the description might mention \"heroic deeds\" or something similar.\n", "\n", - "Let's do the same search again, but this time let's return the **Search Score** so we can see the value returned by the cosine similarity vector store calculation." + "Let's now try the same again, but this time we'll ask a question instead of just searching for a keyword." ] }, { @@ -402,38 +419,18 @@ "metadata": {}, "outputs": [], "source": [ - "# Try again, but this time let's add the relevance score to maybe see why\n", - "query = \"What are the best 80s movies I should look at?\"\n", + "query = \"What are the best movies about superheroes?\"\n", "\n", - "search_client = SearchClient(\n", - " acs_endpoint_name,\n", - " acs_index_name,\n", - " AzureKeyCredential(acs_api_key)\n", - ")\n", - "\n", - "# You can see here that we are getting the embedding representation of the query.\n", - "vector = Vector(\n", - " value=embeddings.embed_query(query),\n", - " k=5,\n", - " fields=\"content_vector\"\n", - ")\n", - "\n", - "# Execute the search\n", "results = list(search_client.search(\n", - " search_text=\"\",\n", + " search_text=query,\n", + " query_type=\"simple\",\n", " include_total_count=True,\n", - " vectors=[vector],\n", - " select=[\"id\", \"content\", \"title\"],\n", + " top=5\n", "))\n", "\n", - "# Print count of total results.\n", - "print(f\"Returned {len(results)} results using vector search.\")\n", - "print(\"----------\")\n", - "# Iterate over results and print out the id and search score.\n", - "for result in results: \n", - " print(f\"Id: {result['id']}\")\n", - " print(f\"Id: {result['title']}\")\n", - " print(f\"Score: {result['@search.score']}\")\n", + "for result in results:\n", + " print(\"Movie: {}\".format(result[\"title\"]))\n", + " print(\"Genre: {}\".format(result[\"genre\"]))\n", " print(\"----------\")" ] }, @@ -441,7 +438,9 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "If you look at the Search Score you will see the relevant ranking of the closest vector match to the query inputted. The lower the score the farther apart the two vectors are. Let's change the search term and see if we can get a higher Search Score which means a higher match and closer vector proximity." + "As before, you will likely get mixed results. Some of the movies returned could be about heroes, but others may not be. This is because the search is still based on keywords.\n", + "\n", + "Next, let's try a vector search." ] }, { @@ -450,38 +449,24 @@ "metadata": {}, "outputs": [], "source": [ - "# Try again, but this time let's add the relevance score to maybe see why\n", - "query = \"Who are the actors in the movie Hidden Figures?\"\n", + "query = \"What are the best movies about superheroes?\"\n", "\n", - "search_client = SearchClient(\n", - " acs_endpoint_name,\n", - " acs_index_name,\n", - " AzureKeyCredential(acs_api_key)\n", - ")\n", + "vector = VectorizedQuery(vector=azure_openai_embeddings.embed_query(query), k_nearest_neighbors=5, fields=\"vector\")\n", "\n", - "# You can see here that we are getting the embedding representation of the query.\n", - "vector = Vector(\n", - " value=embeddings.embed_query(query),\n", - " k=5,\n", - " fields=\"content_vector\"\n", - ")\n", + "# Note the `None` value for the `search_text` parameter. This is because we're not sending the query text to Azure AI Search. We're sending the embedded version of the query text instead via the `vector_queries` parameter.\n", "\n", - "# Execute the search\n", "results = list(search_client.search(\n", - " search_text=\"\",\n", - " include_total_count=True,\n", - " vectors=[vector],\n", - " select=[\"id\", \"content\", \"title\"],\n", + " search_text=None,\n", + " query_type=\"semantic\",\n", + " semantic_configuration_name=\"movies-semantic-config\",\n", + " vector_queries=[vector],\n", + " select=[\"title\", \"genre\"],\n", + " top=5\n", "))\n", "\n", - "# Print count of total results.\n", - "print(f\"Returned {len(results)} results using vector search.\")\n", - "print(\"----------\")\n", - "# Iterate over results and print out the id and search score.\n", - "for result in results: \n", - " print(f\"Id: {result['id']}\")\n", - " print(f\"Id: {result['title']}\")\n", - " print(f\"Score: {result['@search.score']}\")\n", + "for result in results:\n", + " print(\"Movie: {}\".format(result[\"title\"]))\n", + " print(\"Genre: {}\".format(result[\"genre\"]))\n", " print(\"----------\")" ] }, @@ -489,18 +474,15 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "**NOTE:** As you have seen from the results, different inputs can return different results, it all depends on what data is in the Vector Store. The higher the score the higher the likelihood of a match." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Hybrid Searching using Azure Cognitive Search\n", + "It's likely that the raw vector search didn't return exactly what you were expecting. You were probably expecting a list of superhero movies, but now we're getting a list of movies that are **similar** to the vector we provided. Some of these may be hero movies, but others may not be. The vector search is returning the nearest neighbours to the vector we provided, so it's possible that at least one of the results is a superhero movie, and the others are similar to that movie in some way.\n", + "\n", + "So, both the keyword search and the vector search have their limitations. The keyword search is limited to the keywords in the index, so it's possible that we might miss some movies that are about heroes. The vector search is limited to returning the nearest neighbours to the vector we provide, so it's possible that we might get some movies that are not about heroes.\n", + "\n", + "## Hybrid search using Azure AI Search\n", "\n", - "What is Hybrid Search? The search is implemented at the field level, which means you can build queries that include vector fields and searchable text fields. The queries execute in parallel and the results are merged into a single response. Optionally, add semantic search, currently in preview, for even more accuracy with L2 reranking using the same language models that power Bing.\n", + "To overcome the limitations of both keyword search and vector search, we can use a combination of both. This is known as Hybrid Search. Let's run the same query again, but this time we'll use Hybrid Search.\n", "\n", - "**NOTE:** Hybrid Search is a key value proposition of Azure Cognitive Search in comparison to vector only data stores. Click [Hybrid Search](https://learn.microsoft.com/en-us/azure/search/hybrid-search-overview) for more details." + "The only significant difference is that this time we will submit both the original query text and the embedding vector to Azure AI Search. Azure AI Search will then use both the query text and the vector to perform the search and combine the results." ] }, { @@ -509,86 +491,36 @@ "metadata": {}, "outputs": [], "source": [ - "# Hybrid Search\n", - "# Let's try our original query again using Hybrid Search (ie. Combination of Text & Vector Search)\n", - "query = \"What are the best 80s movies I should look at?\"\n", + "query = \"What are the best movies about superheroes?\"\n", "\n", - "search_client = SearchClient(\n", - " acs_endpoint_name,\n", - " acs_index_name,\n", - " AzureKeyCredential(acs_api_key)\n", - ")\n", + "vector = VectorizedQuery(vector=azure_openai_embeddings.embed_query(query), k_nearest_neighbors=5, fields=\"vector\")\n", "\n", - "# You can see here that we are getting the embedding representation of the query.\n", - "vector = Vector(\n", - " value=embeddings.embed_query(query),\n", - " k=5,\n", - " fields=\"content_vector\"\n", - ")\n", + "# Note the `None` value for the `search_text` parameter. This is because we're not sending the query text to Azure AI Search. We're sending the embedded version of the query text instead via the `vector_queries` parameter.\n", "\n", - "# Notice we also fill in the search_text parameter with the query.\n", "results = list(search_client.search(\n", " search_text=query,\n", - " include_total_count=True,\n", - " top=10,\n", - " vectors=[vector],\n", - " select=[\"id\", \"content\", \"title\"],\n", + " query_type=\"semantic\",\n", + " semantic_configuration_name=\"movies-semantic-config\",\n", + " vector_queries=[vector],\n", + " select=[\"title\", \"genre\"],\n", + " top=5\n", "))\n", "\n", - "# Print count of total results.\n", - "print(f\"Returned {len(results)} results using vector search.\")\n", - "print(\"----------\")\n", - "# Iterate over results and print out the id and search score.\n", - "for result in results: \n", - " print(f\"Id: {result['id']}\")\n", - " print(result['title'])\n", - " print(f\"Hybrid Search Score: {result['@search.score']}\")\n", + "for result in results:\n", + " print(\"Movie: {}\".format(result[\"title\"]))\n", + " print(\"Genre: {}\".format(result[\"genre\"]))\n", + " print(\"Score: {}\".format(result[\"@search.score\"]))\n", + " print(\"Reranked score: {}\".format(result[\"@search.reranker_score\"]))\n", " print(\"----------\")" ] }, { - "cell_type": "code", - "execution_count": null, + "cell_type": "markdown", "metadata": {}, - "outputs": [], "source": [ - "# Hybrid Search\n", - "# Let's try our more specific query again to see the difference in the score returned.\n", - "query = \"Who are the actors in the movie Hidden Figures?\"\n", - "\n", - "search_client = SearchClient(\n", - " acs_endpoint_name,\n", - " acs_index_name,\n", - " AzureKeyCredential(acs_api_key)\n", - ")\n", - "\n", - "# You can see here that we are getting the embedding representation of the query.\n", - "vector = Vector(\n", - " value=embeddings.embed_query(query),\n", - " k=5,\n", - " fields=\"content_vector\"\n", - ")\n", + "Hopefully, you'll now see a much better set of results. Performing a hybrid search has allowed us to combine the benefits of both keyword search and vector search. But also, Azure AI Search performs a further step when using hybrid search. It makes use of a Semantic Ranker to further improve the search results. The Semantic Ranker uses a language understanding model to understand the query text and the documents in the index and then uses this information to rerank the search results. So, after performing the keyword and vector search, Azure AI Search will then use the Semantic Ranker to re-order the search results based on the context of the original query.\n", "\n", - "# -----\n", - "# Notice we also fill in the search_text parameter with the query along with the vector.\n", - "# -----\n", - "results = list(search_client.search(\n", - " search_text=query,\n", - " include_total_count=True,\n", - " top=10,\n", - " vectors=[vector],\n", - " select=[\"id\", \"content\", \"title\"],\n", - "))\n", - "\n", - "# Print count of total results.\n", - "print(f\"Returned {len(results)} results using hybrid search.\")\n", - "print(\"----------\")\n", - "# Iterate over results and print out the id and search score.\n", - "for result in results: \n", - " print(f\"Id: {result['id']}\")\n", - " print(f\"Title: {result['title']}\")\n", - " print(f\"Hybrid Search Score: {result['@search.score']}\")\n", - " print(\"----------\")" + "In the results above, you can see a `Reranked Score`. This is the score that has been calculated by the Semantic Ranker. The `Score` is the score calculated by the keyword and vector search. You'll note that the results are returned in the order determined by the reranked score." ] }, { @@ -601,7 +533,7 @@ "1. Ask the question\n", "2. Create Prompt Template with inputs\n", "3. Get Embedding representation of inputted question\n", - "4. Use embedded version of the question to search Azure Cognitive Search (ie. The Vector Store)\n", + "4. Use embedded version of the question to search Azure AI Search (ie. The Vector Store)\n", "5. Inject the results of the search into the Prompt Template & Execute the Prompt to get the completion" ] }, @@ -613,39 +545,21 @@ "source": [ "# Implement RAG using Langchain (LC)\n", "\n", - "from langchain.embeddings.openai import OpenAIEmbeddings\n", - "from langchain.chat_models import AzureChatOpenAI\n", + "from langchain_openai import AzureOpenAIEmbeddings\n", + "from langchain_openai import AzureChatOpenAI\n", "from langchain.chains import LLMChain\n", "\n", - "# Setup Langchain\n", - "# Create an Embeddings Instance of Azure OpenAI\n", - "embeddings = OpenAIEmbeddings(\n", - " model=\"text-embedding-ada-002\",\n", - " deployment=embedding_name,\n", - " openai_api_type = openai_api_type,\n", - " openai_api_version = openai_api_version,\n", - " openai_api_base = openai_api_base,\n", - " openai_api_key = openai_api_key,\n", - " embedding_ctx_length=8191,\n", - " chunk_size=1000,\n", - " max_retries=6\n", + "\n", + "azure_openai_embeddings = AzureOpenAIEmbeddings(\n", + " azure_deployment = os.getenv(\"AZURE_OPENAI_EMBEDDINGS_DEPLOYMENT_NAME\")\n", ")\n", "\n", - "# Create a Completion Instance of Azure OpenAI\n", - "llm = AzureChatOpenAI(\n", - " model=\"gpt-3.5-turbo\",\n", - " deployment_name = deployment_name,\n", - " openai_api_type = openai_api_type,\n", - " openai_api_version = openai_api_version,\n", - " openai_api_base = openai_api_base,\n", - " openai_api_key = openai_api_key,\n", - " temperature=0.7,\n", - " max_retries=6,\n", - " max_tokens=4000\n", + "azure_openai = AzureChatOpenAI(\n", + " azure_deployment = os.getenv(\"AZURE_OPENAI_COMPLETION_DEPLOYMENT_NAME\")\n", ")\n", "\n", "# Ask the question\n", - "question = \"List the movies about ships on the water.\"\n", + "query = \"What are the best movies about superheroes?\"\n", "\n", "# Create a prompt template with variables, note the curly braces\n", "from langchain.prompts import PromptTemplate\n", @@ -656,35 +570,34 @@ "\n", " Do not use any other data.\n", " Only use the movie data below when responding.\n", + " Provide detailed information about the synopsis of the movie.\n", " {search_results}\n", " \"\"\",\n", ")\n", "\n", - "# Get Embedding for the original question\n", - "question_embedded=embeddings.embed_query(question)\n", - "\n", "# Search Vector Store\n", "search_client = SearchClient(\n", - " acs_endpoint_name,\n", - " acs_index_name,\n", - " AzureKeyCredential(acs_api_key)\n", - ")\n", - "vector = Vector(\n", - " value=question_embedded,\n", - " k=5,\n", - " fields=\"content_vector\"\n", + " azure_ai_search_endpoint,\n", + " azure_ai_search_index_name,\n", + " AzureKeyCredential(azure_ai_search_api_key)\n", ")\n", + "\n", + "vector = VectorizedQuery(vector=azure_openai_embeddings.embed_query(query), k_nearest_neighbors=5, fields=\"vector\")\n", + "\n", "results = list(search_client.search(\n", - " search_text=\"\",\n", + " search_text=query,\n", + " query_type=\"semantic\",\n", + " semantic_configuration_name=\"movies-semantic-config\",\n", " include_total_count=True,\n", - " vectors=[vector],\n", - " select=[\"title\"],\n", + " vector_queries=[vector],\n", + " select=[\"title\",\"genre\",\"overview\",\"tagline\",\"release_date\",\"popularity\",\"vote_average\",\"vote_count\",\"runtime\",\"revenue\",\"original_language\"],\n", + " top=5\n", "))\n", "\n", "# Build the Prompt and Execute against the Azure OpenAI to get the completion\n", - "chain = LLMChain(llm=llm, prompt=prompt, verbose=True)\n", - "response = chain.run({\"original_question\": question, \"search_results\": results})\n", - "print(response)" + "chain = LLMChain(llm=azure_openai, prompt=prompt, verbose=False)\n", + "response = chain.invoke({\"original_question\": query, \"search_results\": results})\n", + "print(response['text'])" ] }, { @@ -713,7 +626,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.11.5" + "version": "3.11.6" }, "orig_nbformat": 4 }, diff --git a/labs/03-orchestration/04-ACS/acs-sk-csharp.ipynb b/labs/03-orchestration/04-ACS/acs-sk-csharp.ipynb index a17abfd..5d34d0c 100644 --- a/labs/03-orchestration/04-ACS/acs-sk-csharp.ipynb +++ b/labs/03-orchestration/04-ACS/acs-sk-csharp.ipynb @@ -5,20 +5,21 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# 04 - AI Orchestration with Azure Cognitive Search\n", + "# 04 - AI Orchestration with Azure AI Search\n", + "**(Semantic Kernel / C# version)**\n", "\n", - "In this lab, we will do a deeper dive around the Azure Cognitive Search vector store and different ways to interact with it." + "In this lab, we will do a deeper dive into using Azure AI Search as a vector store, the different search methods it supports and how you can use it as part of the Retrieval Augmented Generation (RAG) pattern for working with large language models." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## Create Azure Cognitive Search Vector Store in Azure\n", + "## Create an Azure AI Search Vector Store in Azure\n", "\n", - "First, we need to create an Azure Cognitive Search service in Azure, which will act as a vector store. We'll use the Azure CLI to do this.\n", + "First, we will create an Azure AI Search service in Azure. We'll use the Azure CLI to do this, so you'll need to cut and paste the following commands into a terminal window.\n", "\n", - "**NOTE:** Update **``** to make the name unique." + "**NOTE:** Before running the commands, replace the **``** with your own initials or some random characters, as we need to provide a unique name for the Azure AI Search service." ] }, { @@ -37,10 +38,11 @@ }, "outputs": [], "source": [ - "// Execute the following commands using the Azure CLI to create the Azure Cognitive Search resource in Azure.\n", - "RESOURCE_GROUP=\"azure-cognitive-search-rg\"\n", + "// Execute the following commands using the Azure CLI to create the Azure AI Search resource in Azure.\n", + "\n", + "RESOURCE_GROUP=\"azure-ai-search-rg\"\n", "LOCATION=\"westeurope\"\n", - "NAME=\"acs-vectorstore-\"\n", + "NAME=\"ai-vectorstore-\"\n", "az group create --name $RESOURCE_GROUP --location $LOCATION\n", "az search service create -g $RESOURCE_GROUP -n $NAME -l $LOCATION --sku Basic --partition-count 1 --replica-count 1" ] @@ -49,13 +51,13 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Next, we need to find and update the following values in the `.env` file with the Azure Cognitive Search **endpoint**, **admin key**, and **index name** values. Use the Azure Portal or CLI.\n", + "Next, we need to find and update the following values in the `.env` file with the Azure AI Search **name**, **endpoint** and **admin key** values, which you can get from the Azure portal. You also need to provide an **index name** value. The index will be created during this lab, so you can use any name you like.\n", "\n", "```\n", - "AZURE_COGNITIVE_SEARCH_SERVICE_NAME = \"\"\n", - "AZURE_COGNITIVE_SEARCH_ENDPOINT_NAME = \"\"\n", - "AZURE_COGNITIVE_SEARCH_API_KEY = \"\"\n", + "AZURE_AI_SEARCH_SERVICE_NAME = \"\"\n", + "AZURE_AI_SEARCH_ENDPOINT = \"\"\n", + "AZURE_AI_SEARCH_API_KEY = \"\"\n", "```" ] }, @@ -63,11 +65,9 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Setup Azure OpenAI\n", - "\n", - "We'll start as usual by defining our Azure OpenAI service API key and endpoint details, specifying the model deployment we want to use and then we'll initiate a connection to the Azure OpenAI service.\n", + "## Load the required .NET packages\n", "\n", - "**NOTE**: As with previous labs, we'll use the values from the `.env` file in the root of this repository." + "This lab uses C#, so we will need to load the required .NET packages." ] }, { @@ -87,30 +87,60 @@ "outputs": [], "source": [ "// Add the Packages\n", - "#r \"nuget: DotNetEnv, 2.5.0\"\n", - "#r \"nuget: Microsoft.SemanticKernel, 0.24.230918.1-preview\"\n", - "#r \"nuget: Azure.AI.OpenAI, 1.0.0-beta.7\"\n", - "#r \"nuget: Azure.Identity, 1.10.1\"\n", - "#r \"nuget: Azure.Search.Documents, 11.5.0-beta.4\"\n", - "\n", + "#r \"nuget: dotenv.net, 3.1.2\"\n", + "#r \"nuget: Azure.AI.OpenAI, 1.0.0-beta.12\"\n", + "#r \"nuget: Azure.Identity, 1.10.4\"\n", + "#r \"nuget: Azure.Search.Documents, 11.5.0-beta.5\"" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Load environment variable values\n", + "As with previous labs, we'll use the values from the `.env` file in the root of this repository." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "dotnet_interactive": { + "language": "csharp" + }, + "polyglot_notebook": { + "kernelName": "csharp" + }, + "vscode": { + "languageId": "polyglot-notebook" + } + }, + "outputs": [], + "source": [ "using System.IO;\n", + "using dotenv.net;\n", "\n", "// Read values from .env file\n", - "DotNetEnv.Env.Load(\"../../../.env\");\n", - "\n", - "// Load values into variables\n", - "var openai_api_type = System.Environment.GetEnvironmentVariable(\"OPENAI_API_TYPE\");\n", - "var openai_api_key = System.Environment.GetEnvironmentVariable(\"OPENAI_API_KEY\");\n", - "var openai_api_base = System.Environment.GetEnvironmentVariable(\"OPENAI_API_BASE\");\n", - "var openai_api_version = System.Environment.GetEnvironmentVariable(\"OPENAI_API_VERSION\");\n", - "var deployment_name = System.Environment.GetEnvironmentVariable(\"AZURE_OPENAI_COMPLETION_DEPLOYMENT_NAME\");\n", - "var embedding_name = System.Environment.GetEnvironmentVariable(\"AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME\");\n", - "var acs_service_name = System.Environment.GetEnvironmentVariable(\"AZURE_COGNITIVE_SEARCH_SERVICE_NAME\");\n", - "var acs_endpoint_name = System.Environment.GetEnvironmentVariable(\"AZURE_COGNITIVE_SEARCH_ENDPOINT_NAME\");\n", - "var acs_index_name = System.Environment.GetEnvironmentVariable(\"AZURE_COGNITIVE_SEARCH_INDEX_NAME\");\n", - "var acs_api_key = System.Environment.GetEnvironmentVariable(\"AZURE_COGNITIVE_SEARCH_API_KEY\");\n", - "\n", - "Console.WriteLine(\"Environment Variables loaded.\");" + "var envVars = DotEnv.Fluent()\n", + " .WithoutExceptions()\n", + " .WithEnvFiles(\"../../../.env\")\n", + " .WithTrimValues()\n", + " .WithDefaultEncoding()\n", + " .WithOverwriteExistingVars()\n", + " .WithoutProbeForEnv()\n", + " .Read();\n", + "\n", + "var azure_openai_api_key = envVars[\"AZURE_OPENAI_API_KEY\"].Replace(\"\\\"\", \"\");\n", + "var azure_openai_endpoint = envVars[\"AZURE_OPENAI_ENDPOINT\"].Replace(\"\\\"\", \"\");\n", + "var azure_openai_embedding_deployment_name = envVars[\"AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME\"].Replace(\"\\\"\", \"\");\n", + "var azure_ai_search_name = envVars[\"AZURE_AI_SEARCH_SERVICE_NAME\"].Replace(\"\\\"\", \"\");\n", + "var azure_ai_search_endpoint = envVars[\"AZURE_AI_SEARCH_ENDPOINT\"].Replace(\"\\\"\", \"\");\n", + "var azure_ai_search_api_key = envVars[\"AZURE_AI_SEARCH_API_KEY\"].Replace(\"\\\"\", \"\");\n", + "var azure_ai_search_index_name = envVars[\"AZURE_AI_SEARCH_INDEX_NAME\"].Replace(\"\\\"\", \"\");\n", + "\n", + "Console.WriteLine(\"This lab exercise will use the following values:\");\n", + "Console.WriteLine($\"Azure OpenAI Endpoint: {azure_openai_endpoint}\");\n", + "Console.WriteLine($\"Azure AI Search: {azure_ai_search_name}\");" ] }, { @@ -118,7 +148,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "First, we will load the data from the movies.csv file and then extract a subset to load into the Azure Cognitive Search index. We do this to help avoid the Azure OpenAI embedding limits and long loading times when inserting data into the index." + "First, we will load the data from the movies.csv file and then extract a subset to load into the Azure AI Search index. We do this to help avoid the Azure OpenAI embedding limits and long loading times when inserting data into the index." ] }, { @@ -139,13 +169,14 @@ "source": [ "using System.IO;\n", "\n", - "// Movie Fields in CSV\n", - "// id,original_language,original_title,popularity,release_date,vote_average,vote_count,genre,overview,revenue,runtime,tagline\n", "string path = @\"./movies.csv\";\n", "string[] allMovies;\n", "string[] movieSubset;\n", - "// The subset of movies to load into the Azure Cognitive Search Index.\n", - "// The more movies you load, the longer it will take due to Embedding limits.\n", + "//\n", + "// Rather than load all 500 movies into Azure AI search, we will use a\n", + "// smaller subset of movie data to make things quicker. The more movies you load,\n", + "// the more time it will take for embeddings to be generated.\n", + "//\n", "int movieSubsetCount = 50;\n", "try\n", "{\n", @@ -161,7 +192,7 @@ "}\n", "\n", "// Write out the results.\n", - "Console.WriteLine($\"CSV File Loaded {movieSubset.Length}.\");" + "Console.WriteLine($\"Loaded {movieSubset.Length} movies.\");" ] }, { @@ -169,7 +200,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Next, we will create an Azure OpenAI Client to do embeddings and completions." + "During this lab, we will need to work with embeddings. We use embeddings to create a vector representation of a piece of text. We will need to create embeddings for the documents we want to store in our Azure AI Search index and also for the queries we want to use to search the index. We will create an Azure OpenAI client to do this." ] }, { @@ -192,19 +223,16 @@ "using Azure.AI.OpenAI;\n", "using Azure.Identity;\n", "\n", - "// Create the Azure OpenAI client.\n", - "OpenAIClient azureOpenAIClient = new OpenAIClient(new Uri(openai_api_base),new AzureKeyCredential(openai_api_key));\n", - "\n", - "Console.WriteLine(\"Azure OpenAI client Created.\");" + "OpenAIClient azureOpenAIClient = new OpenAIClient(new Uri(azure_openai_endpoint),new AzureKeyCredential(azure_openai_api_key));" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## Load Movies into Azure Cognitive Search\n", + "## Create an Azure AI Search index and load movie data\n", "\n", - "Next, we'll create the Azure Cognitive Search index, embed the loaded movies from the CSV file, and upload the data into the newly created index. Depending on the number of movies loaded and rate limiting, this might take a while to do the embeddings so be patient." + "Next, we'll step through the process of configuring an Azure AI Search index to store our movie data and then loading the data into the index. " ] }, { @@ -229,57 +257,179 @@ "using Azure.Search.Documents.Indexes.Models;\n", "using Microsoft.Extensions.Configuration;\n", "\n", - "// Setup variables.\n", - "string vectorSearchConfigName = \"my-vector-config\";\n", - "string semanticSearchConfigName = \"my-semantic-config\";\n", - "int modelDimensions = 1536;\n", + "// These variables store the names of Azure AI Search profiles and configs that we will use\n", "\n", - "// Create an Azure Cognitive Search index client.\n", - "AzureKeyCredential indexCredential = new AzureKeyCredential(acs_api_key);\n", - "SearchIndexClient indexClient = new SearchIndexClient(new Uri(acs_endpoint_name), indexCredential);\n", + "string vectorSearchHnswProfile = \"movies-vector-profile\";\n", + "string vectorSearchHnswConfig = \"movies-vector-config\";\n", + "string semanticSearchConfigName = \"movies-semantic-config\";\n", "\n", - "// Create the index definition.\n", - "// Setup the VectorSearch configuration.\n", - "VectorSearch vectorSearch = new VectorSearch();\n", - "vectorSearch.AlgorithmConfigurations.Add(new HnswVectorSearchAlgorithmConfiguration(vectorSearchConfigName));\n", - "// Setup the SemanticSettings configuration.\n", - "SemanticSettings semanticSettings = new SemanticSettings();\n", - "semanticSettings.Configurations.Add(new SemanticConfiguration(semanticSearchConfigName, new PrioritizedFields()\n", - "{\n", - " TitleField = new SemanticField() { FieldName = \"content\" },\n", - " KeywordFields = { new SemanticField() { FieldName = \"content\" } },\n", - " ContentFields = { new SemanticField() { FieldName = \"content\" } },\n", - "}));\n", - "// Setup the Fields configuration.\n", + "// The OpenAI embeddings model has 1,536 dimensions\n", + "\n", + "int modelDimensions = 1536;" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "When configuring an Azure AI Search index, we need to specify the fields we want to store in the index and the data types for each field. These match the fields in the movie data, containing values such as the movie title, genre, year of release and so on.\n", + "\n", + "**NOTE:** It is possible just to use Azure AI Search as a vector store only, in which case we probably wouldn't need to define all of the index fields below. However, in this lab, we're also going to demonstrate Hybrid Search, a feature which makes use of both traditional keyword based search in combination with vector search." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "dotnet_interactive": { + "language": "csharp" + }, + "polyglot_notebook": { + "kernelName": "csharp" + }, + "vscode": { + "languageId": "polyglot-notebook" + } + }, + "outputs": [], + "source": [ "IList fields = new List();\n", - "fields.Add(new SimpleField(\"id\", SearchFieldDataType.String) { IsKey = true, IsFilterable = true, IsSortable = true, IsFacetable = true });\n", - "fields.Add(new SearchableField(\"content\", false) {});\n", - "fields.Add(new SearchField(\"content_vector\", SearchFieldDataType.Collection(SearchFieldDataType.Single))\n", + "fields.Add(new SimpleField(\"id\", SearchFieldDataType.String) { IsKey = true, IsFilterable = true, IsSortable = true });\n", + "fields.Add(new SearchableField(\"title\", false) { IsFilterable = true, IsSortable = true });\n", + "fields.Add(new SearchableField(\"overview\", false) { IsFilterable = true, IsSortable = true });\n", + "fields.Add(new SearchableField(\"genre\", false) { IsFilterable = true, IsSortable = true, IsFacetable = true });\n", + "fields.Add(new SearchableField(\"tagline\", false) { IsFilterable = true, IsSortable = true });\n", + "fields.Add(new SearchableField(\"release_date\", false) { IsFilterable = true, IsSortable = true });\n", + "fields.Add(new SearchableField(\"popularity\", false) { IsFilterable = true, IsSortable = true });\n", + "fields.Add(new SearchableField(\"vote_average\", false) { IsFilterable = true, IsSortable = true });\n", + "fields.Add(new SearchableField(\"vote_count\", false) { IsFilterable = true, IsSortable = true });\n", + "fields.Add(new SearchableField(\"runtime\", false) { IsFilterable = true, IsSortable = true });\n", + "fields.Add(new SearchableField(\"revenue\", false) { IsFilterable = true, IsSortable = true });\n", + "fields.Add(new SearchableField(\"original_language\", false) { IsFilterable = true, IsSortable = true });" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "To use Azure AI Search as a vector store, we will need to define a field to hold the vector representaion of the movie data. We indicate to Azure AI Search that this field will contain vector data by providing details of the vector dimensions and a profile. We'll also define the vector search configuration and profile with default values." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "dotnet_interactive": { + "language": "csharp" + }, + "polyglot_notebook": { + "kernelName": "csharp" + }, + "vscode": { + "languageId": "polyglot-notebook" + } + }, + "outputs": [], + "source": [ + "fields.Add(new SearchField(\"vector\", SearchFieldDataType.Collection(SearchFieldDataType.Single))\n", "{\n", " IsSearchable = true,\n", " VectorSearchDimensions = modelDimensions,\n", - " VectorSearchConfiguration = vectorSearchConfigName\n", + " VectorSearchProfile = vectorSearchHnswProfile,\n", "});\n", "\n", + "VectorSearch vectorSearch = new VectorSearch();\n", + "vectorSearch.Algorithms.Add(new HnswVectorSearchAlgorithmConfiguration(vectorSearchHnswConfig));\n", + "vectorSearch.Profiles.Add(new VectorSearchProfile(vectorSearchHnswProfile, vectorSearchHnswConfig));" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We're going to be using Semantic Ranking, a feature of Azure AI Search that improves search results by using language understanding to rerank the search results. We configure Semantic Settings to help the ranking model understand the movie data, by telling it which fields contain the movie title, which fields contain keywords and which fields contain general free text content." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "dotnet_interactive": { + "language": "csharp" + }, + "polyglot_notebook": { + "kernelName": "csharp" + }, + "vscode": { + "languageId": "polyglot-notebook" + } + }, + "outputs": [], + "source": [ + "SemanticSettings semanticSettings = new SemanticSettings();\n", + "semanticSettings.Configurations.Add(new SemanticConfiguration(semanticSearchConfigName, new PrioritizedFields()\n", + "{\n", + " TitleField = new SemanticField() { FieldName = \"title\" },\n", + " KeywordFields = { new SemanticField() { FieldName = \"genre\" } },\n", + " ContentFields = { new SemanticField() { FieldName = \"title\" },\n", + " new SemanticField() { FieldName = \"overview\" },\n", + " new SemanticField() { FieldName = \"tagline\" },\n", + " new SemanticField() { FieldName = \"genre\" },\n", + " new SemanticField() { FieldName = \"release_date\" },\n", + " new SemanticField() { FieldName = \"popularity\" },\n", + " new SemanticField() { FieldName = \"vote_average\" },\n", + " new SemanticField() { FieldName = \"vote_count\" },\n", + " new SemanticField() { FieldName = \"runtime\" },\n", + " new SemanticField() { FieldName = \"revenue\" },\n", + " new SemanticField() { FieldName = \"original_language\" }\n", + " }\n", + "}));" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Finally, we'll go ahead and create the index by creating an instance of the `SearchIndex` class and adding the keyword and vectors fields and the semantic search profile." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "dotnet_interactive": { + "language": "csharp" + }, + "polyglot_notebook": { + "kernelName": "csharp" + }, + "vscode": { + "languageId": "polyglot-notebook" + } + }, + "outputs": [], + "source": [ "// Setup SearchIndex\n", - "SearchIndex searchIndex = new SearchIndex(acs_index_name);\n", + "SearchIndex searchIndex = new SearchIndex(azure_ai_search_index_name);\n", "searchIndex.VectorSearch = vectorSearch;\n", "searchIndex.SemanticSettings = semanticSettings;\n", "searchIndex.Fields = fields;\n", "\n", - "// Create the index\n", + "// Create an Azure AI Search index client and create the index.\n", + "AzureKeyCredential indexCredential = new AzureKeyCredential(azure_ai_search_api_key);\n", + "SearchIndexClient indexClient = new SearchIndexClient(new Uri(azure_ai_search_endpoint), indexCredential);\n", "indexClient.CreateOrUpdateIndex(searchIndex);\n", "\n", - "Console.WriteLine($\"Index {acs_index_name} created.\");" + "Console.WriteLine($\"Index {azure_ai_search_index_name} created.\");" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "Next we will create the document structure needed to upload the data into the Azure Cognitive Search index.\n", + "The index is now ready, so next we need to prepare the movie data to load into the index.\n", "\n", - "**NOTE**: Be patient, creating the embeddings will take some time due to the Azure OpenAI embedding Token Per Minute (TPM) limits." + "**NOTE**: During this phase, we send the data for each movie to an Azure OpenAI embeddings model to create the vector data. This may take some time due to rate limiting in the API." ] }, { @@ -298,23 +448,48 @@ }, "outputs": [], "source": [ - "using Azure.Search.Documents;\n", - "using Azure.Search.Documents.Models;\n", + "using System.Text.RegularExpressions;\n", "\n", - "// Create structure to match the Azure Cognitive Search index.\n", + "// Loop through all of the movies and create an array of documents that we can load into the index.\n", "List movieDocuments = new List();\n", "for (int i = 0; i < movieSubset.Length; i++) \n", "{\n", - " Console.WriteLine($\"Movie {i} being added.\");\n", - " string id = System.Guid.NewGuid().ToString();\n", - " string content = movieSubset[i];\n", - " float[] contentEmbeddings = azureOpenAIClient.GetEmbeddings(embedding_name, new EmbeddingsOptions(content)).Value.Data[0].Embedding.ToArray();\n", + " // Extract the data for the movie from the CSV file into variables\n", + " var values = Regex.Split(movieSubset[i], \",(?=(?:[^\\\"]*\\\"[^\\\"]*\\\")*[^\\\"]*$)\");\n", + " var movieId = values[0];\n", + " var movieTitle = values[2];\n", + " var movieOverview = Regex.Replace(values[8], \"^\\\"|\\\"$\", \"\");\n", + " var movieGenre = values[7].Substring(2, values[7].Length - 4);\n", + " var movieTagline = Regex.Replace(values[11], \"^\\\"|\\\"$\", \"\");\n", + " var movieReleaseDate = values[4];\n", + " var movieRuntime = values[10];\n", + " var moviePopularity = values[3];\n", + " var movieVoteAverage = values[5];\n", + " var movieVoteCount = values[6];\n", + " var movieRevenue = values[9];\n", + " var movieLanguage = values[1];\n", + " \n", + " // Take the entire set of data for the movie and generate the embeddings\n", + " IEnumerable content = new List() { movieSubset[i] };\n", + " Response contentEmbeddings = azureOpenAIClient.GetEmbeddings(new EmbeddingsOptions(azure_openai_embedding_deployment_name, content));\n", + "\n", + " // Add the movie data and embeddings to a search document\n", " SearchDocument document = new SearchDocument();\n", - " document.Add(\"id\", id);\n", - " document.Add(\"content\", content);\n", - " document.Add(\"content_vector\", contentEmbeddings);\n", + " document.Add(\"id\", movieId.Substring(0, movieId.Length - 2));\n", + " document.Add(\"title\", movieTitle);\n", + " document.Add(\"overview\", movieOverview);\n", + " document.Add(\"genre\", movieGenre);\n", + " document.Add(\"tagline\", movieTagline);\n", + " document.Add(\"release_date\", movieReleaseDate);\n", + " document.Add(\"runtime\", movieRuntime);\n", + " document.Add(\"popularity\", moviePopularity);\n", + " document.Add(\"vote_average\", movieVoteAverage);\n", + " document.Add(\"vote_count\", movieVoteCount);\n", + " document.Add(\"revenue\", movieRevenue);\n", + " document.Add(\"original_language\", movieLanguage);\n", + " document.Add(\"vector\", contentEmbeddings.Value.Data[0].Embedding.ToArray());\n", " movieDocuments.Add(new SearchDocument(document));\n", - " Console.WriteLine($\"Movie {i} added.\");\n", + " Console.WriteLine($\"Movie {movieTitle} added.\");\n", "}\n", "\n", "Console.WriteLine($\"New SearchDocument structure with embeddings created for {movieDocuments.Count} movies.\");" @@ -324,7 +499,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Next we will upload the movie documents in the newly created structure to the Azure Cognitive Search index." + "We can write out the contents of one of the documents to see what it looks like. You can see that it contains the movie data at the top and then a long array containing the vector data." ] }, { @@ -343,29 +518,14 @@ }, "outputs": [], "source": [ - "using Azure.Search.Documents;\n", - "using Azure.Search.Documents.Models;\n", - "using Azure.Search.Documents.Indexes;\n", - "using Azure.Search.Documents.Indexes.Models;\n", - "\n", - "// Create an Azure Cognitive Search index client.\n", - "AzureKeyCredential indexCredential = new AzureKeyCredential(acs_api_key);\n", - "SearchIndexClient indexClient = new SearchIndexClient(new Uri(acs_endpoint_name), indexCredential);\n", - "SearchClient searchIndexClient = indexClient.GetSearchClient(acs_index_name);\n", - "\n", - "IndexDocumentsOptions options = new IndexDocumentsOptions { ThrowOnAnyError = true };\n", - "searchIndexClient.IndexDocuments(IndexDocumentsBatch.Upload(movieDocuments), options);\n", - "\n", - "Console.WriteLine($\"Successfully loaded {movieDocuments.Count} movies into Azure Cognitive Search index.\");" + "Console.WriteLine(movieDocuments[0]);" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## Vector Store Searching using Azure Cognitive Search\n", - "\n", - "Now that we have the movies loaded into Azure Cognitive Search, let's do some different types of searches using the Azure Cognitive Search SDK." + "Now we have the movie data stored in the correct format, so let's load it into the Azure AI Search index we created earlier." ] }, { @@ -384,19 +544,22 @@ }, "outputs": [], "source": [ - "// Setup an Azure Cognitive Search client for searching.\n", + "SearchClient searchIndexClient = indexClient.GetSearchClient(azure_ai_search_index_name);\n", + "IndexDocumentsOptions options = new IndexDocumentsOptions { ThrowOnAnyError = true };\n", + "searchIndexClient.IndexDocuments(IndexDocumentsBatch.Upload(movieDocuments), options);\n", "\n", - "using Azure;\n", - "using Azure.Search.Documents;\n", - "using Azure.Search.Documents.Models;\n", - "using Azure.Search.Documents.Indexes;\n", - "using System.Text.Json.Serialization;\n", + "Console.WriteLine($\"Successfully loaded {movieDocuments.Count} movies into Azure AI Search index.\");" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Vector store searching using Azure AI Search\n", "\n", - "// Create a client\n", - "AzureKeyCredential credential = new AzureKeyCredential(acs_api_key);\n", - "SearchClient searchClient = new SearchClient(new Uri(acs_endpoint_name), acs_index_name, credential);\n", + "We've loaded the movies into Azure AI Search, so now let's experiment with some of the different types of searches you can perform.\n", "\n", - "Console.WriteLine($\"Successfully created Azure Cognitive Search SearchClient.\");" + "First we'll just perform a simple keyword search." ] }, { @@ -415,43 +578,36 @@ }, "outputs": [], "source": [ - "// First, let's do a plain vanilla text search, no vectors or embeddings.\n", - "\n", - "using Azure;\n", - "using Azure.Search.Documents;\n", - "using Azure.Search.Documents.Models;\n", - "using Azure.Search.Documents.Indexes;\n", - "using System.Text.Json.Serialization;\n", - "\n", - "var query = \"What are the best 80s movies I should look at?\";\n", + "var query = \"hero\";\n", "\n", "SearchOptions searchOptions = new SearchOptions\n", "{\n", - " // Filter to only Content greater than or equal our preference\n", - " // Filter = SearchFilter.Create($\"Content ge {content}\"),\n", - " // OrderBy = { \"Content desc\" } // Sort by Content from high to low\n", - " // Size = 5, // Take only 5 results\n", - " // Select = { \"id\", \"content\", \"content_vector\" }, // Which fields to return\n", " Size = 5,\n", - " Select = { \"content\" },\n", + " Select = { \"title\", \"genre\" },\n", + " SearchMode = SearchMode.Any,\n", + " QueryType = SearchQueryType.Simple,\n", "};\n", "\n", - "SearchResults response = searchClient.Search(query, searchOptions);\n", + "SearchResults response = searchIndexClient.Search(query, searchOptions);\n", "Pageable> results = response.GetResults();\n", "\n", - "// Print count of total results.\n", - "Console.WriteLine($\"Returned {results.Count()} results using only text-based search.\");\n", - "Console.WriteLine(\"----------\");\n", - "\n", - "// Iterate over Results\n", - "// Index Fields - id, content, content_vector\n", "foreach (SearchResult result in results)\n", "{\n", - " Console.WriteLine($\"Movie: {result.Document[\"content\"]}\");\n", + " Console.WriteLine($\"Movie: {result.Document[\"title\"]}\");\n", + " Console.WriteLine($\"Genre: {result.Document[\"genre\"]}\");\n", " Console.WriteLine(\"----------\");\n", "};" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We get some results, but they're not necessarily movies about heroes. It could be that there is some text in the index for these results that relates to the word \"hero\". For example, the description might mention \"heroic deeds\" or something similar.\n", + "\n", + "Let's now try the same again, but this time we'll ask a question instead of just searching for a keyword." + ] + }, { "cell_type": "code", "execution_count": null, @@ -468,44 +624,15 @@ }, "outputs": [], "source": [ - "// Now let's do a vector search that uses the embeddings we created and inserted into content_vector field in the index.\n", - "\n", - "using Azure;\n", - "using Azure.Search.Documents;\n", - "using Azure.Search.Documents.Models;\n", - "using Azure.Search.Documents.Indexes;\n", - "using System.Text.Json.Serialization;\n", - "\n", - "var query = \"What are the best 80s movies I should look at?\";\n", - "\n", - "float[] queryEmbedding = azureOpenAIClient.GetEmbeddings(embedding_name, new EmbeddingsOptions(query)).Value.Data[0].Embedding.ToArray();\n", - "\n", - "// Note the Vectors addition in the SearchOptions\n", - "SearchOptions searchOptions = new SearchOptions\n", - "{\n", - " // Filter to only Content greater than or equal our preference\n", - " // Filter = SearchFilter.Create($\"Content ge {content}\"),\n", - " // OrderBy = { \"Content desc\" } // Sort by Content from high to low\n", - " // Size = 5, // Take only 5 results\n", - " // Select = { \"id\", \"content\", \"content_vector\" }, // Which fields to return\n", - " Vectors = { new() { Value = queryEmbedding, KNearestNeighborsCount = 5, Fields = { \"content_vector\" } } }, // Vector Search\n", - " Size = 5,\n", - " Select = { \"content\" },\n", - "};\n", + "var query = \"What are the best movies about superheroes?\";\n", "\n", - "// Note the search text is null when doing a vector search.\n", - "SearchResults response = searchClient.Search(null, searchOptions);\n", + "SearchResults response = searchIndexClient.Search(query, searchOptions);\n", "Pageable> results = response.GetResults();\n", "\n", - "// Print count of total results.\n", - "Console.WriteLine($\"Returned {results.Count()} results using only vector-based search.\");\n", - "Console.WriteLine(\"----------\");\n", - "\n", - "// Iterate over Results\n", - "// Index Fields - id, content, content_vector\n", "foreach (SearchResult result in results)\n", "{\n", - " Console.WriteLine($\"Movie: {result.Document[\"content\"]}\");\n", + " Console.WriteLine($\"Movie: {result.Document[\"title\"]}\");\n", + " Console.WriteLine($\"Genre: {result.Document[\"genre\"]}\");\n", " Console.WriteLine(\"----------\");\n", "};" ] @@ -514,9 +641,9 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Did that return what you expected? Probably not, let's dig deeper to see why.\n", + "As before, you will likely get mixed results. Some of the movies returned could be about heroes, but others may not be. This is because the search is still based on keywords.\n", "\n", - "Let's do the same search again, but this time let's return the **Search Score** so we can see the value returned by the cosine similarity vector store calculation." + "Next, let's try a vector search." ] }, { @@ -535,45 +662,31 @@ }, "outputs": [], "source": [ - "// Try again, but this time let's add the relevance score to maybe see why\n", - "\n", - "using Azure;\n", - "using Azure.Search.Documents;\n", - "using Azure.Search.Documents.Models;\n", - "using Azure.Search.Documents.Indexes;\n", - "using System.Text.Json.Serialization;\n", - "\n", - "var query = \"What are the best 80s movies I should look at?\";\n", + "var query = \"What are the best movies about superheroes?\";\n", "\n", - "float[] queryEmbedding = azureOpenAIClient.GetEmbeddings(embedding_name, new EmbeddingsOptions(query)).Value.Data[0].Embedding.ToArray();\n", + "// Convert the query to an embedding\n", + "float[] queryEmbedding = azureOpenAIClient.GetEmbeddings(new EmbeddingsOptions(azure_openai_embedding_deployment_name, new List() { query })).Value.Data[0].Embedding.ToArray();\n", "\n", - "// Note the Vectors addition in the SearchOptions\n", + "// This time we will set the search type to Semantic and we'll pass in the embedded version of the query text and parameters to configure the vector search.\n", "SearchOptions searchOptions = new SearchOptions\n", "{\n", - " // Filter to only Content greater than or equal our preference\n", - " // Filter = SearchFilter.Create($\"Content ge {content}\"),\n", - " // OrderBy = { \"Content desc\" } // Sort by Content from high to low\n", - " // Size = 5, // Take only 5 results\n", - " // Select = { \"id\", \"content\", \"content_vector\" }, // Which fields to return\n", - " Vectors = { new() { Value = queryEmbedding, KNearestNeighborsCount = 5, Fields = { \"content_vector\" } } }, // Vector Search\n", + " QueryType = SearchQueryType.Semantic,\n", + " SemanticConfigurationName = semanticSearchConfigName,\n", + " VectorQueries = { new RawVectorQuery() { Vector = queryEmbedding, KNearestNeighborsCount = 5, Fields = { \"vector\" } } },\n", " Size = 5,\n", - " Select = { \"id\", \"content\" },\n", + " Select = { \"title\", \"genre\" },\n", "};\n", "\n", - "// Note the search text is null when doing a vector search.\n", - "SearchResults response = searchClient.Search(null, searchOptions);\n", + "// Note the `null` value for the query parameter. This is because we're not sending the query text to Azure AI Search. We're sending the embedded version of the query text instead.\n", + "SearchResults response = searchIndexClient.Search(null, searchOptions);\n", "Pageable> results = response.GetResults();\n", "\n", - "// Print count of total results.\n", - "Console.WriteLine($\"Returned {results.Count()} results using only vector-based search.\");\n", - "Console.WriteLine(\"----------\");\n", - "\n", - "// Iterate over Results\n", - "// Index Fields - id, content, content_vector\n", "foreach (SearchResult result in results)\n", "{\n", - " Console.WriteLine($\"Id: {result.Document[\"id\"]}\");\n", + " Console.WriteLine($\"Movie: {result.Document[\"title\"]}\");\n", + " Console.WriteLine($\"Genre: {result.Document[\"genre\"]}\");\n", " Console.WriteLine($\"Score: {result.Score}\");\n", + " //Console.WriteLine($\"Reranked Score: {result.RerankerScore}\");\n", " Console.WriteLine(\"----------\");\n", "};" ] @@ -582,7 +695,15 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "If you look at the Search Score you will see the relevant ranking of the closest vector match to the query inputted. The lower the score the farther apart the two vectors are. Let's change the search term and see if we can get a higher Search Score which means a higher match and closer vector proximity." + "It's likely that the raw vector search didn't return exactly what you were expecting. You were probably expecting a list of superhero movies, but now we're getting a list of movies that are **similar** to the vector we provided. Some of these may be hero movies, but others may not be. The vector search is returning the nearest neighbours to the vector we provided, so it's possible that at least one of the results is a superhero movie, and the others are similar to that movie in some way.\n", + "\n", + "So, both the keyword search and the vector search have their limitations. The keyword search is limited to the keywords in the index, so it's possible that we might miss some movies that are about heroes. The vector search is limited to returning the nearest neighbours to the vector we provide, so it's possible that we might get some movies that are not about heroes.\n", + "\n", + "## Hybrid search using Azure AI Search\n", + "\n", + "To overcome the limitations of both keyword search and vector search, we can use a combination of both. This is known as Hybrid Search. Let's run the same query again, but this time we'll use Hybrid Search.\n", + "\n", + "The only significant difference is that this time we will submit both the original query text and the embedding vector to Azure AI Search. Azure AI Search will then use both the query text and the vector to perform the search and combine the results." ] }, { @@ -601,45 +722,15 @@ }, "outputs": [], "source": [ - "// Try again, but this time let's add the relevance score to maybe see why\n", - "\n", - "using Azure;\n", - "using Azure.Search.Documents;\n", - "using Azure.Search.Documents.Models;\n", - "using Azure.Search.Documents.Indexes;\n", - "using System.Text.Json.Serialization;\n", - "\n", - "var query = \"Who are the actors in the movie Hidden Figures?\";\n", - "\n", - "float[] queryEmbedding = azureOpenAIClient.GetEmbeddings(embedding_name, new EmbeddingsOptions(query)).Value.Data[0].Embedding.ToArray();\n", - "\n", - "// Note the Vectors addition in the SearchOptions\n", - "SearchOptions searchOptions = new SearchOptions\n", - "{\n", - " // Filter to only Content greater than or equal our preference\n", - " // Filter = SearchFilter.Create($\"Content ge {content}\"),\n", - " // OrderBy = { \"Content desc\" } // Sort by Content from high to low\n", - " // Size = 5, // Take only 5 results\n", - " // Select = { \"id\", \"content\", \"content_vector\" }, // Which fields to return\n", - " Vectors = { new() { Value = queryEmbedding, KNearestNeighborsCount = 5, Fields = { \"content_vector\" } } }, // Vector Search\n", - " Size = 5,\n", - " Select = { \"id\", \"content\" },\n", - "};\n", - "\n", - "// Note the search text is null when doing a vector search.\n", - "SearchResults response = searchClient.Search(null, searchOptions);\n", + "SearchResults response = searchIndexClient.Search(query, searchOptions);\n", "Pageable> results = response.GetResults();\n", "\n", - "// Print count of total results.\n", - "Console.WriteLine($\"Returned {results.Count()} results using only vector-based search.\");\n", - "Console.WriteLine(\"----------\");\n", - "\n", - "// Iterate over Results\n", - "// Index Fields - id, content, content_vector\n", "foreach (SearchResult result in results)\n", "{\n", - " Console.WriteLine($\"Id: {result.Document[\"id\"]}\");\n", + " Console.WriteLine($\"Movie: {result.Document[\"title\"]}\");\n", + " Console.WriteLine($\"Genre: {result.Document[\"genre\"]}\");\n", " Console.WriteLine($\"Score: {result.Score}\");\n", + " Console.WriteLine($\"Reranked Score: {result.RerankerScore}\");\n", " Console.WriteLine(\"----------\");\n", "};" ] @@ -648,18 +739,27 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "**NOTE:** As you have seen from the results, different inputs can return different results, it all depends on what data is in the Vector Store. The higher the score the higher the likelihood of a match." + "Hopefully, you'll now see a much better set of results. Performing a hybrid search has allowed us to combine the benefits of both keyword search and vector search. But also, Azure AI Search performs a further step when using hybrid search. It makes use of a Semantic Ranker to further improve the search results. The Semantic Ranker uses a language understanding model to understand the query text and the documents in the index and then uses this information to rerank the search results. So, after performing the keyword and vector search, Azure AI Search will then use the Semantic Ranker to re-order the search results based on the context of the original query.\n", + "\n", + "In the results above, you can see a `Reranked Score`. This is the score that has been calculated by the Semantic Ranker. The `Score` is the score calculated by the keyword and vector search. You'll note that the results are returned in the order determined by the reranked score." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## Hybrid Searching using Azure Cognitive Search\n", + "## Bringing it All Together with Retrieval Augmented Generation (RAG) + Semantic Kernel (SK)\n", + "\n", + "Now that we have our Vector Store setup and data loaded, we are now ready to implement the RAG pattern using AI Orchestration. At a high-level, the following steps are required:\n", + "1. Ask the question\n", + "2. Create Prompt Template with inputs\n", + "3. Get Embedding representation of inputted question\n", + "4. Use embedded version of the question to search Azure AI Search (ie. The Vector Store)\n", + "5. Inject the results of the search into the Prompt Template & Execute the Prompt to get the completion\n", "\n", - "What is Hybrid Search? The search is implemented at the field level, which means you can build queries that include vector fields and searchable text fields. The queries execute in parallel and the results are merged into a single response. Optionally, add semantic search, currently in preview, for even more accuracy with L2 reranking using the same language models that power Bing.\n", + "**NOTE:** Semantic Kernel has a Semantic Memory feature, as well as a separate Kernel Memory project, that both allow you to store and retrieve information from a vector store. These are both currently in preview / experimental states. In the following code, we're implementing the feature using our own code to firstly retrieve results from Azure AI Search, then add those search results to the query we send to Azure OpenAI.\n", "\n", - "**NOTE:** Hybrid Search is a key value proposition of Azure Cognitive Search in comparison to vector only data stores." + "First, let's setup the packages and load the values from the `.env` file." ] }, { @@ -678,48 +778,58 @@ }, "outputs": [], "source": [ - "// Hybrid Search\n", - "// Let's try our original query again using Hybrid Search (ie. Combination of Text & Vector Search)\n", + "// Add the Packages\n", + "#r \"nuget: dotenv.net, 3.1.2\"\n", + "#r \"nuget: Microsoft.SemanticKernel, 1.0.1\"\n", + "#r \"nuget: Microsoft.SemanticKernel.Connectors.OpenAI, 1.0.1\"\n", + "#r \"nuget: Azure.AI.OpenAI, 1.0.0-beta.12\"\n", + "#r \"nuget: Azure.Identity, 1.10.4\"\n", + "#r \"nuget: Azure.Search.Documents, 11.5.0-beta.5\"\n", + "#r \"nuget: Microsoft.Extensions.Logging, 7.0.0\"\n", + "#r \"nuget: Microsoft.Extensions.Logging.Console, 7.0.0\"\n", "\n", + "using System.IO;\n", + "using dotenv.net;\n", + "using Microsoft.SemanticKernel;\n", + "using Microsoft.SemanticKernel.Connectors.OpenAI;\n", "using Azure;\n", + "using Azure.AI.OpenAI;\n", + "using Azure.Identity;\n", "using Azure.Search.Documents;\n", "using Azure.Search.Documents.Models;\n", "using Azure.Search.Documents.Indexes;\n", - "using System.Text.Json.Serialization;\n", - "\n", - "var query = \"What are the best 80s movies I should look at?\";\n", - "\n", - "float[] queryEmbedding = azureOpenAIClient.GetEmbeddings(embedding_name, new EmbeddingsOptions(query)).Value.Data[0].Embedding.ToArray();\n", - "\n", - "// Note the Vectors addition in the SearchOptions\n", - "SearchOptions searchOptions = new SearchOptions\n", - "{\n", - " // Filter to only Content greater than or equal our preference\n", - " // Filter = SearchFilter.Create($\"Content ge {content}\"),\n", - " // OrderBy = { \"Content desc\" } // Sort by Content from high to low\n", - " // Size = 5, // Take only 5 results\n", - " // Select = { \"id\", \"content\", \"content_vector\" }, // Which fields to return\n", - " Vectors = { new() { Value = queryEmbedding, KNearestNeighborsCount = 5, Fields = { \"content_vector\" } } }, // Vector Search\n", - " Size = 5,\n", - " Select = { \"id\", \"content\" },\n", - "};\n", - "\n", - "// Note the search text and the vector search are both filled in.\n", - "SearchResults response = searchClient.Search(query, searchOptions);\n", - "Pageable> results = response.GetResults();\n", - "\n", - "// Print count of total results.\n", - "Console.WriteLine($\"Returned {results.Count()} results using hybrid search search.\");\n", - "Console.WriteLine(\"----------\");\n", + "using Azure.Search.Documents.Indexes.Models;\n", + "using Microsoft.Extensions.Configuration;\n", "\n", - "// Iterate over Results\n", - "// Index Fields - id, content, content_vector\n", - "foreach (SearchResult result in results)\n", - "{\n", - " Console.WriteLine($\"Id: {result.Document[\"id\"]}\");\n", - " Console.WriteLine($\"Score: {result.Score}\");\n", - " Console.WriteLine(\"----------\");\n", - "};" + "// Read values from .env file\n", + "var envVars = DotEnv.Fluent()\n", + " .WithoutExceptions()\n", + " .WithEnvFiles(\"../../../.env\")\n", + " .WithTrimValues()\n", + " .WithDefaultEncoding()\n", + " .WithOverwriteExistingVars()\n", + " .WithoutProbeForEnv()\n", + " .Read();\n", + "\n", + "var azure_openai_api_key = envVars[\"AZURE_OPENAI_API_KEY\"].Replace(\"\\\"\", \"\");\n", + "var azure_openai_endpoint = envVars[\"AZURE_OPENAI_ENDPOINT\"].Replace(\"\\\"\", \"\");\n", + "var azure_openai_completion_deployment_name = envVars[\"AZURE_OPENAI_COMPLETION_DEPLOYMENT_NAME\"].Replace(\"\\\"\", \"\");\n", + "var azure_openai_embedding_deployment_name = envVars[\"AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME\"].Replace(\"\\\"\", \"\");\n", + "var azure_ai_search_name = envVars[\"AZURE_AI_SEARCH_SERVICE_NAME\"].Replace(\"\\\"\", \"\");\n", + "var azure_ai_search_endpoint = envVars[\"AZURE_AI_SEARCH_ENDPOINT\"].Replace(\"\\\"\", \"\");\n", + "var azure_ai_search_api_key = envVars[\"AZURE_AI_SEARCH_API_KEY\"].Replace(\"\\\"\", \"\");\n", + "var azure_ai_search_index_name = envVars[\"AZURE_AI_SEARCH_INDEX_NAME\"].Replace(\"\\\"\", \"\");\n", + "\n", + "Console.WriteLine(\"This lab exercise will use the following values:\");\n", + "Console.WriteLine($\"Azure OpenAI Endpoint: {azure_openai_endpoint}\");\n", + "Console.WriteLine($\"Azure AI Search: {azure_ai_search_name}\");" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We'll setup Semantic Kernel with an Azure OpenAI service." ] }, { @@ -738,62 +848,16 @@ }, "outputs": [], "source": [ - "// Hybrid Search\n", - "// Let's try our more specific query again to see the difference in the score returned.\n", - "\n", - "using Azure;\n", - "using Azure.Search.Documents;\n", - "using Azure.Search.Documents.Models;\n", - "using Azure.Search.Documents.Indexes;\n", - "using System.Text.Json.Serialization;\n", - "\n", - "var query = \"Who are the actors in the movie Hidden Figures?\";\n", - "\n", - "float[] queryEmbedding = azureOpenAIClient.GetEmbeddings(embedding_name, new EmbeddingsOptions(query)).Value.Data[0].Embedding.ToArray();\n", - "\n", - "// Note the Vectors addition in the SearchOptions\n", - "SearchOptions searchOptions = new SearchOptions\n", - "{\n", - " // Filter to only Content greater than or equal our preference\n", - " // Filter = SearchFilter.Create($\"Content ge {content}\"),\n", - " // OrderBy = { \"Content desc\" } // Sort by Content from high to low\n", - " // Size = 5, // Take only 5 results\n", - " // Select = { \"id\", \"content\", \"content_vector\" }, // Which fields to return\n", - " Vectors = { new() { Value = queryEmbedding, KNearestNeighborsCount = 5, Fields = { \"content_vector\" } } }, // Vector Search\n", - " Size = 5,\n", - " Select = { \"id\", \"content\" },\n", - "};\n", - "\n", - "// Note the search text and the vector search are both filled in.\n", - "SearchResults response = searchClient.Search(query, searchOptions);\n", - "Pageable> results = response.GetResults();\n", - "\n", - "// Print count of total results.\n", - "Console.WriteLine($\"Returned {results.Count()} results using hybrid search search.\");\n", - "Console.WriteLine(\"----------\");\n", - "\n", - "// Iterate over Results\n", - "// Index Fields - id, content, content_vector\n", - "foreach (SearchResult result in results)\n", - "{\n", - " Console.WriteLine($\"Id: {result.Document[\"id\"]}\");\n", - " Console.WriteLine($\"Score: {result.Score}\");\n", - " Console.WriteLine(\"----------\");\n", - "};" + "var builder = Kernel.CreateBuilder();\n", + "builder.Services.AddAzureOpenAIChatCompletion(azure_openai_completion_deployment_name, azure_openai_endpoint, azure_openai_api_key);\n", + "var kernel = builder.Build();" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## Bringing it All Together with Retrieval Augmented Generation (RAG) + Semantic Kernel (SK)\n", - "\n", - "Now that we have our Vector Store setup and data loaded, we are now ready to implement the RAG pattern using AI Orchestration. At a high-level, the following steps are required:\n", - "1. Ask the question\n", - "2. Create Prompt Template with inputs\n", - "3. Get Embedding representation of inputted question\n", - "4. Use embedded version of the question to search Azure Cognitive Search (ie. The Vector Store)\n", - "5. Inject the results of the search into the Prompt Template & Execute the Prompt to get the completion" + "Now we need to define a prompt template. In this template, we'll give the model the `System` instructions telling it to answer the users question using only the data provided. We'll then inject the search results that we'll later get from Azure AI Search into the `searchResults` variable. The `User` instructions will be the question that the user asks." ] }, { @@ -812,159 +876,186 @@ }, "outputs": [], "source": [ - "// Implement RAG using Semantic Kernel (SK)\n", - "\n", - "// Add the Packages\n", - "#r \"nuget: DotNetEnv, 2.5.0\"\n", - "#r \"nuget: Microsoft.SemanticKernel, 0.24.230918.1-preview\"\n", - "#r \"nuget: Microsoft.SemanticKernel.Abstractions, 0.24.230918.1-preview\"\n", - "#r \"nuget: Azure.AI.OpenAI, 1.0.0-beta.7\"\n", - "#r \"nuget: Azure.Identity, 1.10.1\"\n", - "#r \"nuget: Azure.Search.Documents, 11.5.0-beta.4\"\n", - "#r \"nuget: Microsoft.Extensions.Logging, 7.0.0\"\n", - "#r \"nuget: Microsoft.Extensions.Logging.Console, 7.0.0\"\n", - "\n", - "using System.IO;\n", - "using Microsoft.SemanticKernel;\n", - "using Microsoft.SemanticKernel.SemanticFunctions;\n", - "using Microsoft.SemanticKernel.Orchestration;\n", - "using Microsoft.SemanticKernel.SkillDefinition;\n", - "using Azure;\n", - "using Azure.Search.Documents;\n", - "using Azure.Search.Documents.Models;\n", - "using Azure.Search.Documents.Indexes;\n", - "using Azure.AI.OpenAI;\n", - "using Azure.Identity;\n", - "\n", - "// Read values from .env file\n", - "DotNetEnv.Env.Load(\"../../../.env\");\n", - "\n", - "// Load values into variables\n", - "var openai_api_type = System.Environment.GetEnvironmentVariable(\"OPENAI_API_TYPE\");\n", - "var openai_api_key = System.Environment.GetEnvironmentVariable(\"OPENAI_API_KEY\");\n", - "var openai_api_base = System.Environment.GetEnvironmentVariable(\"OPENAI_API_BASE\");\n", - "var openai_api_version = System.Environment.GetEnvironmentVariable(\"OPENAI_API_VERSION\");\n", - "var deployment_name = System.Environment.GetEnvironmentVariable(\"AZURE_OPENAI_COMPLETION_DEPLOYMENT_NAME\");\n", - "var embedding_name = System.Environment.GetEnvironmentVariable(\"AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME\");\n", - "var acs_service_name = System.Environment.GetEnvironmentVariable(\"AZURE_COGNITIVE_SEARCH_SERVICE_NAME\");\n", - "var acs_endpoint_name = System.Environment.GetEnvironmentVariable(\"AZURE_COGNITIVE_SEARCH_ENDPOINT_NAME\");\n", - "var acs_index_name = System.Environment.GetEnvironmentVariable(\"AZURE_COGNITIVE_SEARCH_INDEX_NAME\");\n", - "var acs_api_key = System.Environment.GetEnvironmentVariable(\"AZURE_COGNITIVE_SEARCH_API_KEY\");\n", - "\n", - "Console.WriteLine(\"Environment Variables loaded.\");\n", - "\n", - "// Setup Semantic Kernel\n", - "IKernel kernel = Kernel.Builder\n", - " .WithAzureChatCompletionService(deployment_name, openai_api_base, openai_api_key)\n", - " .WithAzureTextEmbeddingGenerationService(embedding_name, openai_api_base, openai_api_key)\n", - " .Build();\n", - "\n", - "Console.WriteLine(\"SK Kernel with ChatCompletion and EmbeddingsGeneration services created.\");\n", - "\n", - "// Ask the question\n", - "var question = \"List the movies about ships on the water.\";\n", - "\n", - "// Create a prompt template with variables, note the double curly braces with dollar sign for the variables\n", - "// First let's create the prompt string.\n", - "var sk_prompt = @\"\n", - "Question: {{$original_question}}\n", - "\n", - "Do not use any other data.\n", - "Only use the movie data below when responding.\n", - "{{$search_results}}\n", - "\";\n", - "// Create the PromptTemplateConfig\n", - "PromptTemplateConfig promptConfig = new PromptTemplateConfig\n", - "{\n", - " Schema = 1,\n", - " Type = \"completion\",\n", - " Description = \"Gets the intent of the user.\",\n", - " Completion = \n", + "var askQuery = kernel.CreateFunctionFromPrompt(\n", + " new PromptTemplateConfig()\n", " {\n", - " Temperature = 0.1,\n", - " TopP = 0.5,\n", - " PresencePenalty = 0.0,\n", - " FrequencyPenalty = 0.0,\n", - " MaxTokens = 500\n", - " // StopSequences = null,\n", - " // ChatSystemPprompt = null;\n", - " },\n", - " Input = \n", - " {\n", - " Parameters = new List\n", - " {\n", - " new PromptTemplateConfig.InputParameter\n", - " {\n", - " Name=\"original_question\",\n", - " Description=\"The user's request.\",\n", - " DefaultValue=\"\"\n", - " },\n", - " new PromptTemplateConfig.InputParameter\n", - " {\n", - " Name=\"search_results\",\n", - " Description=\"Vector Search results from Azure Cognitive Search.\",\n", - " DefaultValue=\"\"\n", - " }\n", + " Name = \"askQuery\",\n", + " Description = \"Ask Azure OpenAI a question using results from Azure AI Search.\",\n", + " Template = @\"System: Answer the user's question using only the movie data below. Do not use any other data. Provide detailed information about the synopsis of the movie. {{$searchResults}}\n", + " User: {{$originalQuestion}}\n", + " Assistant: \",\n", + " TemplateFormat = \"semantic-kernel\",\n", + " InputVariables = [\n", + " new() { Name = \"originalQuestion\", Description = \"The query we sent to Azure AI Search.\", IsRequired = true },\n", + " new() { Name = \"searchResults\", Description = \"The results retrieved from Azure AI Search.\", IsRequired = true }\n", + " ],\n", + " ExecutionSettings = {\n", + " { \"default\", new OpenAIPromptExecutionSettings() {\n", + " MaxTokens = 1000,\n", + " Temperature = 0.1,\n", + " TopP = 0.5,\n", + " PresencePenalty = 0.0,\n", + " FrequencyPenalty = 0.0\n", + " } }\n", " }\n", " }\n", - "};\n", - "// Create the SemanticFunctionConfig object\n", - "PromptTemplate promptTemplate = new PromptTemplate(\n", - " sk_prompt,\n", - " promptConfig,\n", - " kernel\n", - ");\n", - "SemanticFunctionConfig functionConfig = new SemanticFunctionConfig(promptConfig, promptTemplate);\n", - "// Register the GetIntent function with the Kernel\n", - "ISKFunction getIntentFunction = kernel.RegisterSemanticFunction(\"CustomPlugin\", \"GetIntent\", functionConfig);\n", - "\n", - "Console.WriteLine(\"Semantic Function GetIntent with SK has been completed.\");\n", - "\n", - "// Get Embedding for the original question\n", - "OpenAIClient azureOpenAIClient = new OpenAIClient(new Uri(openai_api_base),new AzureKeyCredential(openai_api_key));\n", - "float[] questionEmbedding = azureOpenAIClient.GetEmbeddings(embedding_name, new EmbeddingsOptions(question)).Value.Data[0].Embedding.ToArray();\n", + ");" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Next we define our question and send the question to an embedding model to generate the vector representation of the question." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "dotnet_interactive": { + "language": "csharp" + }, + "polyglot_notebook": { + "kernelName": "csharp" + }, + "vscode": { + "languageId": "polyglot-notebook" + } + }, + "outputs": [], + "source": [ + "var query = \"What are the best movies about superheroes?\";\n", "\n", - "Console.WriteLine(\"Embedding of original question has been completed.\");\n", + "OpenAIClient azureOpenAIClient = new OpenAIClient(new Uri(azure_openai_endpoint),new AzureKeyCredential(azure_openai_api_key));\n", + "float[] queryEmbedding = azureOpenAIClient.GetEmbeddings(new EmbeddingsOptions(azure_openai_embedding_deployment_name, new List() { query })).Value.Data[0].Embedding.ToArray();" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let's prepare the `SearchOptions` to carry out a vector search." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "dotnet_interactive": { + "language": "csharp" + }, + "polyglot_notebook": { + "kernelName": "csharp" + }, + "vscode": { + "languageId": "polyglot-notebook" + } + }, + "outputs": [], + "source": [ + "string semanticSearchConfigName = \"movies-semantic-config\";\n", "\n", - "// Search Vector Store\n", "SearchOptions searchOptions = new SearchOptions\n", "{\n", - " // Filter to only Content greater than or equal our preference\n", - " // Filter = SearchFilter.Create($\"Content ge {content}\"),\n", - " // OrderBy = { \"Content desc\" } // Sort by Content from high to low\n", - " // Size = 5, // Take only 5 results\n", - " // Select = { \"id\", \"content\", \"content_vector\" }, // Which fields to return\n", - " Vectors = { new() { Value = questionEmbedding, KNearestNeighborsCount = 5, Fields = { \"content_vector\" } } }, // Vector Search\n", + " QueryType = SearchQueryType.Semantic,\n", + " SemanticConfigurationName = semanticSearchConfigName,\n", + " VectorQueries = { new RawVectorQuery() { Vector = queryEmbedding, KNearestNeighborsCount = 5, Fields = { \"vector\" } } },\n", " Size = 5,\n", - " Select = { \"id\", \"content\" },\n", - "};\n", + " Select = { \"title\", \"genre\" },\n", + "};" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now we can go ahead and perform the search. As noted previously, we'll provide both the original question text and the vector representation of the text in order to perform a hybrid search with semantic ranking." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "dotnet_interactive": { + "language": "csharp" + }, + "polyglot_notebook": { + "kernelName": "csharp" + }, + "vscode": { + "languageId": "polyglot-notebook" + } + }, + "outputs": [], + "source": [ + "AzureKeyCredential indexCredential = new AzureKeyCredential(azure_ai_search_api_key);\n", + "SearchIndexClient indexClient = new SearchIndexClient(new Uri(azure_ai_search_endpoint), indexCredential);\n", + "SearchClient searchClient = indexClient.GetSearchClient(azure_ai_search_index_name);\n", "\n", - "// Note the search text is null and the vector search is filled in.\n", - "AzureKeyCredential credential = new AzureKeyCredential(acs_api_key);\n", - "SearchClient searchClient = new SearchClient(new Uri(acs_endpoint_name), acs_index_name, credential);\n", - "SearchResults response = searchClient.Search(null, searchOptions);\n", - "Pageable> results = response.GetResults();\n", - "// Create string from the results\n", + "//Perform the search\n", + "SearchResults response = searchClient.Search(query, searchOptions);\n", + "Pageable> results = response.GetResults();" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The results of the search are now in the `results` variable. We'll iterate through them and turn them into a string that we can inject into the prompt template. When finished, we display these results so that you can see what is going to be injected into the template." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "dotnet_interactive": { + "language": "csharp" + }, + "polyglot_notebook": { + "kernelName": "csharp" + }, + "vscode": { + "languageId": "polyglot-notebook" + } + }, + "outputs": [], + "source": [ "StringBuilder stringBuilderResults = new StringBuilder();\n", "foreach (SearchResult result in results)\n", "{\n", - " stringBuilderResults.AppendLine($\"{result.Document[\"content\"]}\");\n", + " stringBuilderResults.AppendLine($\"{result.Document[\"title\"]}\");\n", "};\n", "\n", - "Console.WriteLine(\"Searching of Vector Store has been completed.\");\n", - "\n", - "// Build the Prompt and Execute against the Azure OpenAI to get the completion\n", - "// Initialize the prompt variables\n", - "ContextVariables variables = new ContextVariables\n", - "{\n", - " [\"original_question\"] = question,\n", - " [\"search_results\"] = stringBuilderResults.ToString()\n", - "};\n", - "// Use SK Chaining to Invoke Semantic Function\n", - "string completion = (await kernel.RunAsync(variables, getIntentFunction)).Result;\n", - "Console.WriteLine(completion);\n", + "Console.WriteLine(stringBuilderResults.ToString());" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Finally, we can inject the search results into the prompt template and execute the prompt to get the completion." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "dotnet_interactive": { + "language": "csharp" + }, + "polyglot_notebook": { + "kernelName": "csharp" + }, + "vscode": { + "languageId": "polyglot-notebook" + } + }, + "outputs": [], + "source": [ + "var completion = await kernel.InvokeAsync(askQuery, new () { { \"originalQuestion\", query }, { \"searchResults\", stringBuilderResults.ToString() }});\n", "\n", - "Console.WriteLine(\"Implementation of RAG using SK, C# and Azure Cognitive Search has been completed.\");" + "Console.WriteLine(completion);" ] }, { @@ -993,7 +1084,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.11.5" + "version": "3.11.6" }, "orig_nbformat": 4 }, diff --git a/labs/03-orchestration/04-ACS/acs-sk-python.ipynb b/labs/03-orchestration/04-ACS/acs-sk-python.ipynb deleted file mode 100644 index 3128d39..0000000 --- a/labs/03-orchestration/04-ACS/acs-sk-python.ipynb +++ /dev/null @@ -1,788 +0,0 @@ -{ - "cells": [ - { - "attachments": {}, - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# 04 - AI Orchestration with Azure Cognitive Search\n", - "\n", - "In this lab, we will do a deeper dive around the Azure Cognitive Search vector store and different ways to interact with it." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Create Azure Cognitive Search Vector Store in Azure\n", - "\n", - "First, we need to create an Azure Cognitive Search service in Azure, which will act as a vector store. We'll use the Azure CLI to do this.\n", - "\n", - "**NOTE:** Update **``** to make the name unique." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "RESOURCE_GROUP=\"azure-cognitive-search-rg\"\n", - "LOCATION=\"westeurope\"\n", - "NAME=\"acs-vectorstore-\"\n", - "!az group create --name $RESOURCE_GROUP --location $LOCATION\n", - "!az search service create -g $RESOURCE_GROUP -n $NAME -l $LOCATION --sku Basic --partition-count 1 --replica-count 1" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Next, we need to find and update the following values in the `.env` file with the Azure Cognitive Search **endpoint**, **admin key**, and **index name** values. Use the Azure Portal or CLI.\n", - "\n", - "```\n", - "AZURE_COGNITIVE_SEARCH_SERVICE_NAME = \"\"\n", - "AZURE_COGNITIVE_SEARCH_ENDPOINT_NAME = \"\"\n", - "AZURE_COGNITIVE_SEARCH_API_KEY = \"\"\n", - "```" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Setup Azure OpenAI\n", - "\n", - "We'll start as usual by defining our Azure OpenAI service API key and endpoint details, specifying the model deployment we want to use and then we'll initiate a connection to the Azure OpenAI service.\n", - "\n", - "**NOTE**: As with previous labs, we'll use the values from the `.env` file in the root of this repository." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import os\n", - "from dotenv import load_dotenv\n", - "\n", - "# Load environment variables\n", - "if load_dotenv():\n", - " print(\"Found OpenAPI Base Endpoint: \" + os.getenv(\"OPENAI_API_BASE\"))\n", - "else: \n", - " print(\"No file .env found\")\n", - "\n", - "openai_api_type = os.getenv(\"OPENAI_API_TYPE\")\n", - "openai_api_key = os.getenv(\"OPENAI_API_KEY\")\n", - "openai_api_base = os.getenv(\"OPENAI_API_BASE\")\n", - "openai_api_version = os.getenv(\"OPENAI_API_VERSION\")\n", - "deployment_name = os.getenv(\"AZURE_OPENAI_COMPLETION_DEPLOYMENT_NAME\")\n", - "embedding_name = os.getenv(\"AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME\")\n", - "acs_service_name = os.getenv(\"AZURE_COGNITIVE_SEARCH_SERVICE_NAME\")\n", - "acs_endpoint_name = os.getenv(\"AZURE_COGNITIVE_SEARCH_ENDPOINT_NAME\")\n", - "acs_index_name = os.getenv(\"AZURE_COGNITIVE_SEARCH_INDEX_NAME\")\n", - "acs_api_key = os.getenv(\"AZURE_COGNITIVE_SEARCH_API_KEY\")" - ] - }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": {}, - "source": [ - "First, we will load the data from the movies.csv file using the Langchain CSV document loader." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from langchain.document_loaders.csv_loader import CSVLoader\n", - "\n", - "# Movie Fields in CSV\n", - "# id,original_language,original_title,popularity,release_date,vote_average,vote_count,genre,overview,revenue,runtime,tagline\n", - "loader = CSVLoader(file_path='./movies.csv', source_column='original_title', encoding='utf-8', csv_args={'delimiter':',', 'fieldnames': ['id', 'original_language', 'original_title', 'popularity', 'release_date', 'vote_average', 'vote_count', 'genre', 'overview', 'revenue', 'runtime', 'tagline']})\n", - "data = loader.load()\n", - "data = data[1:51] # reduce dataset if you want\n", - "print('Loaded %s movies' % len(data))" - ] - }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Next, we will create an Azure OpenAI embedding and completion deployments in order to create the vector representation of the movies so we can start asking our questions." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from langchain.embeddings.openai import OpenAIEmbeddings\n", - "from langchain.chat_models import AzureChatOpenAI\n", - "\n", - "# Create an Embeddings Instance of Azure OpenAI\n", - "embeddings = OpenAIEmbeddings(\n", - " model=\"text-embedding-ada-002\",\n", - " deployment=embedding_name,\n", - " openai_api_type = openai_api_type,\n", - " openai_api_version = openai_api_version,\n", - " openai_api_base = openai_api_base,\n", - " openai_api_key = openai_api_key,\n", - " embedding_ctx_length=8191,\n", - " chunk_size=1000,\n", - " max_retries=6\n", - ")\n", - "\n", - "# Create a Completion Instance of Azure OpenAI\n", - "llm = AzureChatOpenAI(\n", - " model=\"gpt-3.5-turbo\",\n", - " deployment_name = deployment_name,\n", - " openai_api_type = openai_api_type,\n", - " openai_api_version = openai_api_version,\n", - " openai_api_base = openai_api_base,\n", - " openai_api_key = openai_api_key,\n", - " temperature=0.7,\n", - " max_retries=6,\n", - " max_tokens=4000\n", - ")\n", - "\n", - "print('Completed creation of embedding and completion instances.')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Load Movies into Azure Cognitive Search\n", - "\n", - "Next, we'll create the Azure Cognitive Search index, embed the loaded movies from the CSV file, and upload the data into the newly created index. Depending on the number of movies loaded and rate limiting, this might take a while to do the embeddings so be patient." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from azure.core.credentials import AzureKeyCredential\n", - "from azure.search.documents.indexes import SearchIndexClient\n", - "from azure.search.documents.indexes.models import (\n", - " SearchIndex,\n", - " SearchField,\n", - " SearchFieldDataType,\n", - " SimpleField,\n", - " SearchableField,\n", - " SearchIndex,\n", - " SemanticConfiguration,\n", - " PrioritizedFields,\n", - " SemanticField,\n", - " SearchField,\n", - " SemanticSettings,\n", - " VectorSearch,\n", - " HnswVectorSearchAlgorithmConfiguration,\n", - ")\n", - "\n", - "# Let's Create the Azure Cognitive Search Index\n", - "index_client = SearchIndexClient(\n", - " acs_endpoint_name,\n", - " AzureKeyCredential(acs_api_key)\n", - ")\n", - "# Movie Fields in CSV\n", - "# id,original_language,original_title,popularity,release_date,vote_average,vote_count,genre,overview,revenue,runtime,tagline\n", - "fields = [\n", - " SimpleField(name=\"id\", type=SearchFieldDataType.String, key=True, sortable=True, filterable=True, facetable=True),\n", - " SearchableField(name=\"content\", type=SearchFieldDataType.String),\n", - " SearchField(name=\"content_vector\", type=SearchFieldDataType.Collection(SearchFieldDataType.Single), searchable=True, vector_search_dimensions=1536, vector_search_configuration=\"my-vector-config\"),\n", - "]\n", - "\n", - "# Configure Vector Search Configuration\n", - "vector_search = VectorSearch(\n", - " algorithm_configurations=[\n", - " HnswVectorSearchAlgorithmConfiguration(\n", - " name=\"my-vector-config\",\n", - " kind=\"hnsw\",\n", - " parameters={\n", - " \"m\": 4,\n", - " \"efConstruction\": 400,\n", - " \"efSearch\": 500,\n", - " \"metric\": \"cosine\"\n", - " }\n", - " )\n", - " ]\n", - ")\n", - "\n", - "# Configure Semantic Configuration\n", - "semantic_config = SemanticConfiguration(\n", - " name=\"my-semantic-config\",\n", - " prioritized_fields=PrioritizedFields(\n", - " title_field=SemanticField(field_name=\"content\"),\n", - " prioritized_keywords_fields=[SemanticField(field_name=\"content\")],\n", - " prioritized_content_fields=[SemanticField(field_name=\"content\")]\n", - " )\n", - ")\n", - "\n", - "# Create the semantic settings with the configuration\n", - "semantic_settings = SemanticSettings(configurations=[semantic_config])\n", - "\n", - "# Create the search index with the desired vector search and semantic configurations\n", - "index = SearchIndex(\n", - " name=acs_index_name,\n", - " fields=fields,\n", - " vector_search=vector_search,\n", - " semantic_settings=semantic_settings\n", - ")\n", - "result = index_client.create_or_update_index(index)\n", - "print(f'The {result.name} index was created.')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Next we will create the document structure needed to upload the data into the Azure Cognitive Search index." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Now that the index is created, let's load the documents into it.\n", - "\n", - "import uuid\n", - "\n", - "# Let's take a quick look at the data structure of the CSVLoader\n", - "print(data[0])\n", - "print(\"----------\")\n", - "\n", - "# Generate Document Embeddings for page_content field in the movies CSVLoader dataset using Azure OpenAI\n", - "items = []\n", - "for movie in data:\n", - " content = movie.page_content\n", - " items.append(dict([(\"id\", str(uuid.uuid4())), (\"content\", content), (\"content_vector\", embeddings.embed_query(content))]))\n", - "\n", - "# Print out a sample item to validate the updated data structure.\n", - "# It should have the id, content, and content_vector values.\n", - "print(items[0])\n", - "print(f\"Movie Count: {len(items)}\")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Next we will upload the movie documents in the newly created structure to the Azure Cognitive Search index." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Upload movies to Azure Cognitive Search index.\n", - "from azure.search.documents.models import Vector\n", - "from azure.search.documents import SearchClient\n", - "\n", - "# Insert Text and Embeddings into the Azure Cognitive Search index created.\n", - "search_client = SearchClient(\n", - " acs_endpoint_name,\n", - " acs_index_name,\n", - " AzureKeyCredential(acs_api_key)\n", - ")\n", - "result = search_client.upload_documents(items)\n", - "print(\"Successfully added documents to Azure Cognitive Search index.\")\n", - "print(f\"Uploaded {len(data)} documents\")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Vector Store Searching using Azure Cognitive Search\n", - "\n", - "Now that we have the movies loaded into Azure Cognitive Search, let's do some different types of searches using the Azure Cognitive Search SDK." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# First, let's do a plain vanilla text search, no vectors or embeddings.\n", - "query = \"What are the best 80s movies I should look at?\"\n", - "\n", - "search_client = SearchClient(\n", - " acs_endpoint_name,\n", - " acs_index_name,\n", - " AzureKeyCredential(acs_api_key)\n", - ")\n", - "\n", - "# Execute the search\n", - "results = list(search_client.search(\n", - " search_text=query,\n", - " include_total_count=True,\n", - " top=5\n", - "))\n", - "\n", - "# Print count of total results.\n", - "print(f\"Returned {len(results)} results using only text-based search.\")\n", - "print(\"----------\")\n", - "# Iterate over Results\n", - "# Index Fields - id, content, content_vector\n", - "for result in results:\n", - " print(\"Movie: {}\".format(result[\"content\"]))\n", - " print(\"----------\")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Now let's do a vector search that uses the embeddings we created and inserted into content_vector field in the index.\n", - "query = \"What are the best 80s movies I should look at?\"\n", - "\n", - "search_client = SearchClient(\n", - " acs_endpoint_name,\n", - " acs_index_name,\n", - " AzureKeyCredential(acs_api_key)\n", - ")\n", - "\n", - "# You can see here that we are getting the embedding representation of the query.\n", - "vector = Vector(\n", - " value=embeddings.embed_query(query),\n", - " k=5,\n", - " fields=\"content_vector\"\n", - ")\n", - "\n", - "# Execute the search\n", - "results = list(search_client.search(\n", - " search_text=\"\",\n", - " include_total_count=True,\n", - " vectors=[vector],\n", - " select=[\"id\", \"content\"],\n", - "))\n", - "\n", - "# Print count of total results.\n", - "print(f\"Returned {len(results)} results using only vector-based search.\")\n", - "print(\"----------\")\n", - "# Iterate over results and print out the content.\n", - "for result in results:\n", - " print(result[\"content\"])\n", - " print(\"----------\")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Did that return what you expected? Probably not, let's dig deeper to see why.\n", - "\n", - "Let's do the same search again, but this time let's return the **Search Score** so we can see the value returned by the cosine similarity vector store calculation." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Try again, but this time let's add the relevance score to maybe see why\n", - "query = \"What are the best 80s movies I should look at?\"\n", - "\n", - "search_client = SearchClient(\n", - " acs_endpoint_name,\n", - " acs_index_name,\n", - " AzureKeyCredential(acs_api_key)\n", - ")\n", - "\n", - "# You can see here that we are getting the embedding representation of the query.\n", - "vector = Vector(\n", - " value=embeddings.embed_query(query),\n", - " k=5,\n", - " fields=\"content_vector\"\n", - ")\n", - "\n", - "# Execute the search\n", - "results = list(search_client.search(\n", - " search_text=\"\",\n", - " include_total_count=True,\n", - " vectors=[vector],\n", - " select=[\"id\", \"content\"],\n", - "))\n", - "\n", - "# Print count of total results.\n", - "print(f\"Returned {len(results)} results using vector search.\")\n", - "print(\"----------\")\n", - "# Iterate over results and print out the id and search score.\n", - "for result in results: \n", - " print(f\"Id: {result['id']}\")\n", - " print(f\"Score: {result['@search.score']}\")\n", - " print(\"----------\")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "If you look at the Search Score you will see the relevant ranking of the closest vector match to the query inputted. The lower the score the farther apart the two vectors are. Let's change the search term and see if we can get a higher Search Score which means a higher match and closer vector proximity." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Try again, but this time let's add the relevance score to maybe see why\n", - "query = \"Who are the actors in the movie Hidden Figures?\"\n", - "\n", - "search_client = SearchClient(\n", - " acs_endpoint_name,\n", - " acs_index_name,\n", - " AzureKeyCredential(acs_api_key)\n", - ")\n", - "\n", - "# You can see here that we are getting the embedding representation of the query.\n", - "vector = Vector(\n", - " value=embeddings.embed_query(query),\n", - " k=5,\n", - " fields=\"content_vector\"\n", - ")\n", - "\n", - "# Execute the search\n", - "results = list(search_client.search(\n", - " search_text=\"\",\n", - " include_total_count=True,\n", - " vectors=[vector],\n", - " select=[\"id\", \"content\"],\n", - "))\n", - "\n", - "# Print count of total results.\n", - "print(f\"Returned {len(results)} results using vector search.\")\n", - "print(\"----------\")\n", - "# Iterate over results and print out the id and search score.\n", - "for result in results: \n", - " print(f\"Id: {result['id']}\")\n", - " print(f\"Score: {result['@search.score']}\")\n", - " print(\"----------\")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "**NOTE:** As you have seen from the results, different inputs can return different results, it all depends on what data is in the Vector Store. The higher the score the higher the likelihood of a match." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Hybrid Searching using Azure Cognitive Search\n", - "\n", - "What is Hybrid Search? The search is implemented at the field level, which means you can build queries that include vector fields and searchable text fields. The queries execute in parallel and the results are merged into a single response. Optionally, add semantic search, currently in preview, for even more accuracy with L2 reranking using the same language models that power Bing.\n", - "\n", - "**NOTE:** Hybrid Search is a key value proposition of Azure Cognitive Search in comparison to vector only data stores. Click [Hybrid Search](https://learn.microsoft.com/en-us/azure/search/hybrid-search-overview) for more details." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Hybrid Search\n", - "# Let's try our original query again using Hybrid Search (ie. Combination of Text & Vector Search)\n", - "query = \"What are the best 80s movies I should look at?\"\n", - "\n", - "search_client = SearchClient(\n", - " acs_endpoint_name,\n", - " acs_index_name,\n", - " AzureKeyCredential(acs_api_key)\n", - ")\n", - "\n", - "# You can see here that we are getting the embedding representation of the query.\n", - "vector = Vector(\n", - " value=embeddings.embed_query(query),\n", - " k=5,\n", - " fields=\"content_vector\"\n", - ")\n", - "\n", - "# Notice we also fill in the search_text parameter with the query.\n", - "results = list(search_client.search(\n", - " search_text=query,\n", - " include_total_count=True,\n", - " top=10,\n", - " vectors=[vector],\n", - " select=[\"id\", \"content\"],\n", - "))\n", - "\n", - "# Print count of total results.\n", - "print(f\"Returned {len(results)} results using vector search.\")\n", - "print(\"----------\")\n", - "# Iterate over results and print out the id and search score.\n", - "for result in results: \n", - " print(f\"Id: {result['id']}\")\n", - " print(f\"Hybrid Search Score: {result['@search.score']}\")\n", - " print(\"----------\")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Hybrid Search\n", - "# Let's try our more specific query again to see the difference in the score returned.\n", - "query = \"Who are the actors in the movie Hidden Figures?\"\n", - "\n", - "search_client = SearchClient(\n", - " acs_endpoint_name,\n", - " acs_index_name,\n", - " AzureKeyCredential(acs_api_key)\n", - ")\n", - "\n", - "# You can see here that we are getting the embedding representation of the query.\n", - "vector = Vector(\n", - " value=embeddings.embed_query(query),\n", - " k=5,\n", - " fields=\"content_vector\"\n", - ")\n", - "\n", - "# -----\n", - "# Notice we also fill in the search_text parameter with the query along with the vector.\n", - "# -----\n", - "results = list(search_client.search(\n", - " search_text=query,\n", - " include_total_count=True,\n", - " top=10,\n", - " vectors=[vector],\n", - " select=[\"id\", \"content\"],\n", - "))\n", - "\n", - "# Print count of total results.\n", - "print(f\"Returned {len(results)} results using hybrid search.\")\n", - "print(\"----------\")\n", - "# Iterate over results and print out the id and search score.\n", - "for result in results: \n", - " print(f\"Id: {result['id']}\")\n", - " print(f\"Hybrid Search Score: {result['@search.score']}\")\n", - " print(\"----------\")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Bringing it All Together with Retrieval Augmented Generation (RAG) + Semantic Kernel (SK)\n", - "\n", - "Now that we have our Vector Store setup and data loaded, we are now ready to implement the RAG pattern using AI Orchestration. At a high-level, the following steps are required:\n", - "1. Ask the question\n", - "2. Create Prompt Template with inputs\n", - "3. Get Embedding representation of inputted question\n", - "4. Use embedded version of the question to search Azure Cognitive Search (ie. The Vector Store)\n", - "5. Inject the results of the search into the Prompt Template & Execute the Prompt to get the completion" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Implement RAG using Semantic Kernel (SK)\n", - "import asyncio\n", - "import semantic_kernel as sk\n", - "from semantic_kernel.connectors.ai.open_ai import (\n", - " AzureChatCompletion,\n", - " AzureTextEmbedding,\n", - ")\n", - "from semantic_kernel.connectors.memory.azure_cognitive_search import (\n", - " AzureCognitiveSearchMemoryStore,\n", - ")\n", - "from semantic_kernel.semantic_functions.prompt_template import PromptTemplate\n", - "from semantic_kernel.semantic_functions.prompt_template_config import PromptTemplateConfig\n", - "from semantic_kernel.semantic_functions.semantic_function_config import SemanticFunctionConfig\n", - "from azure.search.documents.models import Vector\n", - "from azure.search.documents import SearchClient\n", - "from azure.search.documents.indexes import SearchIndexClient\n", - "from azure.search.documents.indexes.models import (\n", - " SearchIndex,\n", - " SearchField,\n", - " SearchFieldDataType,\n", - " SimpleField,\n", - " SearchableField,\n", - " SearchIndex,\n", - " SemanticConfiguration,\n", - " PrioritizedFields,\n", - " SemanticField,\n", - " SearchField,\n", - " SemanticSettings,\n", - " VectorSearch,\n", - " HnswVectorSearchAlgorithmConfiguration,\n", - ")\n", - "from azure.core.credentials import AzureKeyCredential\n", - "\n", - "# Setup Semantic Kernel\n", - "my_logger = sk.NullLogger()\n", - "kernel = sk.Kernel(\n", - " log=my_logger\n", - ")\n", - "\n", - "# Add Chat Service to SK\n", - "kernel.add_chat_service(\n", - " \"gpt35turbo\",\n", - " AzureChatCompletion(\n", - " deployment_name=deployment_name,\n", - " endpoint=openai_api_base,\n", - " api_key=openai_api_key,\n", - " logger=my_logger\n", - " )\n", - ")\n", - "# Add Text Embeddings Service to SK\n", - "embeddingsService = AzureTextEmbedding(\n", - " deployment_name=embedding_name,\n", - " endpoint=openai_api_base,\n", - " api_key=openai_api_key,\n", - " logger=my_logger\n", - ")\n", - "kernel.add_text_embedding_generation_service(\n", - " \"ada002\",\n", - " embeddingsService\n", - ")\n", - "\n", - "# Ask the question\n", - "question = \"List the movies about ships on the water.\"\n", - "\n", - "# Create a prompt template with variables, note the double curly braces with dollar sign for the variables\n", - "sk_prompt = \"\"\"\n", - "Question: {{$original_question}}\n", - "\n", - "Do not use any other data.\n", - "Only use the movie data below when responding.\n", - "{{$search_results}}\n", - "\"\"\"\n", - "# Create the PromptTemplateConfig\n", - "prompt_config = PromptTemplateConfig(\n", - " description=\"Gets the intent of the user.\",\n", - " type=\"completion\",\n", - " completion=PromptTemplateConfig.CompletionConfig(\n", - " temperature=0.1,\n", - " top_p=0.5,\n", - " presence_penalty=0.0,\n", - " frequency_penalty=0.0,\n", - " max_tokens=500,\n", - " # number_of_responses=1,\n", - " # stop_sequences=None,\n", - " # token_selection_biases=None,\n", - " # chat_system_prompt=None\n", - " ),\n", - " input=PromptTemplateConfig.InputConfig(\n", - " parameters=[\n", - " PromptTemplateConfig.InputParameter(\n", - " name=\"original_question\",\n", - " description=\"The user's request.\",\n", - " default_value=\"\"\n", - " ),\n", - " PromptTemplateConfig.InputParameter(\n", - " name=\"search_results\",\n", - " description=\"Vector Search results from Azure Cognitive Search.\",\n", - " default_value=\"\"\n", - " )\n", - " ]\n", - " ),\n", - ")\n", - "# Create the SemanticFunctionConfig object\n", - "prompt_template = PromptTemplate(\n", - " template=sk_prompt,\n", - " template_engine=kernel.prompt_template_engine,\n", - " prompt_config=prompt_config,\n", - ")\n", - "function_config = SemanticFunctionConfig(prompt_config, prompt_template)\n", - "# Register the Semantic Function with SK\n", - "get_intent = kernel.register_semantic_function(\n", - " skill_name=\"CustomPlugin\",\n", - " function_name=\"GetIntent\",\n", - " function_config=function_config,\n", - ")\n", - "\n", - "# Get Embedding for the original question\n", - "question_embedded = await embeddingsService.generate_embeddings_async(question)\n", - "\n", - "# Search Vector Store\n", - "search_client = SearchClient(\n", - " acs_endpoint_name,\n", - " acs_index_name,\n", - " AzureKeyCredential(acs_api_key)\n", - ")\n", - "vector = Vector(\n", - " value=question_embedded[0],\n", - " k=5,\n", - " fields=\"content_vector\"\n", - ")\n", - "results = list(search_client.search(\n", - " search_text=\"\",\n", - " include_total_count=True,\n", - " vectors=[vector],\n", - " select=[\"content\"],\n", - "))\n", - "\n", - "# Build the Prompt and Execute against the Azure OpenAI to get the completion\n", - "# Initialize the prompt variables\n", - "variables = sk.ContextVariables()\n", - "variables[\"original_question\"] = question\n", - "variables[\"search_results\"] = str(results)\n", - "# Use SK Chaining to Invoke Semantic Function\n", - "completion = await kernel.run_async(\n", - " get_intent,\n", - " input_vars=variables\n", - ")\n", - "print(completion)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Next Section\n", - "\n", - "📣 [Deploy AI](../../04-deploy-ai/README.md)" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.11.5" - }, - "orig_nbformat": 4 - }, - "nbformat": 4, - "nbformat_minor": 2 -} diff --git a/labs/03-orchestration/README.md b/labs/03-orchestration/README.md index b6505f6..ac53a9a 100644 --- a/labs/03-orchestration/README.md +++ b/labs/03-orchestration/README.md @@ -24,8 +24,8 @@ In this lab, we'll walk through using an open source vector store called Qdrant. ## 04-ACS -[Azure Cognitive Search + Semantic Kernel C#](04-ACS/acs-sk-csharp.ipynb) -[Azure Cognitive Search + Semantic Kernel Python](04-ACS/acs-sk-python.ipynb) -[Azure Cognitive Search + Langchain Python](04-ACS/acs-lc-python.ipynb) +[Azure AI Search + Semantic Kernel C#](04-ACS/acs-sk-csharp.ipynb) +[Azure AI Search + Semantic Kernel Python](04-ACS/acs-sk-python.ipynb) +[Azure AI Search + Langchain Python](04-ACS/acs-lc-python.ipynb) -In this lab, we'll walk through using one of Azure's vector stores, **Azure Cognitive Search**. +In this lab, we'll walk through using one of Azure's vector stores, **Azure AI Search**. diff --git a/labs/99-removed/acs-sk-python.ipynb b/labs/99-removed/acs-sk-python.ipynb new file mode 100644 index 0000000..42bac51 --- /dev/null +++ b/labs/99-removed/acs-sk-python.ipynb @@ -0,0 +1,736 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**Temporarily Removed**\n", + "\n", + "Original location:\n", + "intro-to-intelligent-apps/labs/03-orchestration/04-ACS\n", + "\n", + "The Semantic Kernel python libraries are undergoing significant change to align them with the functionality of the .NET libraries.\n", + "\n", + "As at 22nd January 2024, the azure-search-documents package latest version is 11.4.0. The code in the first half of this lab - i.e. the Azure AI Search parts - have been updated to work with this version of the package and are fully functional.\n", + "\n", + "However, Semantic Kernel currently has a dependency on azure-search-documents 11.4.0b9. If this version is not used, the second part of this lab (Bringing it all together) will not work. However, if you use 11.4.0b9, the first part of the lab will not work.\n", + "\n", + "Also, since this lab was first created, the semantic-kernel package has been updated so the code in the second part of this lab will not work with the latest version of the package anyway.\n", + "\n", + "This lab will be reintroduced once the Semantic Kernel python libraries appear to have stabilised." + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# 04 - AI Orchestration with Azure AI Search\n", + "**(Semantic Kernel / Python version)**\n", + "\n", + "In this lab, we will do a deeper dive into using Azure AI Search as a vector store, the different search methods it supports and how you can use it as part of the Retrieval Augmented Generation (RAG) pattern for working with large language models." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Create an Azure AI Search Vector Store in Azure\n", + "\n", + "First, we will create an Azure AI Search service in Azure. The following are command line instructions and require the Azure CLI to be installed.\n", + "\n", + "**NOTE:** Before running the commands, replace the **``** with your own initials or some random characters, as we need to provide a unique name for the Azure AI Search service." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "RESOURCE_GROUP=\"azure-ai-search-rg\"\n", + "LOCATION=\"westeurope\"\n", + "NAME=\"ai-vectorstore-\"\n", + "!az group create --name $RESOURCE_GROUP --location $LOCATION\n", + "!az search service create -g $RESOURCE_GROUP -n $NAME -l $LOCATION --sku Basic --partition-count 1 --replica-count 1" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Next, we need to find and update the following values in the `.env` file with the Azure AI Search **name**, **endpoint** and **admin key** values, which you can get from the Azure portal. You also need to provide an **index name** value. The index will be created during this lab, so you can use any name you like.\n", + "\n", + "```\n", + "AZURE_AI_SEARCH_SERVICE_NAME = \"\"\n", + "AZURE_AI_SEARCH_ENDPOINT = \"\"\n", + "AZURE_AI_SEARCH_API_KEY = \"\"\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Load environment variable values\n", + "As with previous labs, we'll use the values from the `.env` file in the root of this repository." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "from dotenv import load_dotenv\n", + "\n", + "# Load environment variables\n", + "if load_dotenv():\n", + " print(\"This lab exercise will use the following values:\")\n", + " print(\"Azure OpenAI Endpoint: \" + os.getenv(\"AZURE_OPENAI_ENDPOINT\"))\n", + " print(\"Azure AI Search: \" + os.getenv(\"AZURE_AI_SEARCH_SERVICE_NAME\"))\n", + "else: \n", + " print(\"No file .env found\")\n", + "\n", + "azure_openai_api_key = os.getenv(\"AZURE_OPENAI_API_KEY\")\n", + "azure_openai_endpoint = os.getenv(\"AZURE_OPENAI_ENDPOINT\")\n", + "openai_api_version = os.getenv(\"OPENAI_API_VERSION\")\n", + "azure_openai_completion_deployment_name = os.getenv(\"AZURE_OPENAI_COMPLETION_DEPLOYMENT_NAME\")\n", + "azure_openai_embedding_deployment_name = os.getenv(\"AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME\")\n", + "azure_ai_search_name = os.getenv(\"AZURE_AI_SEARCH_SERVICE_NAME\")\n", + "azure_ai_search_endpoint = os.getenv(\"AZURE_AI_SEARCH_ENDPOINT\")\n", + "azure_ai_search_index_name = os.getenv(\"AZURE_AI_SEARCH_INDEX_NAME\")\n", + "azure_ai_search_api_key = os.getenv(\"AZURE_AI_SEARCH_API_KEY\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "First, we will load the data from the movies.csv file and then extract a subset to load into the Azure AI Search index. We do this to help avoid the Azure OpenAI embedding limits and long loading times when inserting data into the index. We use a Langchain document loader to do this." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from langchain.document_loaders.csv_loader import CSVLoader\n", + "\n", + "loader = CSVLoader(file_path='./movies.csv', source_column='original_title', encoding='utf-8', csv_args={'delimiter':',', 'fieldnames': ['id', 'original_language', 'original_title', 'popularity', 'release_date', 'vote_average', 'vote_count', 'genre', 'overview', 'revenue', 'runtime', 'tagline']})\n", + "data = loader.load()\n", + "\n", + "# Rather than load all 500 movies into Azure AI search, we will use a\n", + "# smaller subset of movie data to make things quicker. The more movies you load,\n", + "# the more time it will take for embeddings to be generated.\n", + "\n", + "data = data[1:51]\n", + "print('Loaded %s movies.' % len(data))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "During this lab, we will need to work with embeddings. We use embeddings to create a vector representation of a piece of text. We will need to create embeddings for the documents we want to store in our Azure AI Search index and also for the queries we want to use to search the index. We will create an Azure OpenAI client to do this." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from langchain.embeddings import AzureOpenAIEmbeddings\n", + "\n", + "azure_openai_embeddings = AzureOpenAIEmbeddings(\n", + " azure_deployment = os.getenv(\"AZURE_OPENAI_EMBEDDINGS_DEPLOYMENT_NAME\")\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Create an Azure AI Search index and load movie data\n", + "\n", + "Next, we'll step through the process of configuring an Azure AI Search index to store our movie data and then loading the data into the index. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from azure.core.credentials import AzureKeyCredential\n", + "from azure.search.documents import SearchClient\n", + "from azure.search.documents.indexes import SearchIndexClient\n", + "from azure.search.documents.indexes.models import (\n", + " VectorSearch,\n", + " VectorSearchProfile,\n", + " HnswAlgorithmConfiguration,\n", + " SemanticPrioritizedFields,\n", + " SemanticSearch,\n", + " SemanticField,\n", + " SemanticConfiguration,\n", + " SimpleField,\n", + " SearchableField,\n", + " SearchField,\n", + " SearchFieldDataType,\n", + " SearchIndex\n", + ")\n", + "from azure.search.documents.models import (\n", + " VectorizedQuery\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "When configuring an Azure AI Search index, we need to specify the fields we want to store in the index and the data types for each field. These match the fields in the movie data, containing values such as the movie title, genre, year of release and so on.\n", + "\n", + "To use Azure AI Search as a vector store, we will also need to define a field to hold the vector representaion of the movie data. We indicate to Azure AI Search that this field will contain vector data by providing details of the vector dimensions and a profile. We'll also define the vector search configuration and profile with default values.\n", + "\n", + "**NOTE:** It is possible just to use Azure AI Search as a vector store only, in which case we probably wouldn't need to define all of the index fields below. However, in this lab, we're also going to demonstrate Hybrid Search, a feature which makes use of both traditional keyword based search in combination with vector search." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "fields = [\n", + " SimpleField(name=\"id\", type=SearchFieldDataType.String, key=True, sortable=True, filterable=True, facetable=True),\n", + " SearchableField(name=\"title\", type=SearchFieldDataType.String),\n", + " SearchableField(name=\"overview\", type=SearchFieldDataType.String),\n", + " SearchableField(name=\"genre\", type=SearchFieldDataType.String),\n", + " SearchableField(name=\"tagline\", type=SearchFieldDataType.String),\n", + " SearchableField(name=\"release_date\", type=SearchFieldDataType.DateTimeOffset, sortable=True),\n", + " SearchableField(name=\"popularity\", type=SearchFieldDataType.Double, sortable=True),\n", + " SearchableField(name=\"vote_average\", type=SearchFieldDataType.Double, sortable=True),\n", + " SearchableField(name=\"vote_count\", type=SearchFieldDataType.Int32, sortable=True),\n", + " SearchableField(name=\"runtime\", type=SearchFieldDataType.Int32, sortable=True),\n", + " SearchableField(name=\"revenue\", type=SearchFieldDataType.Int64, sortable=True),\n", + " SearchableField(name=\"original_language\", type=SearchFieldDataType.String),\n", + " SearchField(name=\"vector\", type=SearchFieldDataType.Collection(SearchFieldDataType.Single), searchable=True, vector_search_dimensions=1536, vector_search_profile_name=\"movies-vector-profile\"),\n", + "]\n", + "\n", + "vector_search = VectorSearch(\n", + " profiles=[VectorSearchProfile(name=\"movies-vector-profile\", algorithm_configuration_name=\"movies-vector-config\")],\n", + " algorithms=[HnswAlgorithmConfiguration(name=\"movies-vector-config\")],\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We're going to be using Semantic Ranking, a feature of Azure AI Search that improves search results by using language understanding to rerank the search results. We provide a Semantic Search Configuration to help the ranking model understand the movie data, by telling it which fields contain the movie title, which fields contain keywords and which fields contain general free text content." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "semantic_config = SemanticConfiguration(\n", + " name=\"movies-semantic-config\",\n", + " prioritized_fields=SemanticPrioritizedFields(\n", + " title_field=SemanticField(field_name=\"title\"),\n", + " keywords_fields=[SemanticField(field_name=\"genre\")],\n", + " content_fields=[SemanticField(field_name=\"title\"),\n", + " SemanticField(field_name=\"overview\"),\n", + " SemanticField(field_name=\"tagline\"),\n", + " SemanticField(field_name=\"genre\"),\n", + " SemanticField(field_name=\"release_date\"),\n", + " SemanticField(field_name=\"popularity\"),\n", + " SemanticField(field_name=\"vote_average\"),\n", + " SemanticField(field_name=\"vote_count\"),\n", + " SemanticField(field_name=\"runtime\"),\n", + " SemanticField(field_name=\"revenue\"),\n", + " SemanticField(field_name=\"original_language\")],\n", + " )\n", + ")\n", + "\n", + "semantic_search = SemanticSearch(configurations=[semantic_config])" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Finally, we'll go ahead and create the index by creating an instance of the `SearchIndex` class and adding the keyword and vectors fields and the semantic search profile." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Create the search index with the desired vector search and semantic configurations\n", + "index = SearchIndex(\n", + " name=azure_ai_search_index_name,\n", + " fields=fields,\n", + " vector_search=vector_search,\n", + " semantic_search=semantic_search\n", + ")\n", + "\n", + "index_client = SearchIndexClient(\n", + " azure_ai_search_endpoint,\n", + " AzureKeyCredential(azure_ai_search_api_key)\n", + ")\n", + "\n", + "result = index_client.create_or_update_index(index)\n", + "\n", + "print(f'Index {result.name} created.')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The index is now ready, so next we need to prepare the movie data to load into the index.\n", + "\n", + "**NOTE**: During this phase, we send the data for each movie to an Azure OpenAI embeddings model to create the vector data. This may take some time due to rate limiting in the API." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Loop through all of the movies and create a new item for each one.\n", + "\n", + "items = []\n", + "for movie in data:\n", + " content = movie.page_content\n", + " fields = movie.page_content.split('\\n')\n", + " movieId = (fields[0].split(': ')[1])[:-2]\n", + " movieTitle = (fields[2].split(': ')[1])\n", + " movieOverview = (fields[8].split(': ')[1])\n", + " movieGenre = (fields[7].split(': ')[1])[1:-1]\n", + " movieTagline = (fields[11].split(': ')[1])\n", + " movieReleaseDate = (fields[4].split(': ')[1])\n", + " moviePopularity = (fields[3].split(': ')[1])\n", + " movieVoteAverage = (fields[5].split(': ')[1])\n", + " movieVoteCount = (fields[6].split(': ')[1])\n", + " movieRuntime = (fields[10].split(': ')[1])\n", + " movieRevenue = (fields[9].split(': ')[1])\n", + " movieOriginalLanguage = (fields[1].split(': ')[1])\n", + "\n", + " items.append(dict([\n", + " (\"id\", movieId), \n", + " (\"title\", movieTitle),\n", + " (\"overview\", movieOverview),\n", + " (\"genre\", movieGenre),\n", + " (\"tagline\", movieTagline),\n", + " (\"release_date\", movieReleaseDate),\n", + " (\"popularity\", moviePopularity),\n", + " (\"vote_average\", movieVoteAverage),\n", + " (\"vote_count\", movieVoteCount),\n", + " (\"runtime\", movieRuntime),\n", + " (\"revenue\", movieRevenue),\n", + " (\"original_language\", movieOriginalLanguage),\n", + " (\"vector\", azure_openai_embeddings.embed_query(content))\n", + " ]))\n", + "\n", + " print(f\"Movie {movieTitle} added.\")\n", + "\n", + "print(f\"New items structure with embeddings created for {len(items)} movies.\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can write out the contents of one of the documents to see what it looks like. You can see that it contains the movie data at the top and then a long array containing the vector data." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "print(items[0])" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now we have the movie data stored in the correct format, so let's load it into the Azure AI Search index we created earlier." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from azure.search.documents import SearchClient\n", + "\n", + "search_client = SearchClient(\n", + " azure_ai_search_endpoint,\n", + " azure_ai_search_index_name,\n", + " AzureKeyCredential(azure_ai_search_api_key)\n", + ")\n", + "\n", + "result = search_client.upload_documents(items)\n", + "\n", + "print(f\"Successfully loaded {len(data)} movies into Azure AI Search index.\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Vector store searching using Azure AI Search\n", + "\n", + "We've loaded the movies into Azure AI Search, so now let's experiment with some of the different types of searches you can perform.\n", + "\n", + "First we'll just perform a simple keyword search." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "query = \"hero\"\n", + "\n", + "results = list(search_client.search(\n", + " search_text=query,\n", + " query_type=\"simple\",\n", + " include_total_count=True,\n", + " top=5\n", + "))\n", + "\n", + "for result in results:\n", + " print(\"Movie: {}\".format(result[\"title\"]))\n", + " print(\"Genre: {}\".format(result[\"genre\"]))\n", + " print(\"----------\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We get some results, but they're not necessarily movies about heroes. It could be that there is some text in the index for these results that relates to the word \"hero\". For example, the description might mention \"heroic deeds\" or something similar.\n", + "\n", + "Let's now try the same again, but this time we'll ask a question instead of just searching for a keyword." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "query = \"What are the best movies about superheroes?\"\n", + "\n", + "results = list(search_client.search(\n", + " search_text=query,\n", + " query_type=\"simple\",\n", + " include_total_count=True,\n", + " top=5\n", + "))\n", + "\n", + "for result in results:\n", + " print(\"Movie: {}\".format(result[\"title\"]))\n", + " print(\"Genre: {}\".format(result[\"genre\"]))\n", + " print(\"----------\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "As before, you will likely get mixed results. Some of the movies returned could be about heroes, but others may not be. This is because the search is still based on keywords.\n", + "\n", + "Next, let's try a vector search." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "query = \"What are the best movies about superheroes?\"\n", + "\n", + "vector = VectorizedQuery(vector=azure_openai_embeddings.embed_query(query), k_nearest_neighbors=5, fields=\"vector\")\n", + "\n", + "# Note the `None` value for the `search_text` parameter. This is because we're not sending the query text to Azure AI Search. We're sending the embedded version of the query text instead via the `vector_queries` parameter.\n", + "\n", + "results = list(search_client.search(\n", + " search_text=None,\n", + " query_type=\"semantic\",\n", + " semantic_configuration_name=\"movies-semantic-config\",\n", + " vector_queries=[vector],\n", + " select=[\"title\", \"genre\"],\n", + " top=5\n", + "))\n", + "\n", + "for result in results:\n", + " print(\"Movie: {}\".format(result[\"title\"]))\n", + " print(\"Genre: {}\".format(result[\"genre\"]))\n", + " print(\"----------\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "It's likely that the raw vector search didn't return exactly what you were expecting. You were probably expecting a list of superhero movies, but now we're getting a list of movies that are **similar** to the vector we provided. Some of these may be hero movies, but others may not be. The vector search is returning the nearest neighbours to the vector we provided, so it's possible that at least one of the results is a superhero movie, and the others are similar to that movie in some way.\n", + "\n", + "So, both the keyword search and the vector search have their limitations. The keyword search is limited to the keywords in the index, so it's possible that we might miss some movies that are about heroes. The vector search is limited to returning the nearest neighbours to the vector we provide, so it's possible that we might get some movies that are not about heroes.\n", + "\n", + "## Hybrid search using Azure AI Search\n", + "\n", + "To overcome the limitations of both keyword search and vector search, we can use a combination of both. This is known as Hybrid Search. Let's run the same query again, but this time we'll use Hybrid Search.\n", + "\n", + "The only significant difference is that this time we will submit both the original query text and the embedding vector to Azure AI Search. Azure AI Search will then use both the query text and the vector to perform the search and combine the results." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "query = \"What are the best movies about superheroes?\"\n", + "\n", + "vector = VectorizedQuery(vector=azure_openai_embeddings.embed_query(query), k_nearest_neighbors=5, fields=\"vector\")\n", + "\n", + "# Note the `None` value for the `search_text` parameter. This is because we're not sending the query text to Azure AI Search. We're sending the embedded version of the query text instead via the `vector_queries` parameter.\n", + "\n", + "results = list(search_client.search(\n", + " search_text=query,\n", + " query_type=\"semantic\",\n", + " semantic_configuration_name=\"movies-semantic-config\",\n", + " vector_queries=[vector],\n", + " select=[\"title\", \"genre\"],\n", + " top=5\n", + "))\n", + "\n", + "for result in results:\n", + " print(\"Movie: {}\".format(result[\"title\"]))\n", + " print(\"Genre: {}\".format(result[\"genre\"]))\n", + " print(\"Score: {}\".format(result[\"@search.score\"]))\n", + " print(\"Reranked score: {}\".format(result[\"@search.reranker_score\"]))\n", + " print(\"----------\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Hopefully, you'll now see a much better set of results. Performing a hybrid search has allowed us to combine the benefits of both keyword search and vector search. But also, Azure AI Search performs a further step when using hybrid search. It makes use of a Semantic Ranker to further improve the search results. The Semantic Ranker uses a language understanding model to understand the query text and the documents in the index and then uses this information to rerank the search results. So, after performing the keyword and vector search, Azure AI Search will then use the Semantic Ranker to re-order the search results based on the context of the original query.\n", + "\n", + "In the results above, you can see a `Reranked Score`. This is the score that has been calculated by the Semantic Ranker. The `Score` is the score calculated by the keyword and vector search. You'll note that the results are returned in the order determined by the reranked score." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Bringing it All Together with Retrieval Augmented Generation (RAG) + Semantic Kernel (SK)\n", + "\n", + "Now that we have our Vector Store setup and data loaded, we are now ready to implement the RAG pattern using AI Orchestration. At a high-level, the following steps are required:\n", + "1. Ask the question\n", + "2. Create Prompt Template with inputs\n", + "3. Get Embedding representation of inputted question\n", + "4. Use embedded version of the question to search Azure Cognitive Search (ie. The Vector Store)\n", + "5. Inject the results of the search into the Prompt Template & Execute the Prompt to get the completion" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Implement RAG using Semantic Kernel (SK)\n", + "import asyncio\n", + "\n", + "import semantic_kernel as sk\n", + "from semantic_kernel.connectors.ai.open_ai import (\n", + " AzureChatCompletion,\n", + " AzureTextEmbedding,\n", + ")\n", + "from semantic_kernel.connectors.memory.azure_cognitive_search import (\n", + " AzureCognitiveSearchMemoryStore,\n", + ")\n", + "from semantic_kernel.semantic_functions.prompt_template import PromptTemplate\n", + "from semantic_kernel.semantic_functions.prompt_template_config import PromptTemplateConfig\n", + "from semantic_kernel.semantic_functions.semantic_function_config import SemanticFunctionConfig\n", + "from azure.search.documents.models import Vector\n", + "from azure.search.documents import SearchClient\n", + "from azure.search.documents.indexes import SearchIndexClient\n", + "from azure.search.documents.indexes.models import (\n", + " SearchIndex,\n", + " SearchField,\n", + " SearchFieldDataType,\n", + " SimpleField,\n", + " SearchableField,\n", + " SearchIndex,\n", + " SemanticConfiguration,\n", + " PrioritizedFields,\n", + " SemanticField,\n", + " SearchField,\n", + " SemanticSettings,\n", + " VectorSearch,\n", + " HnswVectorSearchAlgorithmConfiguration,\n", + " )\n", + "from azure.core.credentials import AzureKeyCredential\n", + "\n", + "# Setup Semantic Kernel\n", + "my_logger = sk.NullLogger()\n", + "kernel = sk.Kernel(\n", + " log=my_logger\n", + ")\n", + "\n", + "# Add Chat Service to SK\n", + "kernel.add_chat_service(\n", + " \"gpt35turbo\",\n", + " AzureChatCompletion(\n", + " deployment_name=azure_openai_completion_deployment_name,\n", + " endpoint=azure_openai_endpoint,\n", + " api_key=azure_openai_api_key,\n", + " logger=my_logger\n", + " )\n", + ")\n", + "# Add Text Embeddings Service to SK\n", + "embeddingsService = AzureTextEmbedding(\n", + " deployment_name=azure_openai_embedding_deployment_name,\n", + " endpoint=azure_openai_endpoint,\n", + " api_key=azure_openai_api_key,\n", + " logger=my_logger\n", + ")\n", + "kernel.add_text_embedding_generation_service(\n", + " \"ada002\",\n", + " embeddingsService\n", + ")\n", + "\n", + "# Ask the question\n", + "question = \"List the movies about ships on the water.\"\n", + "\n", + "# Create a prompt template with variables, note the double curly braces with dollar sign for the variables\n", + "sk_prompt = \"\"\"\n", + "Question: {{$original_question}}\n", + "\n", + "Do not use any other data.\n", + "Only use the movie data below when responding.\n", + "{{$search_results}}\n", + "\"\"\"\n", + "# Create the PromptTemplateConfig\n", + "prompt_config = PromptTemplateConfig(\n", + " description=\"Gets the intent of the user.\",\n", + " type=\"completion\",\n", + " input=PromptTemplateConfig.InputConfig(\n", + " parameters=[\n", + " PromptTemplateConfig.InputParameter(\n", + " name=\"original_question\",\n", + " description=\"The user's request.\",\n", + " default_value=\"\"\n", + " ),\n", + " PromptTemplateConfig.InputParameter(\n", + " name=\"search_results\",\n", + " description=\"Vector Search results from Azure Cognitive Search.\",\n", + " default_value=\"\"\n", + " )\n", + " ]\n", + " ),\n", + ")\n", + "# Create the SemanticFunctionConfig object\n", + "prompt_template = PromptTemplate(\n", + " template=sk_prompt,\n", + " template_engine=kernel.prompt_template_engine,\n", + " prompt_config=prompt_config,\n", + ")\n", + "function_config = SemanticFunctionConfig(prompt_config, prompt_template)\n", + "# Register the Semantic Function with SK\n", + "get_intent = kernel.register_semantic_function(\n", + " skill_name=\"CustomPlugin\",\n", + " function_name=\"GetIntent\",\n", + " function_config=function_config,\n", + ")\n", + "\n", + "# Get Embedding for the original question\n", + "question_embedded = await embeddingsService.generate_embeddings_async(question)\n", + "\n", + "# Search Vector Store\n", + "search_client = SearchClient(\n", + " azure_ai_search_endpoint,\n", + " azure_ai_search_index_name,\n", + " AzureKeyCredential(azure_ai_search_api_key)\n", + ")\n", + "vector = Vector(\n", + " value=question_embedded[0],\n", + " k=5,\n", + " fields=\"vector\"\n", + ")\n", + "results = list(search_client.search(\n", + " search_text=\"\",\n", + " include_total_count=True,\n", + " vectors=[vector],\n", + " select=[\"title\",\"genre\",\"overview\",\"tagline\",\"release_date\",\"popularity\",\"vote_average\",\"vote_count\",\"runtime\",\"revenue\",\"original_language\"],\n", + " ))\n", + "\n", + "# Build the Prompt and Execute against the Azure OpenAI to get the completion\n", + "# Initialize the prompt variables\n", + "variables = sk.ContextVariables()\n", + "variables[\"original_question\"] = question\n", + "variables[\"search_results\"] = str(results)\n", + "# Use SK Chaining to Invoke Semantic Function\n", + "completion = await kernel.run_async(\n", + " get_intent,\n", + " input_vars=variables\n", + ")\n", + "print(completion)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Next Section\n", + "\n", + "📣 [Deploy AI](../../04-deploy-ai/README.md)" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.6" + }, + "orig_nbformat": 4 + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/requirements.txt b/requirements.txt index ae108bc..05eb941 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,14 +1,15 @@ -azure-core==1.29.3 -azure-identity==1.14.0 -azure-search-documents==11.4.0b8 -semantic-kernel==0.3.12.dev0 -openai==0.27.9 -langchain==0.0.329 -tiktoken==0.4.0 +azure-core==1.29.7 +azure-identity==1.15.0 +azure-search-documents==11.4.0 +semantic-kernel==0.4.5.dev0 +openai==1.9.0 +langchain==0.1.1 +langchain-openai==0.0.3 +tiktoken==0.5.2 python-dotenv==1.0.0 requests==2.31.0 -unstructured==0.10.5 -markdown==3.4.4 -qdrant-client==1.4.0 -chromadb==0.4.6 -yfinance==0.2.32 \ No newline at end of file +unstructured==0.12.2 +markdown==3.5.2 +qdrant-client==1.7.1 +chromadb==0.4.22 +yfinance==0.2.36 \ No newline at end of file