issue #9, Developing a Search Function Test Utilizing LLM

.devcontainer/devcontainer.json

@@ -19,7 +19,8 @@
 			"extensions": [
 				"timonwong.shellcheck",
 				"GitHub.vscode-pull-request-github",
-				"charliermarsh.ruff"
+				"charliermarsh.ruff",
+				"ms-python.black-formatter"
 			]
 		}
 	},

.env.template

@@ -5,8 +5,10 @@ USER=
 PGHOST=
 POSTGRES_PASSWORD=
 PGPASSWORD=
+PGDATA=
 OPENAI_API_KEY=
-AZURE_OPENAI_SERVICE=
+OPENAI_ENDPOINT=
+OPENAI_API_ENGINE=
 LOUIS_SCHEMA=
 DB_SERVER_CONTAINER_NAME=
-PGDATA=
+FINESSE_WEIGHTS=

.github/workflows/pull-request.yml

@@ -14,7 +14,7 @@ jobs:
     secrets: inherit
 
   build:
-    # if: github.ref == 'refs/heads/main'
+    if: github.ref == 'refs/heads/main'
     needs: lint-test
     uses: ai-cfia/github-workflows/.github/workflows/workflow-build-container.yml@main
     with:
@@ -23,7 +23,7 @@ jobs:
     secrets: inherit
 
   deploy:
-    # if: github.ref == 'refs/heads/main'
+    if: github.ref == 'refs/heads/main'
     needs: build
     uses: ai-cfia/github-workflows/.github/workflows/workflow-deploy-gcp.yml@main
     with:

DEVELOPER.md

@@ -24,14 +24,16 @@
 
   * OPENAI_API_KEY: The API key required for authentication when making requests to the OpenAI API. It can be found [here](https://portal.azure.com/#home).
 
-  * AZURE_OPENAI_ENDPOINT: The link used to call into Azure OpenAI endpoints. It can be found at the same place as the OPENAI_API_KEY.
+  * OPENAI_ENDPOINT: The link used to call into Azure OpenAI endpoints. It can be found at the same place as the OPENAI_API_KEY.
 
   * OPENAI_API_ENGINE: The name of the model deployment you want to use (ex:ailab-gpt-35-turbo).
 
   * LOUIS_SCHEMA: The Louis schema within database (ex: louis_v005).
 
   * DB_SERVER_CONTAINER_NAME: The name of your database server container (ex: louis-db-server).
 
+  * AILAB_SCHEMA_VERSION: The version of the schema you want to use.
+
 * Run database locally (see bin/postgres.sh)
 * Restore latest schema dump
 

README.md

@@ -1,22 +1,32 @@
-# What is louis-db ?
+# What is ailab-db ?
 
-Louis-db is the database of [Louis](https://github.com/ai-cfia/louis). It includes all of the python and bash scripts as well as sql functions. It uses ada-002 API (Application Programming Interface) for semantic search. Ada is a model that vectorises text to make a semantic representation out of it. It follows this process : 
-> Taking the text, passing it through the model, putting it in an embedding table that will work like an index.
+Ailab-db contains the database of [Louis](https://github.com/ai-cfia/louis),
+[Nachet](https://github.com/ai-cfia/nachet-backend) and any other product of the
+CFIA's AI Lab. It includes all of the python and bash scripts as well as sql
+functions. It uses ada-002 API (Application Programming Interface) for semantic
+search. Ada is a model that vectorises text to make a semantic representation
+out of it. It follows this process : 
+> Taking the text, passing it through the model, putting it in an embedding
+> table that will work like an index.
 
-- The *bin* folder : This folder contains all of the bash scripts that would be useful to set up the database.
+- The *bin* folder : This folder contains all of the bash scripts that would be
+  useful for the backends or to set up the database.
 
-- The *louis* folder : This folder is a python module structure that allows connections to the database or the api.
+- The *ailab* folder : This folder is a python module structure that allows
+  connections to the database or the api as well as containing useful functions
+  for product backends.
 
-- The *postgres* folder : This folder contains the bash script that will set up the docker container.
+- The *postgres* folder : This folder contains the bash script that will set up
+  the docker container.
 
 - The *sql* folder : This folder holds all of the sql functions and scripts.
 
 - The *tests* folder : This is the test folder, it allows you to test the code.
 
-Here is the database schema :
-![database schema](img/database-schema.png)
+Here is the database schema : ![database schema](img/database-schema.png)
 
-If you need to set up the database locally, please follow [this procedure](setup-procedure.md).
+If you need to set up the database locally, please follow [this
+procedure](setup-procedure.md).
 
 ---
 
@@ -27,17 +37,20 @@ If you need to set up the database locally, please follow [this procedure](setup
 If you need to interface with the database, use this to install:
 
 ```
-pip install git+https://github.com/ai-cfia/louis[email protected]
+pip install git+https://github.com/ai-cfia/ailab[email protected]
 ```
 
-You'll often want to add, move or modify existing database layer functions found in louis-db from a client repository.
+You'll often want to add, move or modify existing database layer functions found
+in ailab-db from a client repository.
 
-To edit, you can install an editable version of the package dependencies such as:
+To edit, you can install an editable version of the package dependencies such
+as:
 
 ```
 pip install -e git+https://github.com/ai-cfia/louis-db#egg=louis_db
 ```
 
-this will checkout the latest source in a local git in src/louis-db allowing edits in that directory to be immediately available for use by louis-crawler.
+this will checkout the latest source in a local git in src/louis-db allowing
+edits in that directory to be immediately available for use by louis-crawler.
 
-Don't forget to create a PR with your changes once you're done!
+Don't forget to create a PR with your changes once you're done!

ailab/db/__init__.py

@@ -1,4 +1,4 @@
-"""Database functions for the Louis project."""
+"""Database functions for the ailab project."""
 import hashlib
 import logging
 import os
@@ -72,4 +72,4 @@ def hash(text):
 
     We hash using the Python library to remove a roundtrip to the database
     """
-    return hashlib.md5(text.encode()).hexdigest()
+    return hashlib.md5(text.encode()).hexdigest()

ailab/db/api/__init__.py

@@ -1,14 +1,14 @@
-import louis.models.openai as openai
+import ailab.models.openai as openai
 import os
 import json
 import sys
-import louis.db as db
+import ailab.db as db
 import dotenv
 # This is used to load the .env file
 dotenv.load_dotenv()
 
 FINESSE_WEIGHTS = os.environ.get("FINESSE_WEIGHTS") \
-    or db.raise_db_error("FINESSE_WEIGHTS is not set")
+    or db.raise_error("FINESSE_WEIGHTS is not set")
 
 
 if FINESSE_WEIGHTS:
@@ -71,7 +71,7 @@ def search(cursor, query_embedding):
         'text': ' '.join(sys.argv[1:]),
         'query_embedding': query_embedding,
         'match_threshold': 0.5,
-        'match_count': 1,
+        'match_count': 10,
         'weights': json.dumps(FINESSE_JSON_PARSED_WEIGHTS)
     }
 
@@ -105,4 +105,4 @@ def search_from_text_query(cursor, query):
     else:
         data.update(db_data)
     docs = search(cursor, data['embedding'])
-    return docs
+    return docs

ailab/db/crawler/__init__.py

@@ -1,7 +1,7 @@
 import psycopg
 import numpy as np
 
-import louis.db as db
+import ailab.db as db
 
 def link_pages(cursor, source_url, destination_url):
     """Link two pages together in the database."""

ailab/db/finesse/__init__.py

@@ -0,0 +1,32 @@
+import os
+import json
+
+# Constants
+DEFAULT_JSON_PATH = "/ailab/db/finesse/prompt"
+
+
+def load_prompt(prompt_path, filename):
+    full_path = os.path.join(prompt_path, filename)
+
+    with open(full_path, "r") as file:
+        content = file.read()
+
+    return content
+
+
+def load_json_template(json_path=DEFAULT_JSON_PATH):
+    """
+    Load a JSON template from the specified path.
+
+    Returns:
+    str: A JSON-formatted string representing the loaded data.
+    """
+    json_file_path = json_path + "/json_template.json"
+
+    with open(json_file_path, "r") as file:
+        data = json.load(file)  # Load the JSON data as a Python dictionary
+
+    # Convert the dictionary back to a JSON-formatted string
+    content = json.dumps(data, indent=4)  # Use 'indent' for pretty printing
+
+    return content

ailab/db/finesse/prompt/json_template.json

@@ -0,0 +1,10 @@
+{
+    "page_score": "",
+    "crawl_id": "",
+    "chunk_id": "",
+    "title": "",
+    "url": "",
+    "text_content": "",
+    "question": "",
+    "answer": ""
+}

ailab/db/finesse/prompt/qna_system_prompt.txt

@@ -0,0 +1 @@
+Generate Q&A pairs from search results, ensuring that the language in the Q&A pairs matches that of the search results. Create a single JSON file with appropriate formatting and indentation. Always leave "text_content" empty. If you got more than one query, only treat and return result for the first one. Always return a SINGLE JSON file.

ailab/db/finesse/prompt/qna_user_prompt.txt

@@ -0,0 +1 @@
+Your task is to process a JSON file containing text relevant to a search query. Produce a new JSON file completing all existing keys in the JSON template. Specifically, generate entries for 'question' and 'answer' where 'question' represents a question derived from the text and 'answer' provides an appropriate answer addressing the generated question.

ailab/db/finesse/test_queries/__init__.py

@@ -0,0 +1,46 @@
+import time
+import semver
+import math
+
+
+def get_random_chunk(cursor, schema_version, seed=None):
+    assert (
+        semver.compare(schema_version, "0.0.6") >= 0
+    ), "Schema version must be >= 0.0.6"
+    schema_version = "louis_" + schema_version
+
+    if seed is None:
+        seed = math.sin(time.time())
+
+    # Execute the SET commands separately
+    cursor.execute(f'SET SEARCH_PATH TO "{schema_version}", public;')
+    cursor.execute(f"SET SEED TO {seed};")
+
+    query = """
+        SELECT
+            dc.score AS score, cr.id AS crawl_id, ch.id AS chunk_id, ch.title, cr.url, ch.text_content
+        FROM
+            Chunk ch
+        INNER JOIN
+            documents dc ON ch.id = dc.chunk_id
+        INNER JOIN
+            html_content_to_chunk hctc ON ch.id = hctc.chunk_id
+        INNER JOIN
+            html_content hc ON hctc.md5hash = hc.md5hash
+        INNER JOIN
+            crawl cr ON hc.md5hash = cr.md5hash
+        WHERE
+            dc.score > 0.01
+        ORDER BY
+            floor(random() * (
+                SELECT
+                    COUNT(*)
+                FROM
+                    Chunk
+            ))
+        LIMIT
+            1;
+    """
+
+    cursor.execute(query)
+    return cursor.fetchall()

ailab/models/openai.py

@@ -56,4 +56,4 @@ def get_chat_answer(system_prompt, user_prompt, max_token):
             {"role": "user", "content": (user_prompt)}
         ]
     )
-    return response
+    return response

benchmarking/search_results.json

@@ -1,2 +1,70 @@
-{"commit_version": "b2badf1069c3feb0475a4b56da8ddb4d61ff06f0", "function_version": "0.0.1", "args": ["\"how to bring a cat from france to canada\""], "kwargs": {}, "function_name": "init_bench", "start_time": "2023-10-30T20:06:34.197284", "finish_time": "2023-10-30T20:06:34.495414", "return_value": [{"search": [{"id": "e9a64ea3-c7c4-4f0e-bd49-c4e1d9de1a7f", "url": "https://inspection.canada.ca/importing-food-plants-or-animals/pets/eng/1326600389775/1326600500578", "score": 3.017821829567311, "title": "Bringing animals to Canada: Importing and travelling with pets - Canadian Food Inspection Agency", "scores": {"current": 1, "recency": 0.8708793589157114, "traffic": 0.8188254189764937, "similarity": 0.8272663498460967, "typicality": 0.004253509145044662}, "content": "Bringing animals to Canada: Importing and travelling with pets If you are travelling with a pet or planning to import an animal to Canada, you will need the right paperwork at the border to meet Canada's import requirements\n If you don't, you risk experiencing delays at the border and your animal may not be allowed into Canada\n Canada has specific import requirements in place to avoid introducing animal diseases to protect its people, plants and animals", "query_id": "9f5e33a0-6b81-48da-9681-349dd6d3bb9c", "subtitle": "Bringing animals to Canada: Importing and travelling with pets", "last_updated": "2022-10-20", "tokens_count": 84}]}]}
-{"commit_version": "f8c9d7a72944fe0c94fd41bccd2633fc17a11d8c", "function_version": "0.0.1", "args": ["\"how to bring a cat from france to canada\""], "kwargs": {}, "function_name": "init_bench", "start_time": "2023-10-31T14:34:26.463150", "finish_time": "2023-10-31T14:34:32.552876", "return_value": [{"search": [{"id": "e9a64ea3-c7c4-4f0e-bd49-c4e1d9de1a7f", "url": "https://inspection.canada.ca/importing-food-plants-or-animals/pets/eng/1326600389775/1326600500578", "score": 3.017821829567311, "title": "Bringing animals to Canada: Importing and travelling with pets - Canadian Food Inspection Agency", "scores": {"current": 1, "recency": 0.8708793589157114, "traffic": 0.8188254189764937, "similarity": 0.8272663498460967, "typicality": 0.004253509145044662}, "content": "Bringing animals to Canada: Importing and travelling with pets If you are travelling with a pet or planning to import an animal to Canada, you will need the right paperwork at the border to meet Canada's import requirements\n If you don't, you risk experiencing delays at the border and your animal may not be allowed into Canada\n Canada has specific import requirements in place to avoid introducing animal diseases to protect its people, plants and animals", "query_id": "b7b0a221-1239-48e9-98f7-016d75946b38", "subtitle": "Bringing animals to Canada: Importing and travelling with pets", "last_updated": "2022-10-20", "tokens_count": 84}]}]}
+{
+    "commit_version": "b2badf1069c3feb0475a4b56da8ddb4d61ff06f0",
+    "function_version": "0.0.1",
+    "args": [
+        "\"how to bring a cat from france to canada\""
+    ],
+    "kwargs": {},
+    "function_name": "init_bench",
+    "start_time": "2023-10-30T20:06:34.197284",
+    "finish_time": "2023-10-30T20:06:34.495414",
+    "return_value": [
+        {
+            "search": [
+                {
+                    "id": "e9a64ea3-c7c4-4f0e-bd49-c4e1d9de1a7f",
+                    "url": "https://inspection.canada.ca/importing-food-plants-or-animals/pets/eng/1326600389775/1326600500578",
+                    "score": 3.017821829567311,
+                    "title": "Bringing animals to Canada: Importing and travelling with pets - Canadian Food Inspection Agency",
+                    "scores": {
+                        "current": 1,
+                        "recency": 0.8708793589157114,
+                        "traffic": 0.8188254189764937,
+                        "similarity": 0.8272663498460967,
+                        "typicality": 0.004253509145044662
+                    },
+                    "content": "Bringing animals to Canada: Importing and travelling with pets If you are travelling with a pet or planning to import an animal to Canada, you will need the right paperwork at the border to meet Canada's import requirements\n If you don't, you risk experiencing delays at the border and your animal may not be allowed into Canada\n Canada has specific import requirements in place to avoid introducing animal diseases to protect its people, plants and animals",
+                    "query_id": "9f5e33a0-6b81-48da-9681-349dd6d3bb9c",
+                    "subtitle": "Bringing animals to Canada: Importing and travelling with pets",
+                    "last_updated": "2022-10-20",
+                    "tokens_count": 84
+                }
+            ]
+        }
+    ]
+}
+{
+    "commit_version": "f8c9d7a72944fe0c94fd41bccd2633fc17a11d8c",
+    "function_version": "0.0.1",
+    "args": [
+        "\"how to bring a cat from france to canada\""
+    ],
+    "kwargs": {},
+    "function_name": "init_bench",
+    "start_time": "2023-10-31T14:34:26.463150",
+    "finish_time": "2023-10-31T14:34:32.552876",
+    "return_value": [
+        {
+            "search": [
+                {
+                    "id": "e9a64ea3-c7c4-4f0e-bd49-c4e1d9de1a7f",
+                    "url": "https://inspection.canada.ca/importing-food-plants-or-animals/pets/eng/1326600389775/1326600500578",
+                    "score": 3.017821829567311,
+                    "title": "Bringing animals to Canada: Importing and travelling with pets - Canadian Food Inspection Agency",
+                    "scores": {
+                        "current": 1,
+                        "recency": 0.8708793589157114,
+                        "traffic": 0.8188254189764937,
+                        "similarity": 0.8272663498460967,
+                        "typicality": 0.004253509145044662
+                    },
+                    "content": "Bringing animals to Canada: Importing and travelling with pets If you are travelling with a pet or planning to import an animal to Canada, you will need the right paperwork at the border to meet Canada's import requirements\n If you don't, you risk experiencing delays at the border and your animal may not be allowed into Canada\n Canada has specific import requirements in place to avoid introducing animal diseases to protect its people, plants and animals",
+                    "query_id": "b7b0a221-1239-48e9-98f7-016d75946b38",
+                    "subtitle": "Bringing animals to Canada: Importing and travelling with pets",
+                    "last_updated": "2022-10-20",
+                    "tokens_count": 84
+                }
+            ]
+        }
+    ]
+}

bin/README.md

@@ -6,7 +6,7 @@ This assumes:
 
 * you are running WSL
 * you are running a dockerized version of Postgresql 15 under WSL
-* you are running louis-db in a DevContainer under Visual Studio Code
+* you are running ailab-db in a DevContainer under Visual Studio Code
 * your source is on WSL under ~/src
 
 ## configuration
@@ -61,4 +61,4 @@ validate manually that schema is as expected here (dbBeaver ERD diagram) before
 
 ```
 ./bin/load-versioned-data.sh louis_v005
-```
+```

bin/deprecated/migrate.sh

@@ -1,4 +1,3 @@
-
 set -x
 
 SCHEMA_CHANGES=sql/schema2.sql
@@ -34,4 +33,4 @@ if [ "$?" -eq 0 ]; then
 else
     echo "update schema failed"
     exit 1
-fi
+fi

bin/generate-qna.sh

@@ -0,0 +1,7 @@
+#!/bin/bash
+DIRNAME=$(dirname "$0")
+. "$DIRNAME"/lib.sh
+
+PROMPT_PATH=$PROJECT_DIR/ailab/db/finesse/prompt
+
+PYTHONPATH=$PROJECT_DIR python "$DIRNAME"/generate_qna.py "$PROMPT_PATH" --storage_path "../qna-test"