docs: Add a readme for the pgai docs.

.github/workflows/ci.yml

@@ -27,7 +27,6 @@ jobs:
         uses: actions/setup-python@v5
         with:
           python-version: "3.10"
-          cache: "pip" # caching pip dependencies
 
       - name: Verify Docker installation
         run: |

docs/README.md

@@ -0,0 +1,73 @@
+
+<p align="center">
+    <img height="200" src="/docs/images/pgai_logo.png" alt="pgai"/>
+</p>
+
+<div align=center>
+
+<h3>pgai documentation</h3>
+
+[![Discord](https://img.shields.io/badge/Join_us_on_Discord-black?style=for-the-badge&logo=discord&logoColor=white)](https://discord.gg/KRdHVXAmkp)
+[![Try Timescale for free](https://img.shields.io/badge/Try_Timescale_for_free-black?style=for-the-badge&logo=timescale&logoColor=white)](https://tsdb.co/gh-pgai-signup)
+</div>
+
+pgai is a PostgreSQL extension that simplifies data storage and retrieval for [Retrieval Augmented Generation](https://en.wikipedia.org/wiki/Prompt_engineering#Retrieval-augmented_generation) (RAG), and other AI applications.
+In particular, it automates the creation and sync of embeddings for your data stored in PostgreSQL, simplifies
+[semantic search](https://en.wikipedia.org/wiki/Semantic_search), and allows you to call LLM models from SQL.
+
+The pgai documentation helps you setup, use and develop the projects that make up pgai.  
+
+
+## pgai Vectorizer
+
+Vectorizer automates the embedding process within your database management by treating embeddings as a declarative,
+DDL-like feature — like an index.
+
+- **Get started**:
+  * [Vectorizer quickstart for Ollama](/docs/vectorizer-quick-start.md): setup your developer environment, create and run a vectorizer.
+  * [Vectorizer quickstart for OpenAI](/docs/vectorizer-quick-start-openai.md): setup your developer environment, create and run a vectorizer using OpenAI.
+  * [Vectorizer quickstart for Voyage](/docs/vectorizer-quick-start-voyage.md): setup your developer environment, create and run a vectorizer using Voyage. 
+- **Use**:
+  * [Automate AI embedding with pgai Vectorizer](/docs/vectorizer.md): a comprehensive overview of Vectorizer features,
+    demonstrating how it streamlines the process of working with vector embeddings in your database.
+  * [Run vectorizers using pgai vectorizer worker](/docs/vectorizer-worker.md): run vectorizers on a self-hosted TimescaleDB instance.
+- **Develop**:
+  * [Add a Vectorizer embedding integration](/docs/vectorizer-add-a-embedding-integration.md):
+- **Reference**:
+  * [pgai Vectorizer API reference](/docs/vectorizer-api-reference.md): API reference for Vectorizer functions 
+
+## pgai install
+
+* [Install pgai with Docker](/docs/install_docker.md): run pgai in a container environment.
+* [Install pgai from source](/docs/install_from_source.md): create a developer environment for pgai.
+
+## pgai model calling
+
+Simplifies data storage and retrieval for AI apps. 
+
+- **Choose your model**:
+
+  | **Model**       | **Tokenize** | **Embed** | **Chat Complete** | **Generate** | **Moderate** | **Classify** | **Rerank** |
+  |------------------|:------------:|:---------:|:-----------------:|:------------:|:------------:|:------------:|:----------:|
+  | **[Ollama](/docs/ollama.md)**      |              |     ✔️    |         ✔️         |      ✔️      |              |              |           |
+  | **[OpenAI](/docs/openai.md)**      |      ✔️️      |     ✔️    |         ✔️         |              |      ✔️      |              |           |
+  | **[Anthropic](/docs/anthropic.md)**  |              |           |                    |      ✔️      |              |              |           |
+  | **[Cohere](/docs/cohere.md)**      |      ✔️      |     ✔️    |         ✔️         |              |              |      ✔️      |     ✔️     |
+  | **[Voyage AI](/docs/voyageai.md)** |              |     ✔️    |                  |              |              |              |           |
+
+
+- **Use**:
+  * [Delayed embed](/docs/delayed_embed.md): run pgai using pgai or TimescaleDB background actions.
+  * [Load dataset from Hugging Face](/docs/load_dataset_from_huggingface.md): load datasets from Hugging Face's datasets library directly into your PostgreSQL database.
+  * [Moderate comments using OpenAI](/docs/moderate.md): use triggers or actions to moderate comments using OpenAI.
+  * [Secure pgai with user privilages](/docs/privileges.md): grant the necessary permissions for a specific user or role to use pgai functionality.
+
+
+
+
+
+
+
+
+
+

docs/moderate.md

@@ -1,4 +1,4 @@
-# Moderate
+# Moderate comments using OpenAI
 
 Let's say you want to moderate comments using OpenAI. You can do it in two ways:
 

docs/privileges.md

@@ -1,3 +1,4 @@
+# Secure pgai with user privilages
 
 The ai.grant_ai_usage function is an important security and access control tool 
 in the pgai extension. Its primary purpose is to grant the necessary permissions 

docs/adding-embedding-integration.md → docs/vectorizer-add-a-embedding-integration.md

@@ -1,4 +1,4 @@
-# Adding a Vectorizer embedding integration
+# Add a Vectorizer embedding integration
 
 We welcome contributions to add new vectorizer embedding integrations.
 
@@ -31,13 +31,17 @@ integration. Update the tests to account for the new function.
 The vectorizer worker reads the database's vectorizer configuration at runtime
 and turns it into a `pgai.vectorizer.Config`.
 
-To add a new integration, add a new embedding class with fields corresponding
-to the database's jsonb configuration to `pgai/vectorizer/embeddings.py`. See
+To add a new integration, add a new file containing the embedding class 
+with fields corresponding to the database's jsonb configuration into the 
+[embedders directory] directory. See
 the existing implementations for examples of how to do this. Implement the
 `Embedder` class' abstract methods. Use first-party python libraries for the
 integration, if available. If no first-party python libraries are available,
 use direct HTTP requests.
 
+Remember to include the import line of your recently created class into the 
+[embedders \_\_init\_\_.py].
+
 Add tests which perform end-to-end testing of the new integration. There are
 two options for handling API calls to the integration API:
 
@@ -49,6 +53,8 @@ used conservatively. We will determine on a case-by-case basis what level of
 testing we would like.
 
 [vcr.py]:https://vcrpy.readthedocs.io/en/latest/
+[embedders directory]:/projects/pgai/pgai/vectorizer/embedders
+[embedders \_\_init\_\_.py]:/projects/pgai/pgai/vectorizer/embedders/__init__.py
 
 ## Documentation
 

docs/vectorizer-api-reference.md

@@ -602,7 +602,7 @@ HNSW is suitable for in-memory datasets and scenarios where query speed is cruci
 ```sql
   SELECT ai.create_vectorizer(
       'blog_posts'::regclass,
-      indexing => ai.indexing_hnsw(min_rows => 50000, opclass => 'vector_l2_ops'),
+      indexing => ai.indexing_hnsw(min_rows => 50000, opclass => 'vector_l1_ops'),
       -- other parameters...
   );
 ```
@@ -614,10 +614,10 @@ HNSW is suitable for in-memory datasets and scenarios where query speed is cruci
 | Name | Type | Default             | Required | Description                                                                                                    |
 |------|------|---------------------|-|----------------------------------------------------------------------------------------------------------------|
 |min_rows| int  | 100000              |✖| The minimum number of rows before creating the index                                                           |
-|opclass| text  | `vector_cosine_ops` |✖| The operator class for the index. Possible values are:`vector_cosine_ops`, `vector_l2_ops`, or `vector_ip_ops` |
+|opclass| text  | `vector_cosine_ops` |✖| The operator class for the index. Possible values are:`vector_cosine_ops`, `vector_l1_ops`, or `vector_ip_ops` |
 |m| int  | -                   |✖| Advanced [HNSW parameters](https://en.wikipedia.org/wiki/Hierarchical_navigable_small_world)                   |
-|ef_construction| int  | -                   |✖|  Advanced [HNSW parameters](https://en.wikipedia.org/wiki/Hierarchical_navigable_small_world)                                                                                                              |
-| create_when_queue_empty| boolean | true |✖| Create the index only after all of the embeddings have been generated. |
+|ef_construction| int  | -                   |✖| Advanced [HNSW parameters](https://en.wikipedia.org/wiki/Hierarchical_navigable_small_world)                   |
+| create_when_queue_empty| boolean | true |✖| Create the index only after all of the embeddings have been generated.                                         |
 
 
 #### Returns

docs/vectorizer-quick-start.md

@@ -5,7 +5,7 @@ If you prefer working with the OpenAI API instead of self-hosting models, you ca
 
 ## Setup a local development environment
 
-To set up a development environment for OpenAI, use a docker compose file that includes a:
+To set up a development environment, use a docker compose file that includes a:
 - Postgres deployment image with the TimescaleDB and pgai extensions installed
 - pgai vectorizer worker image
 - ollama image to host embedding and large language models

projects/extension/Dockerfile

@@ -47,6 +47,7 @@ RUN set -e; \
 FROM base AS pgai-test-db
 ENV PG_MAJOR=${PG_MAJOR}
 ENV PIP_BREAK_SYSTEM_PACKAGES=1
+RUN pip install uv==0.5.9
 WORKDIR /pgai
 COPY . .
 RUN just build install
@@ -58,17 +59,17 @@ FROM base
 ENV WHERE_AM_I=docker
 USER root
 
+RUN pip install --break-system-packages uv==0.5.9
+
 # install pgspot
-ENV PIP_BREAK_SYSTEM_PACKAGES=1
 RUN set -eux; \
     git clone https://github.com/timescale/pgspot.git /build/pgspot; \
-    pip install /build/pgspot; \
+    uv pip install --system --break-system-packages /build/pgspot; \
     rm -rf /build/pgspot
 
 # install our test python dependencies
-ENV PIP_BREAK_SYSTEM_PACKAGES=1
 COPY requirements-test.txt /build/requirements-test.txt
-RUN pip install -r /build/requirements-test.txt
+RUN uv pip install --system --break-system-packages -r /build/requirements-test.txt
 RUN rm -r /build
 
 WORKDIR /pgai

projects/extension/build.py

@@ -83,7 +83,7 @@ def check_versions():
 
 
 def parse_version(version: str) -> tuple[int, int, int, str | None]:
-    parts = re.split(r"[.-]", version, 4)
+    parts = re.split(r"[.-]", version, maxsplit=4)
     return (
         int(parts[0]),
         int(parts[1]),
@@ -283,6 +283,13 @@ def build_idempotent_sql_file(input_file: Path) -> str:
         r = plpy.execute("select coalesce(pg_catalog.current_setting('ai.python_lib_dir', true), '{python_install_dir()}') as python_lib_dir")
         python_lib_dir = r[0]["python_lib_dir"]
         from pathlib import Path
+        import sys
+        import sysconfig
+        # Note: the "old" (pre-0.4.0) packages are installed as system-level python packages
+        # and take precedence over our extension-version specific packages.
+        # By removing the whole thing from the path we won't run into package conflicts.
+        if "purelib" in sysconfig.get_path_names() and sysconfig.get_path("purelib") in sys.path:
+            sys.path.remove(sysconfig.get_path("purelib"))
         python_lib_dir = Path(python_lib_dir).joinpath("{this_version()}")
         import site
         site.addsitedir(str(python_lib_dir))
@@ -451,9 +458,13 @@ def install_old_py_deps() -> None:
     old_reqs_file = ext_dir().joinpath("old_requirements.txt").resolve()
     if old_reqs_file.is_file():
         env = {k: v for k, v in os.environ.items()}
-        env["PIP_BREAK_SYSTEM_PACKAGES"] = "1"
+        cmd = (
+            f"pip3 install -v --compile --break-system-packages -r {old_reqs_file}"
+            if shutil.which("uv") is None
+            else f"uv pip install -v --compile --system --break-system-packages -r {old_reqs_file}"
+        )
         subprocess.run(
-            f"pip3 install -v --compile -r {old_reqs_file}",
+            cmd,
             shell=True,
             check=True,
             env=env,
@@ -482,8 +493,10 @@ def install_prior_py() -> None:
             env=os.environ,
         )
         tmp_src_dir = tmp_dir.joinpath("projects", "extension").resolve()
+        bin = "pip3" if shutil.which("uv") is None else "uv pip"
+        cmd = f'{bin} install -v --compile --target "{version_target_dir}" "{tmp_src_dir}"'
         subprocess.run(
-            f'pip3 install -v --compile -t "{version_target_dir}" "{tmp_src_dir}"',
+            cmd,
             check=True,
             shell=True,
             env=os.environ,
@@ -524,17 +537,23 @@ def install_py() -> None:
             "pgai-*.dist-info"
         ):  # delete package info if exists
             shutil.rmtree(d)
+        bin = "pip3" if shutil.which("uv") is None else "uv pip"
+        cmd = f'{bin} install -v --no-deps --compile --target "{version_target_dir}" "{ext_dir()}"'
         subprocess.run(
-            f'pip3 install -v --no-deps --compile -t "{version_target_dir}" "{ext_dir()}"',
+            cmd,
             check=True,
             shell=True,
             env=os.environ,
             cwd=str(ext_dir()),
         )
     else:
         version_target_dir.mkdir(exist_ok=True)
+        bin = "pip3" if shutil.which("uv") is None else "uv pip"
+        cmd = (
+            f'{bin} install -v --compile --target "{version_target_dir}" "{ext_dir()}"'
+        )
         subprocess.run(
-            f'pip3 install -v --compile -t "{version_target_dir}" "{ext_dir()}"',
+            cmd,
             check=True,
             shell=True,
             env=os.environ,
@@ -633,7 +652,7 @@ def lint_py() -> None:
 
 def lint() -> None:
     lint_py()
-    lint_sql()
+    # lint_sql()  # TODO: enable this when pgspot is fixed
 
 
 def format_py() -> None:

projects/extension/requirements-test.txt

@@ -7,3 +7,4 @@ python-dotenv==1.0.1
 fastapi==0.112.0
 fastapi-cli==0.0.5
 psycopg[binary]==3.2.1
+uv==0.5.9

projects/extension/sql/idempotent/013-vectorizer-api.sql

@@ -596,3 +596,68 @@ select
   end as pending_items
 from ai.vectorizer v
 ;
+
+-------------------------------------------------------------------------------
+-- vectorizer_embed
+create or replace function ai.vectorizer_embed
+( embedding_config pg_catalog.jsonb
+, input_text pg_catalog.text
+, input_type pg_catalog.text default null
+) returns @extschema:[email protected]
+as $func$
+declare
+    _emb @extschema:[email protected];
+begin
+    case embedding_config operator(pg_catalog.->>) 'implementation'
+        when 'openai' then
+            _emb = ai.openai_embed
+            ( embedding_config operator(pg_catalog.->>) 'model'
+            , input_text
+            , api_key_name=>(embedding_config operator(pg_catalog.->>) 'api_key_name')
+            , dimensions=>(embedding_config operator(pg_catalog.->>) 'dimensions')::pg_catalog.int4
+            , openai_user=>(embedding_config operator(pg_catalog.->>) 'user')
+            );
+        when 'ollama' then
+            _emb = ai.ollama_embed
+            ( embedding_config operator(pg_catalog.->>) 'model'
+            , input_text
+            , host=>(embedding_config operator(pg_catalog.->>) 'base_url')
+            , keep_alive=>(embedding_config operator(pg_catalog.->>) 'keep_alive')
+            , embedding_options=>(embedding_config operator(pg_catalog.->) 'options')
+            );
+        when 'voyageai' then
+            _emb = ai.voyageai_embed
+            ( embedding_config operator(pg_catalog.->>) 'model'
+            , input_text
+            , input_type=>coalesce(input_type, 'query')
+            , api_key_name=>(embedding_config operator(pg_catalog.->>) 'api_key_name')
+            );
+        else
+            raise exception 'unsupported embedding implementation';
+    end case;
+
+    return _emb;
+end
+$func$ language plpgsql immutable security invoker
+set search_path to pg_catalog, pg_temp
+;
+
+-------------------------------------------------------------------------------
+-- vectorizer_embed
+create or replace function ai.vectorizer_embed
+( vectorizer_id pg_catalog.int4
+, input_text pg_catalog.text
+, input_type pg_catalog.text default null
+) returns @extschema:[email protected]
+as $func$
+    select ai.vectorizer_embed
+    ( v.config operator(pg_catalog.->) 'embedding'
+    , input_text
+    , input_type
+    )
+    from ai.vectorizer v
+    where v.id operator(pg_catalog.=) vectorizer_id
+    ;
+$func$ language sql stable security invoker
+set search_path to pg_catalog, pg_temp
+;