.Net: Add PostgresVectorStore Memory connector. #9324

lossyrob · 2024-10-18T14:10:10Z

This PR adds a PostgresVectorStore and related classes to Microsoft.SemanticKernel.Connectors.Postgres.

Motivation and Context

As part of the move to having memory connectors implement the new Microsoft.Extensions.VectorData.IVectorStore architecture (see https://github.com/microsoft/semantic-kernel/blob/main/docs/decisions/0050-updated-vector-store-design.md), each memory connector needs to be updated with the new architecture. This PR tackles updating the existing Microsoft.SemanticKernel.Connectors.Postgres package to include this implementation. This will supercede the PostgresMemoryStore implementation.

Some high level comments about design:

PostgresVectorStore and PostgresVectorStoreRecordCollection get injected with an IPostgresVectorStoreDbClient. This abstracts the database communication and allows for unit tests to mock database interactions.
The PostgresVectorStoreDbClient gets passed in a NpgsqlDataSource from the user, which is used to manage connections to the database. The responsibility of connection pool lifecycle management is on the user.
The IPostgresVectorStoreDbClient is designed to accept and produce the storage model, which in this case is a Dictionary<string, object?> . This is the intermediate type that is mapped to by the IVectorStoreRecordMapper.
The PostgresVectorStoreDbClient also takes a IPostgresVectorStoreCollectionSqlBuilder, which generates SQL command information for interacting with the database. This abstracts the SQL queries related to each task, and allows for future expansion. This is particularly targeted at creating a AzureDBForPostgre vector store that will enable alternate vector implementations like DiskANN, while leveraging the same database client as the Postgres connector.
The integration tests for the vector store utilize Docker.Net to bring up a pgvector/pgvector docker container, which test are run against.

Contribution Checklist

The code builds clean without any errors or warnings
The PR follows the SK Contribution Guidelines and the pre-submission formatting script raises no violations
All unit tests pass, and I have added new tests where possible
I didn't break anyone 😄

Work in progress, some methods are not implemented yet.

…ctor-store-dotnet

dotnet/src/Connectors/Connectors.Memory.Postgres/PostgresConstants.cs

…ctor-store-dotnet

dotnet/src/Connectors/Connectors.Memory.Postgres/PostgresVectorStoreRecordCollection.cs

…ctor-store-dotnet

westey-m · 2024-10-30T18:20:52Z

...src/Connectors/Connectors.Postgres.UnitTests/PostgresVectorStoreCollectionSqlBuilderTests.cs

+                new VectorStoreRecordDataProperty("code", typeof(int)),
+                new VectorStoreRecordDataProperty("rating", typeof(float?)),
+                new VectorStoreRecordDataProperty("description", typeof(string)),
+                new VectorStoreRecordDataProperty("parking_is_included", typeof(bool)),


Might be worth having a record definition where some of the properties have storage names that are different to the main property name, to make sure these are used correctly in the command. Same with the other tests here that have to build names into the command.

westey-m · 2024-10-30T18:26:54Z

dotnet/src/Connectors/Connectors.Postgres.UnitTests/PostgresVectorStoreRecordMapperTests.cs

+    {
+        return new VectorStoreRecordDefinition
+        {
+            Properties = new List<VectorStoreRecordProperty>


Might be good to add one or two enumerables to the property list as well to verify that they word as expected.

westey-m · 2024-10-30T18:32:16Z

dotnet/src/IntegrationTests/Connectors/Memory/Postgres/PostgresVectorStoreFixture.cs

+        {
+            Properties = new List<VectorStoreRecordProperty>
+            {
+                new VectorStoreRecordKeyProperty("HotelId", typeof(ulong)),


Are we using this anywhere? From the constants list I don't believe we support ulong, so I would expect using this to fail.

westey-m · 2024-10-30T18:40:26Z

.../src/IntegrationTests/Connectors/Memory/Postgres/PostgresVectorStoreRecordCollectionTests.cs

+    [InlineData(typeof(long), 7L)]
+    [InlineData(typeof(string), "key1")]
+    [InlineData(typeof(Guid), null)]
+    public async Task ItCanGetAndDeleteRecordAsync(Type idType, object? key)


It is also possible to make test methods generic, e.g. public async Task ItCanGetAndDeleteRecordAsync<TKey>(Type idType, TKey? key), which means you can use TKey in the method and you don't have to use dynamic.

westey-m · 2024-10-30T18:46:42Z

.../src/IntegrationTests/Connectors/Memory/Postgres/PostgresVectorStoreRecordCollectionTests.cs

+namespace SemanticKernel.IntegrationTests.Connectors.Memory.Postgres;
+
+[Collection("PostgresVectorStoreCollection")]
+public sealed class PostgresVectorStoreRecordCollectionTests(PostgresVectorStoreFixture fixture)


It would be nice to have a test that reads some data that wasn't upserted by the Collection. E.g. if you create a table and upsert some records using SQL, and then read them using the Collection. The reason being that we want to ensure that the data in the DB is actually what we want it to be. E.g. if we do two complimentary things wrong, on upsert and read, so it looks fine when using the Collection, but the data isn't stored in the way we need it to in the DB, a test where we didn't write the data via the Collection should catch this.

…ctor-store-dotnet

lossyrob · 2024-11-01T16:00:03Z

@westey-m most recent comments addressed, thanks!

…ctor-store-dotnet

Hanake0 · 2024-11-26T10:43:36Z

Hi @westey-m and @lossyrob,

Thanks for the work on this feature.
Sorry for the inconvenience if that is not the right place to ask, but do you guys know when will it be available in the Microsoft.SemanticKernel.Connectors.Postgres package?

We are currently using the legacy Memory Store but that was recently marked as legacy and the connector for Postgres is currently marked as in-development, but as i see here it is probably at-least ready for alpha testing.

Is it available for testing already in any channel or is any ETA available for it?

Again, sorry if that is not the right place to ask.

Thanks!

westey-m · 2024-11-26T16:40:08Z

@Hanake0, no problem. It's been a bit busy lately with various deadlines, but I was meaning to get back to this again this week. With the start of the holiday period, giving an ETA will be difficult, but we'll certainly be progressing it as a priority.

dmytrostruk

A couple of small comments, but in general looks good to me! Thanks for your contribution!

dmytrostruk · 2024-11-26T21:38:49Z

dotnet/samples/Concepts/Memory/VectorStoreFixtures/VectorStorePostgresContainerFixture.cs

+            this._postgresContainerId = await VectorStoreInfra.SetupPostgresContainerAsync(this._dockerClient);
+
+            // Delay until the Postgres server is ready.
+            var connectionString = "Host=localhost;Port=5432;Username=postgres;Password=example;Database=postgres;";


I'm wondering if it would be better to get this connection string from .NET user secrets to avoid a pattern of keeping connection strings directly in a code, even if it's a test one. I think it's okay to keep it in the code if connection string is just a URL, but in this case, it also contains a password.

dmytrostruk · 2024-11-26T21:47:48Z

dotnet/samples/Concepts/Memory/VectorStore_VectorSearch_MultiStore_Postgres.cs

+    /// <summary>
+    /// The connection string to the Postgres database hosted in the docker container.
+    /// </summary>
+    private const string ConnectionString = "Host=localhost;Port=5432;Username=postgres;Password=example;Database=postgres;";


Same comment as above, we should not encourage users to keep the connection strings in a codebase, even if it's a test settings. We can do it in similar way how it's implemented in Azure AI Search, as an example:

semantic-kernel/dotnet/samples/Concepts/Memory/VectorStore_VectorSearch_MultiStore_AzureAISearch.cs

Lines 25 to 28 in 370c89a

/// To set your secrets use:

/// <para> dotnet user-secrets set "AzureAISearch:Endpoint" "https://... .search.windows.net"</para>

/// <para> dotnet user-secrets set "AzureAISearch:ApiKey" "{Key from your Search service resource}"</para>

/// </summary>

semantic-kernel/dotnet/samples/Concepts/Memory/VectorStore_VectorSearch_MultiStore_AzureAISearch.cs

Lines 45 to 47 in 370c89a

kernelBuilder.AddAzureAISearchVectorStore(

new Uri(TestConfiguration.AzureAISearch.Endpoint),

new AzureKeyCredential(TestConfiguration.AzureAISearch.ApiKey));

dmytrostruk · 2024-11-26T21:52:05Z

dotnet/src/Connectors/Connectors.Memory.Postgres/IPostgresDbClient.cs

+/// <remarks>
+/// This interface is used with the PostgresMemoryStore, which is being deprecated.
+/// Use the <see cref="IPostgresVectorStoreDbClient"/> interface with the PostgresVectorStore
+/// and related classes instead.
+/// </remarks>
 public interface IPostgresDbClient


IPostgresVectorStoreDbClient is internal, but this interface and documentation is public. Instead of this XML documentation we can mark this interface as Obsolete and recommend using new PostgresVectorStore.

dmytrostruk · 2024-11-26T22:01:55Z

dotnet/src/Connectors/Connectors.Memory.Postgres/PostgresDbClient.cs

@@ -15,6 +15,11 @@ namespace Microsoft.SemanticKernel.Connectors.Postgres;
 /// <summary>
 /// An implementation of a client for Postgres. This class is used to managing postgres database operations.
 /// </summary>
+/// <remarks>
+/// This class is used with the PostgresMemoryStore, which is being deprecated.
+/// Use the <see cref="PostgresVectorStoreDbClient"/> class with the PostgresVectorStore


Same comment as above, PostgresVectorStoreDbClient is internal. PostgresVectorStore and/or PostgresVectorStoreRecordCollection can be used as a new reference points.

dmytrostruk · 2024-11-26T22:02:19Z

dotnet/src/Connectors/Connectors.Memory.Postgres/PostgresGenericDataModelMapper.cs

+        this._propertyReader.VerifyDataProperties(PostgresConstants.SupportedDataTypes, PostgresConstants.SupportedEnumerableDataElementTypes);
+        this._propertyReader.VerifyVectorProperties(PostgresConstants.SupportedVectorTypes);
+    }
+    public Dictionary<string, object?> MapFromDataToStorageModel(VectorStoreGenericDataModel<TKey> dataModel)


nit:

Suggested change

public Dictionary<string, object?> MapFromDataToStorageModel(VectorStoreGenericDataModel<TKey> dataModel)

public Dictionary<string, object?> MapFromDataToStorageModel(VectorStoreGenericDataModel<TKey> dataModel)

dmytrostruk · 2024-11-26T22:22:02Z

dotnet/src/InternalUtilities/src/Linq/AsyncEnumerable.cs

+    /// function on each element of the original sequence.
+    /// </returns>
+    /// <exception cref="ArgumentNullException">Thrown when the source or selector is null.</exception>
+    public static async IAsyncEnumerable<TResult> Select<TSource, TResult>(


Is there a reason we can't name this method as SelectAsync?

Add PostgresVectorStore Memory connector.

8778d5f

Work in progress, some methods are not implemented yet.

markwallace-microsoft added .NET Issue or Pull requests regarding .NET code kernel Issues or pull requests impacting the core kernel memory labels Oct 18, 2024

lossyrob had a problem deploying to integration October 18, 2024 14:10 — with GitHub Actions Failure

Rob Emanuele added 3 commits October 18, 2024 18:00

Add UpsertBatch, GetBatch, and DeleteBatch

ddad99a

Remove unused CreateMapping

5447815

Merge remote-tracking branch 'upstream/main' into feature/postgres-ve…

7533f8c

…ctor-store-dotnet

lossyrob had a problem deploying to integration October 18, 2024 22:04 — with GitHub Actions Failure

Rob Emanuele added 2 commits October 21, 2024 10:27

Merge remote-tracking branch 'upstream/main' into feature/postgres-ve…

9a4f836

…ctor-store-dotnet

Add vector search to PostgresVectorStore

68a000e

lossyrob had a problem deploying to integration October 22, 2024 20:30 — with GitHub Actions Failure

JadynWong reviewed Oct 23, 2024

View reviewed changes

dotnet/src/Connectors/Connectors.Memory.Postgres/PostgresConstants.cs Outdated Show resolved Hide resolved

Rob Emanuele added 2 commits October 23, 2024 11:24

create index on collection creation

317f6af

Support Guid, test distance functions

f4f5ba2

lossyrob temporarily deployed to integration October 23, 2024 15:27 — with GitHub Actions Inactive

Rob Emanuele added 2 commits October 23, 2024 11:44

Format tests

2acf118

Merge remote-tracking branch 'upstream/main' into feature/postgres-ve…

5db2c59

…ctor-store-dotnet

lossyrob temporarily deployed to integration October 23, 2024 15:46 — with GitHub Actions Inactive

JadynWong reviewed Oct 23, 2024

View reviewed changes

dotnet/src/Connectors/Connectors.Memory.Postgres/PostgresVectorStoreRecordCollection.cs Outdated Show resolved Hide resolved

dotnet/src/Connectors/Connectors.Memory.Postgres/PostgresVectorStoreRecordCollection.cs Outdated Show resolved Hide resolved

markwallace-microsoft assigned westey-m Oct 24, 2024

Rob Emanuele added 5 commits October 24, 2024 11:11

Add service and kernel extensions

f4b4dc5

Default to Euclidean distance if no distance function is specified

5c58400

Add Postgres sample to concepts

8ea21cd

Add docs for setting configuration in samples\Concepts

4dcd222

Merge remote-tracking branch 'upstream/main' into feature/postgres-ve…

74b3764

…ctor-store-dotnet

markwallace-microsoft added the documentation label Oct 24, 2024

lossyrob temporarily deployed to integration October 24, 2024 15:16 — with GitHub Actions Inactive

Rob Emanuele added 2 commits October 24, 2024 11:27

Enforce dimension size in index creation

5c3e63f

Create index for CreateTableIfNotExistsAsyc

6d9f1fd

westey-m reviewed Oct 30, 2024

View reviewed changes

Rob Emanuele added 13 commits October 31, 2024 11:14

Merge remote-tracking branch 'upstream/main' into feature/postgres-ve…

b9b2487

…ctor-store-dotnet

Remove unnecessary overloads

24577a0

Change tests to be true to name

60d6512

Remove reduntant key type based test

5a66a13

Remove unnecessary overloads

581b6ab

Better error handling for IAsyncEnumerable

494a0d4

Default to Flat (no index) instead of Hnsw

5f19889

Add enumerable to record mapper test

62ac8eb

Remove unused fixture properties

364b592

Test StoragePropertyName in sql builder tests

bf58cab

Remove dynamic from integration test

aa592de

Add test to read from manually inserted record

9a3b216

Merge remote-tracking branch 'upstream/main' into feature/postgres-ve…

1ee09c1

…ctor-store-dotnet

lossyrob had a problem deploying to integration November 1, 2024 16:01 — with GitHub Actions Failure

Formatting, spelling

b037075

lossyrob had a problem deploying to integration November 1, 2024 16:59 — with GitHub Actions Failure

Merge remote-tracking branch 'upstream/main' into feature/postgres-ve…

29d91ba

…ctor-store-dotnet

lossyrob had a problem deploying to integration November 1, 2024 19:07 — with GitHub Actions Failure

Fix test.

c2937e0

lossyrob temporarily deployed to integration November 1, 2024 21:05 — with GitHub Actions Inactive

westey-m approved these changes Nov 26, 2024

View reviewed changes

dmytrostruk approved these changes Nov 26, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

.Net: Add PostgresVectorStore Memory connector. #9324

.Net: Add PostgresVectorStore Memory connector. #9324

lossyrob commented Oct 18, 2024 •

edited

Loading

westey-m Oct 30, 2024 •

edited

Loading

westey-m Oct 30, 2024 •

edited

Loading

westey-m Oct 30, 2024

westey-m Oct 30, 2024

westey-m Oct 30, 2024

lossyrob commented Nov 1, 2024

Hanake0 commented Nov 26, 2024

westey-m commented Nov 26, 2024 •

edited

Loading

dmytrostruk left a comment

dmytrostruk Nov 26, 2024

dmytrostruk Nov 26, 2024

dmytrostruk Nov 26, 2024

dmytrostruk Nov 26, 2024

dmytrostruk Nov 26, 2024

dmytrostruk Nov 26, 2024

	/// To set your secrets use:
	/// <para> dotnet user-secrets set "AzureAISearch:Endpoint" "https://... .search.windows.net"</para>
	/// <para> dotnet user-secrets set "AzureAISearch:ApiKey" "{Key from your Search service resource}"</para>
	/// </summary>

	kernelBuilder.AddAzureAISearchVectorStore(
	new Uri(TestConfiguration.AzureAISearch.Endpoint),
	new AzureKeyCredential(TestConfiguration.AzureAISearch.ApiKey));

	public Dictionary<string, object?> MapFromDataToStorageModel(VectorStoreGenericDataModel<TKey> dataModel)

	public Dictionary<string, object?> MapFromDataToStorageModel(VectorStoreGenericDataModel<TKey> dataModel)

.Net: Add PostgresVectorStore Memory connector. #9324

Are you sure you want to change the base?

.Net: Add PostgresVectorStore Memory connector. #9324

Conversation

lossyrob commented Oct 18, 2024 • edited Loading

Motivation and Context

Contribution Checklist

westey-m Oct 30, 2024 • edited Loading

Choose a reason for hiding this comment

westey-m Oct 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lossyrob commented Nov 1, 2024

Hanake0 commented Nov 26, 2024

westey-m commented Nov 26, 2024 • edited Loading

dmytrostruk left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lossyrob commented Oct 18, 2024 •

edited

Loading

westey-m Oct 30, 2024 •

edited

Loading

westey-m Oct 30, 2024 •

edited

Loading

westey-m commented Nov 26, 2024 •

edited

Loading