Pinecone: Namespace and field remapping support #31373

flash1293 · 2023-10-13T08:03:34Z

What

How

This PR updates the CDK which adds field mapping transformations

and also adds namespace support (fixing the existing bug with vectors not being cleaned out correctly if a namespace is set).

Namespaces are now regularly supported - the configuration made on the connection level will be respected in Pinecone, defaulting to no namespace set.

…ithub.com:airbytehq/airbyte into flash1293/allow-openai-compatible-embedding-modes

…nai-compatible-embedding-modes

…ate limit

…id GHA rate limit" This reverts commit da17199.

This reverts commit 5f68d7c.

…nai-compatible-embedding-modes

vercel · 2023-10-13T08:03:40Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 26, 2023 1:45pm

github-actions · 2023-10-13T08:04:02Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

…ithub.com:airbytehq/airbyte into flash1293/allow-openai-compatible-embedding-modes

… flash1293/pinecone-namespace-support-2

airbyte-oss-build-runner · 2023-10-19T14:02:47Z

destination-pinecone test report (commit `58b34b92c4`) - ❌

⏲️ Total pipeline duration: 20.73s

Step	Result
Build destination-pinecone docker image for platform(s) linux/x86_64	❌
Code format checks	❌
Validate metadata for destination-pinecone	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-pinecone test

airbyte-oss-build-runner · 2023-10-19T14:37:11Z

destination-pinecone test report (commit `2b0da74820`) - ❌

⏲️ Total pipeline duration: 18.09s

Step	Result
Build destination-pinecone docker image for platform(s) linux/x86_64	❌
Code format checks	✅
Validate metadata for destination-pinecone	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-pinecone test

…namespace-support-2

airbyte-oss-build-runner · 2023-10-20T09:05:15Z

destination-pinecone test report (commit `a3430985a9`) - ❌

⏲️ Total pipeline duration: 02mn26s

Step	Result
Build destination-pinecone docker image for platform(s) linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	❌
Code format checks	✅
Validate metadata for destination-pinecone	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-pinecone test

airbyte-oss-build-runner · 2023-10-20T09:12:37Z

destination-pinecone test report (commit `4a70c1a950`) - ❌

⏲️ Total pipeline duration: 01mn43s

Step	Result
Build destination-pinecone docker image for platform(s) linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	❌
Code format checks	✅
Validate metadata for destination-pinecone	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-pinecone test

airbyte-oss-build-runner · 2023-10-20T09:20:15Z

destination-pinecone test report (commit `f75b0471e0`) - ✅

⏲️ Total pipeline duration: 02mn14s

Step	Result
Build destination-pinecone docker image for platform(s) linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for destination-pinecone	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-pinecone test

airbyte-oss-build-runner · 2023-10-20T09:46:32Z

source-paypal-transaction test report (commit `498abbffb0`) - ✅

⏲️ Total pipeline duration: 01mn12s

Step	Result
Build source-paypal-transaction docker image for platform(s) linux/x86_64	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for source-paypal-transaction	✅
Connector version semver check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-paypal-transaction test

airbyte-oss-build-runner · 2023-10-20T09:46:43Z

destination-pinecone test report (commit `498abbffb0`) - ✅

⏲️ Total pipeline duration: 01mn24s

Step	Result
Build destination-pinecone docker image for platform(s) linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for destination-pinecone	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-pinecone test

airbyte-oss-build-runner · 2023-10-20T10:27:21Z

source-paypal-transaction test report (commit `ee14f88841`) - ✅

⏲️ Total pipeline duration: 01mn15s

Step	Result
Build source-paypal-transaction docker image for platform(s) linux/x86_64	✅
Acceptance tests	✅
Check our base image is used	✅
Code format checks	✅
Validate metadata for source-paypal-transaction	✅
Connector version semver check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-paypal-transaction test

airbyte-oss-build-runner · 2023-10-20T10:27:43Z

destination-pinecone test report (commit `ee14f88841`) - ✅

⏲️ Total pipeline duration: 01mn37s

Step	Result
Build destination-pinecone docker image for platform(s) linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Check our base image is used	✅
Code format checks	✅
Validate metadata for destination-pinecone	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-pinecone test

aaronsteers · 2023-10-23T20:45:21Z

airbyte-integrations/connectors/destination-pinecone/unit_tests/pinecone_indexer_test.py

@@ -175,7 +184,7 @@ def generate_catalog():
 def test_pinecone_pre_sync(mock_describe_index):
    indexer = create_pinecone_indexer()
    indexer.pre_sync(generate_catalog())
-    indexer.pinecone_index.delete.assert_called_with(filter={"_ab_stream": "example_stream2"})
+    indexer.pinecone_index.delete.assert_called_with(filter={"_ab_stream": "ns2_example_stream2"}, namespace="ns2")


Just to double check - do you think think we should list this as a breaking change?

Namely, does the namespace come through from the source by default, and would this change the destination index name? Or is the namespace purely opt-in, so we can assume that wherever it was not handled previously, this is 'fixing' the prior omitted prefix?

It's not a breaking change because the namespace is opt-in (can be configured on the connection level) - it changes here because I added a namespace to the test.

aaronsteers

Looks good. 👍

One question added inline.

aaronsteers · 2023-10-24T00:57:49Z

@flash1293 - Just to confirm one more thing. This PR is labeled as adding field remapping support but the diff only shows the addition of spec fields in the integration tests folder. The diff doesn't surface any other code for the field remapping.

Do I understand correctly the field rename/remapping feature landed in a prior PR, or is there another reason I'm not seeing those changes?

flash1293 · 2023-10-24T16:03:08Z

Do I understand correctly the field rename/remapping feature landed in a prior PR, or is there another reason I'm not seeing those changes

The changes are encapsulated in the CDK, so they get added via the update https://github.com/airbytehq/airbyte/pull/31373/files#diff-637a815d4e2b5d268bd8b9307fd775a0be36aff5a3b4f806bec36b0d672d694aL9 (you are seeing the spec changes because there is an integration test that validates the spec emitted from the connector)

…namespace-support-2

airbyte-oss-build-runner · 2023-10-26T13:45:10Z

destination-pinecone test report (commit `412216d146`) - ✅

⏲️ Total pipeline duration: 02mn20s

Step	Result
Build destination-pinecone docker image for platform(s) linux/amd64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Check our base image is used	✅
Code format checks	✅
Validate metadata for destination-pinecone	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-pinecone test

Joe Reuter and others added 13 commits October 12, 2023 10:17

add OpenAI-compatible embedding

cb7a5a8

add changelog

9c32526

Automated Commit - Formatting Changes

eda2bb4

adjust indexers

e2bbdd5

Merge branch 'flash1293/allow-openai-compatible-embedding-modes' of g…

9843ccb

…ithub.com:airbytehq/airbyte into flash1293/allow-openai-compatible-embedding-modes

Merge remote-tracking branch 'origin/master' into flash1293/allow-ope…

68abfb9

…nai-compatible-embedding-modes

use GITHUB_TOKEN instead of GH_PAT_MAINTENANCE_OCTAVIA to avoid GHA r…

da17199

…ate limit

Revert "use GITHUB_TOKEN instead of GH_PAT_MAINTENANCE_OCTAVIA to avo…

024561e

…id GHA rate limit" This reverts commit da17199.

remove repo-token on labeler action to fallback to default

5f68d7c

Revert "remove repo-token on labeler action to fallback to default"

844b89d

This reverts commit 5f68d7c.

Merge remote-tracking branch 'origin/master' into flash1293/allow-ope…

264308e

…nai-compatible-embedding-modes

bump cdk version

e3569c1

add support for namespaces

f1d2eeb

Trigger CI

ec1bc69

vercel bot deployed to Preview October 13, 2023 08:05 View deployment

flash1293 and others added 5 commits October 13, 2023 08:17

Automated Commit - Formatting Changes

ef86321

fix tests

3693b27

Merge branch 'flash1293/allow-openai-compatible-embedding-modes' of g…

0d5ed69

…ithub.com:airbytehq/airbyte into flash1293/allow-openai-compatible-embedding-modes

Merge branch 'flash1293/allow-openai-compatible-embedding-modes' into…

339d55d

… flash1293/pinecone-namespace-support-2

fix tests

ab92cb6

flash1293 mentioned this pull request Oct 13, 2023

Pinecone destination: Add namespace setting #30789

Closed

fix namespace clearing error

58b34b9

vercel bot deployed to Preview October 19, 2023 14:02 View deployment

Automated Commit - Formatting Changes

2b0da74

Joe Reuter added 2 commits October 20, 2023 10:09

Merge remote-tracking branch 'origin/master' into flash1293/pinecone-…

014c442

…namespace-support-2

fix tests

a343098

flash1293 changed the title ~~Pinecone: Namespace support~~ Pinecone: Namespace and field remapping support Oct 20, 2023

remove dockerfile again

4a70c1a

fix tests

f75b047

Automated Commit - Formatting Changes

498abbf

octavia-squidington-iii added the connectors/source/paypal-transaction label Oct 20, 2023

Update destination.py

ee14f88

aaronsteers reviewed Oct 23, 2023

View reviewed changes

aaronsteers approved these changes Oct 23, 2023

View reviewed changes

Merge remote-tracking branch 'origin/master' into flash1293/pinecone-…

412216d

…namespace-support-2

flash1293 enabled auto-merge (squash) October 26, 2023 13:41

vercel bot deployed to Preview October 26, 2023 13:45 View deployment

flash1293 merged commit bee749b into master Oct 26, 2023
16 checks passed

flash1293 deleted the flash1293/pinecone-namespace-support-2 branch October 26, 2023 14:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pinecone: Namespace and field remapping support #31373

Pinecone: Namespace and field remapping support #31373

flash1293 commented Oct 13, 2023 •

edited

Loading

vercel bot commented Oct 13, 2023 •

edited

Loading

github-actions bot commented Oct 13, 2023 •

edited by flash1293

Loading

airbyte-oss-build-runner commented Oct 19, 2023

airbyte-oss-build-runner commented Oct 19, 2023

airbyte-oss-build-runner commented Oct 20, 2023

airbyte-oss-build-runner commented Oct 20, 2023

airbyte-oss-build-runner commented Oct 20, 2023

airbyte-oss-build-runner commented Oct 20, 2023

airbyte-oss-build-runner commented Oct 20, 2023

airbyte-oss-build-runner commented Oct 20, 2023

airbyte-oss-build-runner commented Oct 20, 2023

aaronsteers Oct 23, 2023 •

edited

Loading

flash1293 Oct 26, 2023

aaronsteers left a comment

aaronsteers commented Oct 24, 2023 •

edited

Loading

flash1293 commented Oct 24, 2023

airbyte-oss-build-runner commented Oct 26, 2023

Pinecone: Namespace and field remapping support #31373

Pinecone: Namespace and field remapping support #31373

Conversation

flash1293 commented Oct 13, 2023 • edited Loading

What

How

vercel bot commented Oct 13, 2023 • edited Loading

github-actions bot commented Oct 13, 2023 • edited by flash1293 Loading

Before Merging a Connector Pull Request

airbyte-oss-build-runner commented Oct 19, 2023

destination-pinecone test report (commit 58b34b92c4) - ❌

airbyte-oss-build-runner commented Oct 19, 2023

destination-pinecone test report (commit 2b0da74820) - ❌

airbyte-oss-build-runner commented Oct 20, 2023

destination-pinecone test report (commit a3430985a9) - ❌

airbyte-oss-build-runner commented Oct 20, 2023

destination-pinecone test report (commit 4a70c1a950) - ❌

airbyte-oss-build-runner commented Oct 20, 2023

destination-pinecone test report (commit f75b0471e0) - ✅

airbyte-oss-build-runner commented Oct 20, 2023

source-paypal-transaction test report (commit 498abbffb0) - ✅

airbyte-oss-build-runner commented Oct 20, 2023

destination-pinecone test report (commit 498abbffb0) - ✅

airbyte-oss-build-runner commented Oct 20, 2023

source-paypal-transaction test report (commit ee14f88841) - ✅

airbyte-oss-build-runner commented Oct 20, 2023

destination-pinecone test report (commit ee14f88841) - ✅

aaronsteers Oct 23, 2023 • edited Loading

Choose a reason for hiding this comment

flash1293 Oct 26, 2023

Choose a reason for hiding this comment

aaronsteers left a comment

Choose a reason for hiding this comment

aaronsteers commented Oct 24, 2023 • edited Loading

flash1293 commented Oct 24, 2023

airbyte-oss-build-runner commented Oct 26, 2023

destination-pinecone test report (commit 412216d146) - ✅

flash1293 commented Oct 13, 2023 •

edited

Loading

vercel bot commented Oct 13, 2023 •

edited

Loading

github-actions bot commented Oct 13, 2023 •

edited by flash1293

Loading

destination-pinecone test report (commit `58b34b92c4`) - ❌

destination-pinecone test report (commit `2b0da74820`) - ❌

destination-pinecone test report (commit `a3430985a9`) - ❌

destination-pinecone test report (commit `4a70c1a950`) - ❌

destination-pinecone test report (commit `f75b0471e0`) - ✅

source-paypal-transaction test report (commit `498abbffb0`) - ✅

destination-pinecone test report (commit `498abbffb0`) - ✅

source-paypal-transaction test report (commit `ee14f88841`) - ✅

destination-pinecone test report (commit `ee14f88841`) - ✅

aaronsteers Oct 23, 2023 •

edited

Loading

aaronsteers commented Oct 24, 2023 •

edited

Loading

destination-pinecone test report (commit `412216d146`) - ✅