Destinations snowflake+bigquery: Fix string-json auto conversion for BigQuery #31358

gisripa · 2023-10-12T20:11:25Z

Summary

Add BaseSqlGeneratorIntegrationTest#jsonStringifyTypes to verify BigQuery and Snowflake behavior when it encounters json value for a string type specified in schema.
Fix BigQuery SQL to allow inserting non string types as a stringified value in the column instead of SQL null
Minor nit to clean up if-else branching to just if-return and default return in BigQuerySqlGenerator.java

This addresses #29524

vercel · 2023-10-12T20:11:31Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Oct 12, 2023 11:16pm

CLAassistant · 2023-10-12T20:11:34Z

All committers have signed the CLA.

github-actions · 2023-10-12T20:11:48Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

airbyte-oss-build-runner · 2023-10-12T20:21:26Z

destination-bigquery test report (commit `4546fdeee4`) - ❌

⏲️ Total pipeline duration: 07mn59s

Step	Result
Build connector tar	✅
Build destination-bigquery docker image for platform(s) linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Validate metadata for destination-bigquery	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

edgao

nice! Super clean test + SQL implementation, and this is definitely solving a real problem for our users. alex C is going to be happy :)

(I had two small comments but overall )

All that's left is to go through the publish process 🚀 (let's chat about this later)

.../java/io/airbyte/integrations/destination/bigquery/typing_deduping/BigQuerySqlGenerator.java

airbyte-oss-build-runner · 2023-10-12T22:18:07Z

destination-bigquery test report (commit `85c74c8ce1`) - ❌

⏲️ Total pipeline duration: 08mn04s

Step	Result
Build connector tar	✅
Build destination-bigquery docker image for platform(s) linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Validate metadata for destination-bigquery	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

airbyte-oss-build-runner · 2023-10-12T22:27:06Z

destination-snowflake test report (commit `85c74c8ce1`) - ✅

⏲️ Total pipeline duration: 08mn47s

Step	Result
Build connector tar	✅
Build destination-snowflake docker image for platform(s) linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Validate metadata for destination-snowflake	✅
Connector version semver check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

airbyte-oss-build-runner · 2023-10-12T22:39:18Z

destination-snowflake test report (commit `47a162520d`) - ✅

⏲️ Total pipeline duration: 10mn06s

Step	Result
Build connector tar	✅
Build destination-snowflake docker image for platform(s) linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Validate metadata for destination-snowflake	✅
Connector version semver check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

Summary ======= Add BaseSqlGeneratorIntegrationTest#jsonStringifyTypes to verify BigQuery and Snowflake behavior when it encounters json value for a string type specified in schema. Fix BigQuery SQL to allow inserting non string types as a stringified value in the column instead of SQL null Signed-off-by: Gireesh Sreepathi <[email protected]>

…java/io/airbyte/integrations/destination/bigquery/typing_deduping/BigQuerySqlGenerator.java Co-authored-by: Edward Gao <[email protected]>

airbyte-oss-build-runner · 2023-10-12T22:46:26Z

destination-bigquery test report (commit `47a162520d`) - ❌

⏲️ Total pipeline duration: 06mn50s

Step	Result
Build connector tar	✅
Build destination-bigquery docker image for platform(s) linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Validate metadata for destination-bigquery	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

airbyte-oss-build-runner · 2023-10-12T22:55:24Z

destination-snowflake test report (commit `462e092987`) - ✅

⏲️ Total pipeline duration: 04mn10s

Step	Result
Build connector tar	✅
Build destination-snowflake docker image for platform(s) linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Validate metadata for destination-snowflake	✅
Connector version semver check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

airbyte-oss-build-runner · 2023-10-12T23:02:38Z

destination-bigquery test report (commit `462e092987`) - ✅

⏲️ Total pipeline duration: 07mn02s

Step	Result
Build connector tar	✅
Build destination-bigquery docker image for platform(s) linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Validate metadata for destination-bigquery	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

airbyte-oss-build-runner · 2023-10-12T23:22:55Z

destination-snowflake test report (commit `540def4d9a`) - ✅

⏲️ Total pipeline duration: 04mn15s

Step	Result
Build connector tar	✅
Build destination-snowflake docker image for platform(s) linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Validate metadata for destination-snowflake	✅
Connector version semver check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

airbyte-oss-build-runner · 2023-10-12T23:31:03Z

destination-bigquery test report (commit `540def4d9a`) - ✅

⏲️ Total pipeline duration: 07mn56s

Step	Result
Build connector tar	✅
Build destination-bigquery docker image for platform(s) linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Validate metadata for destination-bigquery	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

…irbytehq#31358) Signed-off-by: Gireesh Sreepathi <[email protected]> Co-authored-by: gisripa <[email protected]> Co-authored-by: Edward Gao <[email protected]>

gisripa requested a review from a team as a code owner October 12, 2023 20:11

octavia-squidington-iii added area/connectors Connector related issues connectors/destination/bigquery connectors/destination/snowflake labels Oct 12, 2023

gisripa changed the title ~~Add tests for string-json auto conversion~~ Destinations snowflake+bigquery: Add tests for string-json auto conversion Oct 12, 2023

gisripa changed the title ~~Destinations snowflake+bigquery: Add tests for string-json auto conversion~~ Destinations snowflake+bigquery: Fix string-json auto conversion for BigQuery Oct 12, 2023

edgao approved these changes Oct 12, 2023

View reviewed changes

.../java/io/airbyte/integrations/destination/bigquery/typing_deduping/BigQuerySqlGenerator.java Show resolved Hide resolved

.../java/io/airbyte/integrations/destination/bigquery/typing_deduping/BigQuerySqlGenerator.java Show resolved Hide resolved

vercel bot deployed to Preview October 12, 2023 22:37 View deployment

gisripa and others added 4 commits October 12, 2023 15:40

Automated Commit - Formatting Changes

aaf4234

Update airbyte-integrations/connectors/destination-bigquery/src/main/…

6ed69a2

…java/io/airbyte/integrations/destination/bigquery/typing_deduping/BigQuerySqlGenerator.java Co-authored-by: Edward Gao <[email protected]>

Minor comment update

abfe2e7

gisripa force-pushed the gireesh/29524 branch from 1067a59 to abfe2e7 Compare October 12, 2023 22:46

Version upgrade and changelog

462e092

vercel bot deployed to Preview October 12, 2023 22:51 View deployment

Automated Commit - Formatting Changes

540def4

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Oct 12, 2023

gisripa merged commit c4ff29e into master Oct 13, 2023
16 checks passed

gisripa deleted the gireesh/29524 branch October 13, 2023 00:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Destinations snowflake+bigquery: Fix string-json auto conversion for BigQuery #31358

Destinations snowflake+bigquery: Fix string-json auto conversion for BigQuery #31358

gisripa commented Oct 12, 2023 •

edited

Loading

vercel bot commented Oct 12, 2023 •

edited

Loading

CLAassistant commented Oct 12, 2023 •

edited

Loading

github-actions bot commented Oct 12, 2023 •

edited by gisripa

Loading

airbyte-oss-build-runner commented Oct 12, 2023

edgao left a comment

airbyte-oss-build-runner commented Oct 12, 2023

airbyte-oss-build-runner commented Oct 12, 2023

airbyte-oss-build-runner commented Oct 12, 2023

airbyte-oss-build-runner commented Oct 12, 2023

airbyte-oss-build-runner commented Oct 12, 2023

airbyte-oss-build-runner commented Oct 12, 2023

airbyte-oss-build-runner commented Oct 12, 2023

airbyte-oss-build-runner commented Oct 12, 2023

Destinations snowflake+bigquery: Fix string-json auto conversion for BigQuery #31358

Destinations snowflake+bigquery: Fix string-json auto conversion for BigQuery #31358

Conversation

gisripa commented Oct 12, 2023 • edited Loading

Summary

vercel bot commented Oct 12, 2023 • edited Loading

CLAassistant commented Oct 12, 2023 • edited Loading

github-actions bot commented Oct 12, 2023 • edited by gisripa Loading

Before Merging a Connector Pull Request

airbyte-oss-build-runner commented Oct 12, 2023

destination-bigquery test report (commit 4546fdeee4) - ❌

edgao left a comment

Choose a reason for hiding this comment

airbyte-oss-build-runner commented Oct 12, 2023

destination-bigquery test report (commit 85c74c8ce1) - ❌

airbyte-oss-build-runner commented Oct 12, 2023

destination-snowflake test report (commit 85c74c8ce1) - ✅

airbyte-oss-build-runner commented Oct 12, 2023

destination-snowflake test report (commit 47a162520d) - ✅

airbyte-oss-build-runner commented Oct 12, 2023

destination-bigquery test report (commit 47a162520d) - ❌

airbyte-oss-build-runner commented Oct 12, 2023

destination-snowflake test report (commit 462e092987) - ✅

airbyte-oss-build-runner commented Oct 12, 2023

destination-bigquery test report (commit 462e092987) - ✅

airbyte-oss-build-runner commented Oct 12, 2023

destination-snowflake test report (commit 540def4d9a) - ✅

airbyte-oss-build-runner commented Oct 12, 2023

destination-bigquery test report (commit 540def4d9a) - ✅

gisripa commented Oct 12, 2023 •

edited

Loading

vercel bot commented Oct 12, 2023 •

edited

Loading

CLAassistant commented Oct 12, 2023 •

edited

Loading

github-actions bot commented Oct 12, 2023 •

edited by gisripa

Loading

destination-bigquery test report (commit `4546fdeee4`) - ❌

destination-bigquery test report (commit `85c74c8ce1`) - ❌

destination-snowflake test report (commit `85c74c8ce1`) - ✅

destination-snowflake test report (commit `47a162520d`) - ✅

destination-bigquery test report (commit `47a162520d`) - ❌

destination-snowflake test report (commit `462e092987`) - ✅

destination-bigquery test report (commit `462e092987`) - ✅

destination-snowflake test report (commit `540def4d9a`) - ✅

destination-bigquery test report (commit `540def4d9a`) - ✅