[Source-Postgres] Update PostgresConverter to correctly stringify BigDecimals without scientific notation #31309

nguyenaiden · 2023-10-11T20:09:37Z

Description

Update how the PostgresConverter's registerNumber method to use toPlainString() instead of toString() to stringify decimals in standard form and not scientific notation.
Added a test to address #28303.

vercel · 2023-10-11T20:09:43Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Oct 13, 2023 4:30pm

github-actions · 2023-10-11T20:10:00Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

github-actions · 2023-10-11T20:34:36Z

Coverage report for source-postgres

There is no coverage information present for the Files changed

Total Project Coverage	71.71%	🍏

airbyte-oss-build-runner · 2023-10-12T18:17:14Z

source-postgres test report (commit `3614089f2c`) - ❌

⏲️ Total pipeline duration: 29mn38s

Step	Result
Build connector tar	✅
Build source-postgres docker image for platform(s) linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	❌
Acceptance tests	✅
Validate metadata for source-postgres	✅
Connector version semver check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

nguyenaiden · 2023-10-12T18:49:13Z

/publish-java-cdk dry-run=true

airbyte-oss-build-runner · 2023-10-12T18:52:48Z

source-postgres test report (commit `7b656c7ce9`) - ❌

⏲️ Total pipeline duration: 02mn14s

Step	Result
Build connector tar	❌
Validate metadata for source-postgres	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

airbyte-oss-build-runner · 2023-10-12T18:53:54Z

source-postgres-strict-encrypt test report (commit `7b656c7ce9`) - ❌

⏲️ Total pipeline duration: 55.75s

Step	Result
Build connector tar	❌
Validate metadata for source-postgres-strict-encrypt	✅
Connector version semver check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres-strict-encrypt test

nguyenaiden · 2023-10-12T19:47:22Z

/publish-java-cdk dry-run=true

nguyenaiden · 2023-10-12T20:40:07Z

/publish-java-cdk

airbyte-oss-build-runner · 2023-10-12T21:06:56Z

source-postgres-strict-encrypt test report (commit `c66f19def5`) - ❌

⏲️ Total pipeline duration: 01mn34s

Step	Result
Build connector tar	❌
Validate metadata for source-postgres-strict-encrypt	✅
Connector version semver check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres-strict-encrypt test

airbyte-oss-build-runner · 2023-10-12T21:08:46Z

source-postgres test report (commit `c66f19def5`) - ❌

⏲️ Total pipeline duration: 01mn38s

Step	Result
Build connector tar	❌
Validate metadata for source-postgres	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

airbyte-oss-build-runner · 2023-10-12T22:21:27Z

source-postgres test report (commit `843f5b2cb6`) - ❌

⏲️ Total pipeline duration: 02mn18s

Step	Result
Build connector tar	❌
Validate metadata for source-postgres	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

airbyte-oss-build-runner · 2023-10-12T22:22:42Z

source-postgres-strict-encrypt test report (commit `843f5b2cb6`) - ❌

⏲️ Total pipeline duration: 01mn04s

Step	Result
Build connector tar	❌
Validate metadata for source-postgres-strict-encrypt	✅
Connector version semver check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres-strict-encrypt test

nguyenaiden · 2023-10-12T22:40:49Z

/publish-java-cdk

airbyte-oss-build-runner · 2023-10-12T23:40:19Z

source-postgres test report (commit `843f5b2cb6`) - ✅

⏲️ Total pipeline duration: 29mn05s

Step	Result
Build connector tar	✅
Build source-postgres docker image for platform(s) linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate metadata for source-postgres	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

airbyte-oss-build-runner · 2023-10-12T23:48:35Z

source-postgres-strict-encrypt test report (commit `843f5b2cb6`) - ✅

⏲️ Total pipeline duration: 08mn05s

Step	Result
Build connector tar	✅
Build source-postgres-strict-encrypt docker image for platform(s) linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate metadata for source-postgres-strict-encrypt	✅
Connector version semver check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres-strict-encrypt test

airbyte-oss-build-runner · 2023-10-13T00:21:52Z

source-postgres-strict-encrypt test report (commit `e558bea7e7`) - ✅

⏲️ Total pipeline duration: 11mn20s

Step	Result
Build connector tar	✅
Build source-postgres-strict-encrypt docker image for platform(s) linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate metadata for source-postgres-strict-encrypt	✅
Connector version semver check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres-strict-encrypt test

airbyte-oss-build-runner · 2023-10-13T00:51:01Z

source-postgres test report (commit `e558bea7e7`) - ✅

⏲️ Total pipeline duration: 28mn57s

Step	Result
Build connector tar	✅
Build source-postgres docker image for platform(s) linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate metadata for source-postgres	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

airbyte-oss-build-runner · 2023-10-13T17:02:40Z

source-postgres test report (commit `d94788be03`) - ✅

⏲️ Total pipeline duration: 29mn58s

Step	Result
Build connector tar	✅
Build source-postgres docker image for platform(s) linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate metadata for source-postgres	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

airbyte-oss-build-runner · 2023-10-13T17:11:06Z

source-postgres-strict-encrypt test report (commit `d94788be03`) - ✅

⏲️ Total pipeline duration: 08mn14s

Step	Result
Build connector tar	✅
Build source-postgres-strict-encrypt docker image for platform(s) linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate metadata for source-postgres-strict-encrypt	✅
Connector version semver check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres-strict-encrypt test

…Decimals without scientific notation (airbytehq#31309)

octavia-squidington-iii added area/connectors Connector related issues CDK Connector Development Kit connectors/source/postgres labels Oct 11, 2023

nguyenaiden force-pushed the cdc-postgres-bad-int branch from f594d47 to 7bea3e8 Compare October 11, 2023 22:07

nguyenaiden changed the title ~~[Source-Postgres] Test to verify integers are not deserialized as scientific notation~~ [Source-Postgres] Update PostgresConverter to correctly stringify BigDecimals without scientific notation Oct 11, 2023

rodireich approved these changes Oct 12, 2023

View reviewed changes

nguyenaiden marked this pull request as ready for review October 12, 2023 17:33

nguyenaiden requested a review from a team as a code owner October 12, 2023 17:33

vercel bot deployed to Preview October 12, 2023 17:34 View deployment

vercel bot deployed to Preview October 12, 2023 17:46 View deployment

nguyenaiden force-pushed the cdc-postgres-bad-int branch from 3614089 to 7b656c7 Compare October 12, 2023 18:48

vercel bot deployed to Preview October 12, 2023 18:50 View deployment

octavia-squidington-iii added the connectors/source/postgres-strict-encrypt label Oct 12, 2023

nguyenaiden force-pushed the cdc-postgres-bad-int branch from 7b656c7 to c66f19d Compare October 12, 2023 21:03

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Oct 12, 2023

vercel bot deployed to Preview October 12, 2023 21:04 View deployment

nguyenaiden marked this pull request as draft October 12, 2023 23:09

nguyenaiden marked this pull request as ready for review October 12, 2023 23:09

nguyenaiden force-pushed the cdc-postgres-bad-int branch from 843f5b2 to e558bea Compare October 13, 2023 00:08

nguyenaiden added 7 commits October 13, 2023 09:30

WIP

dd839bb

Add test to verify the scientific notation bug

e188f26

Reverted extraneous changes back to before

ee2c1bb

convert bigdecimals using toPlainString() to avoid scientific notation

3165a95

formatting

4507c57

Bumped CDK version + source postgres versions

0390aec

Use parseDouble

d94788b

nguyenaiden force-pushed the cdc-postgres-bad-int branch from e558bea to d94788b Compare October 13, 2023 16:30

nguyenaiden merged commit 896fa4b into master Oct 13, 2023
15 of 17 checks passed

nguyenaiden deleted the cdc-postgres-bad-int branch October 13, 2023 17:11

ariesgun pushed a commit to ariesgun/airbyte that referenced this pull request Oct 23, 2023

[Source-Postgres] Update PostgresConverter to correctly stringify Big…

82ff801

…Decimals without scientific notation (airbytehq#31309)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Source-Postgres] Update PostgresConverter to correctly stringify BigDecimals without scientific notation #31309

[Source-Postgres] Update PostgresConverter to correctly stringify BigDecimals without scientific notation #31309

nguyenaiden commented Oct 11, 2023 •

edited

Loading

vercel bot commented Oct 11, 2023 •

edited

Loading

github-actions bot commented Oct 11, 2023 •

edited by nguyenaiden

Loading

github-actions bot commented Oct 11, 2023 •

edited

Loading

airbyte-oss-build-runner commented Oct 12, 2023

nguyenaiden commented Oct 12, 2023

airbyte-oss-build-runner commented Oct 12, 2023

airbyte-oss-build-runner commented Oct 12, 2023

nguyenaiden commented Oct 12, 2023

nguyenaiden commented Oct 12, 2023

airbyte-oss-build-runner commented Oct 12, 2023

airbyte-oss-build-runner commented Oct 12, 2023

airbyte-oss-build-runner commented Oct 12, 2023

airbyte-oss-build-runner commented Oct 12, 2023

nguyenaiden commented Oct 12, 2023

airbyte-oss-build-runner commented Oct 12, 2023

airbyte-oss-build-runner commented Oct 12, 2023

airbyte-oss-build-runner commented Oct 13, 2023

airbyte-oss-build-runner commented Oct 13, 2023

airbyte-oss-build-runner commented Oct 13, 2023

airbyte-oss-build-runner commented Oct 13, 2023

[Source-Postgres] Update PostgresConverter to correctly stringify BigDecimals without scientific notation #31309

[Source-Postgres] Update PostgresConverter to correctly stringify BigDecimals without scientific notation #31309

Conversation

nguyenaiden commented Oct 11, 2023 • edited Loading

Description

vercel bot commented Oct 11, 2023 • edited Loading

github-actions bot commented Oct 11, 2023 • edited by nguyenaiden Loading

Before Merging a Connector Pull Request

github-actions bot commented Oct 11, 2023 • edited Loading

Coverage report for source-postgres

airbyte-oss-build-runner commented Oct 12, 2023

source-postgres test report (commit 3614089f2c) - ❌

nguyenaiden commented Oct 12, 2023

airbyte-oss-build-runner commented Oct 12, 2023

source-postgres test report (commit 7b656c7ce9) - ❌

airbyte-oss-build-runner commented Oct 12, 2023

source-postgres-strict-encrypt test report (commit 7b656c7ce9) - ❌

nguyenaiden commented Oct 12, 2023

nguyenaiden commented Oct 12, 2023

airbyte-oss-build-runner commented Oct 12, 2023

source-postgres-strict-encrypt test report (commit c66f19def5) - ❌

airbyte-oss-build-runner commented Oct 12, 2023

source-postgres test report (commit c66f19def5) - ❌

airbyte-oss-build-runner commented Oct 12, 2023

source-postgres test report (commit 843f5b2cb6) - ❌

airbyte-oss-build-runner commented Oct 12, 2023

source-postgres-strict-encrypt test report (commit 843f5b2cb6) - ❌

nguyenaiden commented Oct 12, 2023

airbyte-oss-build-runner commented Oct 12, 2023

source-postgres test report (commit 843f5b2cb6) - ✅

airbyte-oss-build-runner commented Oct 12, 2023

source-postgres-strict-encrypt test report (commit 843f5b2cb6) - ✅

airbyte-oss-build-runner commented Oct 13, 2023

source-postgres-strict-encrypt test report (commit e558bea7e7) - ✅

airbyte-oss-build-runner commented Oct 13, 2023

source-postgres test report (commit e558bea7e7) - ✅

airbyte-oss-build-runner commented Oct 13, 2023

source-postgres test report (commit d94788be03) - ✅

airbyte-oss-build-runner commented Oct 13, 2023

source-postgres-strict-encrypt test report (commit d94788be03) - ✅

nguyenaiden commented Oct 11, 2023 •

edited

Loading

vercel bot commented Oct 11, 2023 •

edited

Loading

github-actions bot commented Oct 11, 2023 •

edited by nguyenaiden

Loading

github-actions bot commented Oct 11, 2023 •

edited

Loading

source-postgres test report (commit `3614089f2c`) - ❌

source-postgres test report (commit `7b656c7ce9`) - ❌

source-postgres-strict-encrypt test report (commit `7b656c7ce9`) - ❌

source-postgres-strict-encrypt test report (commit `c66f19def5`) - ❌

source-postgres test report (commit `c66f19def5`) - ❌

source-postgres test report (commit `843f5b2cb6`) - ❌

source-postgres-strict-encrypt test report (commit `843f5b2cb6`) - ❌

source-postgres test report (commit `843f5b2cb6`) - ✅

source-postgres-strict-encrypt test report (commit `843f5b2cb6`) - ✅

source-postgres-strict-encrypt test report (commit `e558bea7e7`) - ✅

source-postgres test report (commit `e558bea7e7`) - ✅

source-postgres test report (commit `d94788be03`) - ✅

source-postgres-strict-encrypt test report (commit `d94788be03`) - ✅