Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Destination Iceberg: integration test for glue #49467

Open
wants to merge 15 commits into
base: master
Choose a base branch
from
Open

Destination Iceberg: integration test for glue #49467

wants to merge 15 commits into from

Conversation

edgao
Copy link
Contributor

@edgao edgao commented Dec 13, 2024 •
edited
Loading

closes https://github.com/airbytehq/airbyte-internal-issues/issues/11140

key changes

  • Handle uppercase stream name/namespace (glue requires lowercase here). The TableIdGenerator thing is... kind of dumb, but it works I guess (we should eventually plug in the default namespace stuff there)
    • killed the DestinationStream.Descriptor.toIcebergTableIdentifier extension function
  • implement a destination cleaner. It just nukes any tables older than 30 days. (only needed for glue, since nessie is all ephemeral test container stuff)
  • update data dumper to use the correct catalog, instead of hardcoding nessie
  • enable the check test, targeting the glue config

@vercel Vercel
Copy link

vercel bot commented Dec 13, 2024 •
edited
Loading

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
airbyte-docs ✅ Ready (Inspect) Visit Preview 💬 Add feedback Dec 16, 2024 10:24pm

@octavia-squidington-iii octavia-squidington-iii added area/connectors Connector related issues CDK Connector Development Kit connectors/destination/iceberg-v2 labels Dec 13, 2024
@edgao edgao force-pushed the edgao/iceberg_glue_integration_test branch from 5d23bfb to baa3404 Compare December 16, 2024 18:34
@edgao edgao force-pushed the edgao/iceberg_glue_integration_test branch from 7db4b88 to b30724b Compare December 16, 2024 19:42
@edgao edgao force-pushed the edgao/iceberg_glue_integration_test branch from d12c119 to e1b9e0d Compare December 16, 2024 20:58
@edgao edgao force-pushed the edgao/iceberg_glue_integration_test branch from e1b9e0d to 86ddc19 Compare December 16, 2024 21:00
@edgao edgao requested a review from jdpgrailsdev December 16, 2024 22:09
@edgao edgao marked this pull request as ready for review December 16, 2024 22:09
@edgao edgao requested a review from a team as a code owner December 16, 2024 22:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jdpgrailsdev jdpgrailsdev Awaiting requested review from jdpgrailsdev

@subodh1810 subodh1810 Awaiting requested review from subodh1810

@frifriSF59 frifriSF59 Awaiting requested review from frifriSF59

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
area/connectors Connector related issues CDK Connector Development Kit connectors/destination/iceberg-v2
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@edgao @octavia-squidington-iii