remove update_iceberg_ts from iceberg table when using merge incremental strategy #453
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
aiss93
force-pushed
the
fix/remove_update_iceberg_ts
branch
from
b9ca3d3 to
e5fe7ab
Compare
aiss93
force-pushed
the
fix/remove_update_iceberg_ts
branch
from
e5fe7ab to
7c00efa
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
resolves #452
Description
In the current implementation, when using the merge incremental strategy with the Iceberg file format, the adapter automatically adds a column update_iceberg_ts, which is not utilized in the merge process. This causes issues such as breaking schema comparisons between tables in certain scenarios, making the hard-coding of this column restrictive.
This PR removes the hard-coded update_iceberg_ts column from the adapter. If users require this column for specific use cases, they can now add it directly within their model configuration.
Checklist
CHANGELOG.mdand added information about my change to the "dbt-glue next" section.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.