Make merge write_disposition fall back to append behavior when no merge or primary keys are set #1047
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for dlt-hub-docs ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
rudolfix
requested changes
Mar 4, 2024
There was a problem hiding this comment.
it is a good start :)
- there's already test for it (
test_merge_no_merge_keys) - we should also test what happens if we have hard delete column
- filesystem also fallbacks to replace when there are no keys (and I think there's a test for it)
- what about vector databases? they support limited merge (on primary key). I think they do not fallback to replace but make sure
dlt/destinations/sql_jobs.py
Outdated
| [t["name"] for t in table_chain], | ||
| f"There is no unique column (ie _dlt_id) in top table {root_table['name']} so" | ||
| " it is not possible to link child tables to it.", | ||
| if not fallback_to_append: |
There was a problem hiding this comment.
maybe we could extract delete code block to separate method?
|
@sh-rp also you must update docs where I think we say we replace |
sh-rp
changed the title
sh-rp
changed the title
sh-rp
force-pushed
the
d#/merge_fallback
branch
2 times, most recently
from
4291338 to
61120f2
Compare
sh-rp
force-pushed
the
d#/merge_fallback
branch
from
ef96969 to
0abad12
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Currently the merge jobs will delete all data in the tables if no merge or primary keys are present, this behavior is not intuitive and can also lead to unexpected data loss.