Make merge write-disposition fall back to append if no primary or merge keys are specified #1225
Conversation
✅ Deploy Preview for dlt-hub-docs ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
|
@rudolfix comments from the last pr (which somehow got automatically closed when I reset this branch) === === |
tests/pipeline/utils.py
Outdated
| @@ -3,6 +3,7 @@ | |||
| import pytest | |||
| import random | |||
| from os import environ | |||
| import io | |||
There was a problem hiding this comment.
i pulled this over from my filesystem state branch because i needed those changes here.
| @@ -226,6 +226,10 @@ def destinations_configs( | |||
| DestinationTestConfiguration(destination="synapse", supports_dbt=False), | |||
| ] | |||
|
|
|||
| # sanity check that when selecting default destinations, one of each sql destination is actually | |||
| # provided | |||
| assert set(SQL_DESTINATIONS) == {d.destination for d in destination_configs} | |||
There was a problem hiding this comment.
this has happened a few times already... not really related to this pr
|
|
||
| assert local_path.read_bytes() == client.fs_client.read_bytes(destination_path) | ||
|
|
||
|
|
||
| def test_pipeline_merge_write_disposition(default_buckets_env: str) -> None: |
There was a problem hiding this comment.
this test shows that filesystem always falls back to append when "merge" is set. this is also what it says in the docs. I'm not sure why this test says it would replace when there is a primary key, this did not work before and it does not work that way now and this test was somehow very strange (not sure if I wrote that) but now it runs correctly. I we want to replace in certain cases, I have to add it and specify that in the docs.
sh-rp
changed the title
# Conflicts: # tests/load/pipeline/test_filesystem_pipeline.py
Description
Fall back to append if no merge keys are available