Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Core extensions to support database replication #947

Closed
jorritsandbrink opened this issue Feb 7, 2024 · 2 comments · Fixed by #960
Closed

Core extensions to support database replication #947

jorritsandbrink opened this issue Feb 7, 2024 · 2 comments · Fixed by #960
Assignees
Labels
enhancement New feature or request

Comments

@jorritsandbrink
Copy link
Collaborator

Feature description

Extend the core to enable database replication. This needs to be done before #933.

Context:

  1. https://dlthub-community.slack.com/archives/C06EYJ2Q37C/p1707135498114549?thread_ts=1706550675.731069&cid=C06EYJ2Q37C
    image

  2. Postgres database replication #933 (comment)
    image

Are you a dlt user?

None

Use case

Database replication, CDC, efficient incremental processing

Proposed solution

No response

Related issues

#933

@jorritsandbrink jorritsandbrink self-assigned this Feb 7, 2024
@jorritsandbrink jorritsandbrink added the enhancement New feature or request label Feb 7, 2024
@jorritsandbrink jorritsandbrink moved this from Todo to In Progress in dlt core library Feb 7, 2024
@jorritsandbrink
Copy link
Collaborator Author

@rudolfix It seems like typically the entire record is provided for update operations, even if only a subset of columns has been changed. I checked Postgres, MySQL, SQL Server, and Delta tables.

Do we need partial updates?

@rudolfix
Copy link
Collaborator

@jorritsandbrink does it make things easier? I assume in that case we could reuse existing merge job right? then we do need only a flag to delete records because existing code handles replace rows and inserts by checking the primary keys.

it would be cool to have partial updates but if that is 2x more work then let's do the above instead

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

2 participants