Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Truncate(dev-guide): Added steps for truncate #51

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from
Draft
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions development-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,9 +101,14 @@ This operation should fail if it is asked to create a table that already exists.
#### DescribeTable
This operation should report all columns in the destination table, including Fivetran system columns such as `_fivetran_synced` and `_fivetran_deleted`. It should also provide other additional information as applicable such as data type, `primary_key` and `DecimalParams`.

#### Truncate
#### TruncateRequest
- This operation might be requested for a table that does not exist in the destination. In that case, it should NOT fail, simply ignore the request and return `success = true`.
- `utc_delete_before` has millisecond precision.
- It contains an optional `SoftTruncate` field which either contains a string column `deleted_column` or boolean `history_mode`. If `deleted_column` is present, then current sync mode is `soft-delete-mode` and if `history_mode` is true then current sync mode is `history-mode`, otherwise the current sync mode is `live-mode`.
fivetran-abdulsalam marked this conversation as resolved.
Show resolved Hide resolved
- It contains `utc_delete_before` timestamp field which has millisecond precision and use it to DELETE/UPDATE the rows in destination table as per the sync mode:
fivetran-abdulsalam marked this conversation as resolved.
Show resolved Hide resolved
- **Soft-Delete Mode:** Set `deleted_column` to TRUE where `_fivetran_synced` column value is less than `utc_delete_before`.
fivetran-abdulsalam marked this conversation as resolved.
Show resolved Hide resolved
- **History Mode:** Set `_fivetran_active` column value to FALSE, `_fivetran_end` column value to `utc_delete_before` value, where `_fivetran_synced` column value is less than `utc_delete_before` and `_fivetran_active` is TRUE.
fivetran-abdulsalam marked this conversation as resolved.
Show resolved Hide resolved
- **Live Mode:** DELETE the record where `_fivetran_synced` column value is less than `utc_delete_before`
fivetran-abdulsalam marked this conversation as resolved.
Show resolved Hide resolved


#### WriteBatchRequest
- `replace_files` is for `upsert` operation where the rows should be inserted if they don't exist or updated if they do. Each row will always provide values for all columns. Set the `_fivetran_synced` column in the destination with the values coming in from the csv files.
Expand Down
Loading