Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Truncate(dev-guide): Added steps for truncate #51

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

Truncate(dev-guide): Added steps for truncate #51

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
dd9fc48
modified truncate in dev guide
fivetran-abdulsalam Jun 21, 2024
da2fd19
changed grammar and small refactors
fivetran-abdulsalam Jun 21, 2024
1dd8093
Merge branch 'main' into ft-sdk-truncate-dev-guide
fivetran-abdulsalam Jun 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions development-guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,9 +101,14 @@ This operation should fail if it is asked to create a table that already exists.
#### DescribeTable
This operation should report all columns in the destination table, including Fivetran system columns such as `_fivetran_synced` and `_fivetran_deleted`. It should also provide other additional information as applicable such as data type, `primary_key` and `DecimalParams`.

#### Truncate
#### TruncateRequest
- This operation might be requested for a table that does not exist in the destination. In that case, it should NOT fail, simply ignore the request and return `success = true`.
- `utc_delete_before` has millisecond precision.
- `SoftTruncate`: This optional field can be configured to either regular `soft-truncate` mode by setting the `deleted_column` field or history `soft-truncate` mode by setting the `history_mode` field to TRUE. Both modes keep the rows in the destination and update relevant system columns to mark any matching rows as deleted. If optional `SoftTruncate` field is not configured, then perform a `hard-truncate` operation where matching rows are actually removed from the destination.
- `utc_delete_before`: This timestamp field has millisecond precision and used to DELETE/UPDATE the rows in destination table as per the following:
- **Soft-Truncate:** Set `deleted_column` to TRUE where `_fivetran_synced` column value is less than `utc_delete_before`.
- **History Truncate:** Set `_fivetran_active` column value to FALSE, `_fivetran_end` column value to `utc_delete_before` value, where `_fivetran_synced` column value is less than `utc_delete_before` and `_fivetran_active` is TRUE.
- **Hard Truncate:** DELETE the record where `_fivetran_synced` column value is less than `utc_delete_before`


#### WriteBatchRequest
- `replace_files` is for `upsert` operation where the rows should be inserted if they don't exist or updated if they do. Each row will always provide values for all columns. Set the `_fivetran_synced` column in the destination with the values coming in from the csv files.
Expand Down
Loading