From dd9fc48c9342fc0110bb63e0cb4ff273036776c8 Mon Sep 17 00:00:00 2001 From: fivetran-abdulsalam Date: Fri, 21 Jun 2024 17:43:21 +0530 Subject: [PATCH 1/2] modified truncate in dev guide --- development-guide.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/development-guide.md b/development-guide.md index b5bf80c..38a0e57 100644 --- a/development-guide.md +++ b/development-guide.md @@ -101,9 +101,14 @@ This operation should fail if it is asked to create a table that already exists. #### DescribeTable This operation should report all columns in the destination table, including Fivetran system columns such as `_fivetran_synced` and `_fivetran_deleted`. It should also provide other additional information as applicable such as data type, `primary_key` and `DecimalParams`. -#### Truncate +#### TruncateRequest - This operation might be requested for a table that does not exist in the destination. In that case, it should NOT fail, simply ignore the request and return `success = true`. -- `utc_delete_before` has millisecond precision. +- It contains an optional `SoftTruncate` field which either contains a string column `deleted_column` or boolean `history_mode`. If `deleted_column` is present, then current sync mode is `soft-delete-mode` and if `history_mode` is true then current sync mode is `history-mode`, otherwise the current sync mode is `live-mode`. +- It contains `utc_delete_before` timestamp field which has millisecond precision and use it to DELETE/UPDATE the rows in destination table as per the sync mode: + - **Soft-Delete Mode:** Set `deleted_column` to TRUE where `_fivetran_synced` column value is less than `utc_delete_before`. + - **History Mode:** Set `_fivetran_active` column value to FALSE, `_fivetran_end` column value to `utc_delete_before` value, where `_fivetran_synced` column value is less than `utc_delete_before` and `_fivetran_active` is TRUE. + - **Live Mode:** DELETE the record where `_fivetran_synced` column value is less than `utc_delete_before` + #### WriteBatchRequest - `replace_files` is for `upsert` operation where the rows should be inserted if they don't exist or updated if they do. Each row will always provide values for all columns. Set the `_fivetran_synced` column in the destination with the values coming in from the csv files. From da2fd19b5e6715f7644fbc1fede5d6d7f9c21ef2 Mon Sep 17 00:00:00 2001 From: fivetran-abdulsalam Date: Fri, 21 Jun 2024 21:13:47 +0530 Subject: [PATCH 2/2] changed grammar and small refactors --- development-guide.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/development-guide.md b/development-guide.md index 38a0e57..3fcb8d0 100644 --- a/development-guide.md +++ b/development-guide.md @@ -103,11 +103,11 @@ This operation should report all columns in the destination table, including Fiv #### TruncateRequest - This operation might be requested for a table that does not exist in the destination. In that case, it should NOT fail, simply ignore the request and return `success = true`. -- It contains an optional `SoftTruncate` field which either contains a string column `deleted_column` or boolean `history_mode`. If `deleted_column` is present, then current sync mode is `soft-delete-mode` and if `history_mode` is true then current sync mode is `history-mode`, otherwise the current sync mode is `live-mode`. -- It contains `utc_delete_before` timestamp field which has millisecond precision and use it to DELETE/UPDATE the rows in destination table as per the sync mode: - - **Soft-Delete Mode:** Set `deleted_column` to TRUE where `_fivetran_synced` column value is less than `utc_delete_before`. - - **History Mode:** Set `_fivetran_active` column value to FALSE, `_fivetran_end` column value to `utc_delete_before` value, where `_fivetran_synced` column value is less than `utc_delete_before` and `_fivetran_active` is TRUE. - - **Live Mode:** DELETE the record where `_fivetran_synced` column value is less than `utc_delete_before` +- `SoftTruncate`: This optional field can be configured to either regular `soft-truncate` mode by setting the `deleted_column` field or history `soft-truncate` mode by setting the `history_mode` field to TRUE. Both modes keep the rows in the destination and update relevant system columns to mark any matching rows as deleted. If optional `SoftTruncate` field is not configured, then perform a `hard-truncate` operation where matching rows are actually removed from the destination. +- `utc_delete_before`: This timestamp field has millisecond precision and used to DELETE/UPDATE the rows in destination table as per the following: + - **Soft-Truncate:** Set `deleted_column` to TRUE where `_fivetran_synced` column value is less than `utc_delete_before`. + - **History Truncate:** Set `_fivetran_active` column value to FALSE, `_fivetran_end` column value to `utc_delete_before` value, where `_fivetran_synced` column value is less than `utc_delete_before` and `_fivetran_active` is TRUE. + - **Hard Truncate:** DELETE the record where `_fivetran_synced` column value is less than `utc_delete_before` #### WriteBatchRequest