From 73c0e5625beb6134ce87b660e131c1b370def707 Mon Sep 17 00:00:00 2001 From: Satvik Patil Date: Fri, 7 Jun 2024 12:42:32 +0530 Subject: [PATCH 01/20] doc: setup form and connector guidelines --- development-guide.md | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/development-guide.md b/development-guide.md index 86ad9db..47b7b69 100644 --- a/development-guide.md +++ b/development-guide.md @@ -61,13 +61,32 @@ The following are hard requirements to be able to deploy Partner code to Fivetra - Encrypt HTTP requests: Things like URLs, URL parameters, and query params are always encrypted for logging, and customer approval is needed to decrypt and examine them. -## Connector Guidelines +## Setup Form Guidelines +- Keep the form clear and concise, only requesting essential information for successful connector setup. +- Use clear and descriptive labels for each form field. Avoid technical jargon if possible. +- Organize the fields in a logical order that reflects the setup process. + +### RPC Calls +#### ConfigurationForm +This operation retrieves all the setup form fields and tests info required to establish a connection. You can provide various parameters for the fields to enhance the user experience, such as descriptions, optional fields, and more. + +#### Test +The previous RPC call retrieves the tests that need to be executed during connection setup. This operation then invokes the test with the customer's credentials as parameters. Finally, it should return a success or failure indication for the test execution. + +## Source Connector Guidelines - Don't push anything other than source data to the destination. State will be saved to production DB and returned in `UpdateRequest`. - Don't forget to handle new schemas/tables/columns per the information and user choices in `UpdateRequest#selection`. - Make sure you checkpoint at least once an hour. The more frequently you do it, the better. -## Destination Guidelines +### RPC Calls +#### Schema +This operation should retrieve all the information about the customer's schemas, tables, and columns. It also offers an optional `selection_not_supported` field that indicates whether customers can select or deselect tables and columns within the Fivetran dashboard. + +#### Update +This operation should retrieve data from the source. We provide a request using the `UpdateRequest` message, which includes the customer's state, credentials, and schema information. The response, delivered through the `UpdateResponse` message, should contain data records or other supported operations. + +## Destination Connector Guidelines - Do not push anything other than source data to the destination. From 1e234be980a44a4df58cb083fb2a047b9b937722 Mon Sep 17 00:00:00 2001 From: Satvik Patil Date: Fri, 7 Jun 2024 15:13:36 +0530 Subject: [PATCH 02/20] destination guideline changes --- development-guide.md | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/development-guide.md b/development-guide.md index 47b7b69..1360b5a 100644 --- a/development-guide.md +++ b/development-guide.md @@ -124,13 +124,18 @@ This operation should report all columns in the destination table, including Fiv - This operation might be requested for a table that does not exist in the destination. In that case, it should NOT fail, simply ignore the request and return `success = true`. - `utc_delete_before` has millisecond precision. -#### WriteBatchRequest +#### WriteBatch +This operations provides details about the batch files containing the retrieved data. We provide a parameter `WriteBatchRequest` which contains all the information required for you to read the batch files. Here are some of the fields included in the request message:: - `replace_files` is for `upsert` operation where the rows should be inserted if they don't exist or updated if they do. Each row will always provide values for all columns. Set the `_fivetran_synced` column in the destination with the values coming in from the csv files. - `update_files` is for `update` operation where modified columns have actual values whereas unmodified columns have the special value `unmodified_string` in `CsvFileParams`. Soft-deleted rows will arrive in here as well. Update the `_fivetran_synced` column in the destination with the values coming in from the csv files. - `delete_files` is for `hard delete` operation. Use primary key columns (or `_fivetran_id` system column for primary-keyless tables) to perform `DELETE FROM`. +- `keys` is a map that provides a list of secret keys that can be used to decrypt batch files. + +- `file_params` provides information about the file type and any configurations applied to it, such as encryption or compression. + Also, Fivetran will deduplicate operations such that each primary key will show up only once in any of the operations Do not assume order of columns in the batch files. Always read the CSV file header to determine column order. @@ -139,6 +144,19 @@ Do not assume order of columns in the batch files. Always read the CSV file head - `null_string` value is used to represent `NULL` value in all batch files. - `unmodified_string` value is used to indicate columns in `update_files` where the values did not change. +#### Capabilities +This operation offers several additional features, as listed below: + +- Datatype Mappings: Supports adjusting partner data types for each Fivetran data type. +- Max value for columns: Provides an option to specify the maximum value for each data type. + +#### AlterTable +This operation is used to communicate updates to a table. The `SchemaDiff` message within the `AlterTableRequest` parameter provides details about these updates: +- Adding a column (`add_column`): Fivetran uses this field to provide information about a new column to be added in destination table. +- Update Column type (`change_column_type`): This field provides the info on updated Column from source that needs to be reflected in destination table. +- Primary key updates (`new_primary_key`): If the customer's primary key have changed, this field lists all the new primary key used. + + ### Examples of Data Types Examples of each [DataType](https://github.com/fivetran/fivetran_sdk/blob/main/common.proto#L73C6-L73C14) as they would appear in CSV batch files are as follows: - UNSPECIFIED: This data type will never appear in batch files From 56690d99c4f7dcdfa69d3087d3220303b89842e0 Mon Sep 17 00:00:00 2001 From: SatvikPatil <54502802+fivetran-satvikpatil@users.noreply.github.com> Date: Wed, 12 Jun 2024 20:09:30 +0530 Subject: [PATCH 03/20] Update development-guide.md Co-authored-by: Alex Ilyichov --- development-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/development-guide.md b/development-guide.md index 1360b5a..a3441f9 100644 --- a/development-guide.md +++ b/development-guide.md @@ -125,7 +125,7 @@ This operation should report all columns in the destination table, including Fiv - `utc_delete_before` has millisecond precision. #### WriteBatch -This operations provides details about the batch files containing the retrieved data. We provide a parameter `WriteBatchRequest` which contains all the information required for you to read the batch files. Here are some of the fields included in the request message:: +This operation provides details about the batch files containing the retrieved data. We provide a parameter `WriteBatchRequest` which contains all the information required for you to read the batch files. Here are some of the fields included in the request message: - `replace_files` is for `upsert` operation where the rows should be inserted if they don't exist or updated if they do. Each row will always provide values for all columns. Set the `_fivetran_synced` column in the destination with the values coming in from the csv files. - `update_files` is for `update` operation where modified columns have actual values whereas unmodified columns have the special value `unmodified_string` in `CsvFileParams`. Soft-deleted rows will arrive in here as well. Update the `_fivetran_synced` column in the destination with the values coming in from the csv files. From f4de3d42f964d220c1ad372112cb61d838aa0d5c Mon Sep 17 00:00:00 2001 From: SatvikPatil <54502802+fivetran-satvikpatil@users.noreply.github.com> Date: Wed, 12 Jun 2024 20:09:51 +0530 Subject: [PATCH 04/20] Update development-guide.md Co-authored-by: Alex Ilyichov --- development-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/development-guide.md b/development-guide.md index a3441f9..377516b 100644 --- a/development-guide.md +++ b/development-guide.md @@ -153,7 +153,7 @@ This operation offers several additional features, as listed below: #### AlterTable This operation is used to communicate updates to a table. The `SchemaDiff` message within the `AlterTableRequest` parameter provides details about these updates: - Adding a column (`add_column`): Fivetran uses this field to provide information about a new column to be added in destination table. -- Update Column type (`change_column_type`): This field provides the info on updated Column from source that needs to be reflected in destination table. +- Update Column type (`change_column_type`): This field provides information on updated type of a column in the source that needs to be reflected in a destination table. - Primary key updates (`new_primary_key`): If the customer's primary key have changed, this field lists all the new primary key used. From e11a0681a4aef42caf48186ae9b099fdce1ce00e Mon Sep 17 00:00:00 2001 From: Satvik Patil Date: Thu, 13 Jun 2024 10:33:27 +0530 Subject: [PATCH 05/20] doc changes --- development-guide.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/development-guide.md b/development-guide.md index 377516b..5c2a590 100644 --- a/development-guide.md +++ b/development-guide.md @@ -68,7 +68,7 @@ The following are hard requirements to be able to deploy Partner code to Fivetra ### RPC Calls #### ConfigurationForm -This operation retrieves all the setup form fields and tests info required to establish a connection. You can provide various parameters for the fields to enhance the user experience, such as descriptions, optional fields, and more. +This operation retrieves all the setup form fields and tests information. You can provide various parameters for the fields to enhance the user experience, such as descriptions, optional fields, and more. #### Test The previous RPC call retrieves the tests that need to be executed during connection setup. This operation then invokes the test with the customer's credentials as parameters. Finally, it should return a success or failure indication for the test execution. @@ -81,7 +81,7 @@ The previous RPC call retrieves the tests that need to be executed during connec ### RPC Calls #### Schema -This operation should retrieve all the information about the customer's schemas, tables, and columns. It also offers an optional `selection_not_supported` field that indicates whether customers can select or deselect tables and columns within the Fivetran dashboard. +This operation retrieves the customer's schemas, tables, and columns. It also includes an optional `selection_not_supported` field that indicates whether customers can select or deselect tables and columns within the Fivetran dashboard. #### Update This operation should retrieve data from the source. We provide a request using the `UpdateRequest` message, which includes the customer's state, credentials, and schema information. The response, delivered through the `UpdateResponse` message, should contain data records or other supported operations. @@ -154,7 +154,7 @@ This operation offers several additional features, as listed below: This operation is used to communicate updates to a table. The `SchemaDiff` message within the `AlterTableRequest` parameter provides details about these updates: - Adding a column (`add_column`): Fivetran uses this field to provide information about a new column to be added in destination table. - Update Column type (`change_column_type`): This field provides information on updated type of a column in the source that needs to be reflected in a destination table. -- Primary key updates (`new_primary_key`): If the customer's primary key have changed, this field lists all the new primary key used. +- Primary key updates (`new_primary_key`): If the primary key has changed, this field lists all the new columns used in the updated primary key. ### Examples of Data Types From bd035c88154cee05465fcf30f7909fcfbc73ccda Mon Sep 17 00:00:00 2001 From: SatvikPatil <54502802+fivetran-satvikpatil@users.noreply.github.com> Date: Fri, 14 Jun 2024 10:08:08 +0530 Subject: [PATCH 06/20] Update development-guide.md Co-authored-by: Alex Ilyichov --- development-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/development-guide.md b/development-guide.md index 5c2a590..b2b048a 100644 --- a/development-guide.md +++ b/development-guide.md @@ -125,7 +125,7 @@ This operation should report all columns in the destination table, including Fiv - `utc_delete_before` has millisecond precision. #### WriteBatch -This operation provides details about the batch files containing the retrieved data. We provide a parameter `WriteBatchRequest` which contains all the information required for you to read the batch files. Here are some of the fields included in the request message: +This operation provides details about the batch files containing the retrieved data. We provide the `WriteBatchRequest` parameter that contains all the information required for you to read the batch files. Here are some of the fields included in the request message: - `replace_files` is for `upsert` operation where the rows should be inserted if they don't exist or updated if they do. Each row will always provide values for all columns. Set the `_fivetran_synced` column in the destination with the values coming in from the csv files. - `update_files` is for `update` operation where modified columns have actual values whereas unmodified columns have the special value `unmodified_string` in `CsvFileParams`. Soft-deleted rows will arrive in here as well. Update the `_fivetran_synced` column in the destination with the values coming in from the csv files. From a60f7c0500b42d3a2d9e5ac38f40cabafe198335 Mon Sep 17 00:00:00 2001 From: SatvikPatil <54502802+fivetran-satvikpatil@users.noreply.github.com> Date: Fri, 14 Jun 2024 10:08:55 +0530 Subject: [PATCH 07/20] Update development-guide.md Co-authored-by: Alex Ilyichov --- development-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/development-guide.md b/development-guide.md index b2b048a..147af20 100644 --- a/development-guide.md +++ b/development-guide.md @@ -126,7 +126,7 @@ This operation should report all columns in the destination table, including Fiv #### WriteBatch This operation provides details about the batch files containing the retrieved data. We provide the `WriteBatchRequest` parameter that contains all the information required for you to read the batch files. Here are some of the fields included in the request message: -- `replace_files` is for `upsert` operation where the rows should be inserted if they don't exist or updated if they do. Each row will always provide values for all columns. Set the `_fivetran_synced` column in the destination with the values coming in from the csv files. +- `replace_files` is for the `upsert` operation where the rows should be inserted if they don't exist or updated if they do. Each row will always provide values for all columns. Set the `_fivetran_synced` column in the destination with the values coming in from the .CSV files. - `update_files` is for `update` operation where modified columns have actual values whereas unmodified columns have the special value `unmodified_string` in `CsvFileParams`. Soft-deleted rows will arrive in here as well. Update the `_fivetran_synced` column in the destination with the values coming in from the csv files. From 2e6c2d1a15b6845eb51730ea393584343aad6f25 Mon Sep 17 00:00:00 2001 From: SatvikPatil <54502802+fivetran-satvikpatil@users.noreply.github.com> Date: Fri, 14 Jun 2024 10:09:08 +0530 Subject: [PATCH 08/20] Update development-guide.md Co-authored-by: Alex Ilyichov --- development-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/development-guide.md b/development-guide.md index 147af20..3fe42f5 100644 --- a/development-guide.md +++ b/development-guide.md @@ -128,7 +128,7 @@ This operation should report all columns in the destination table, including Fiv This operation provides details about the batch files containing the retrieved data. We provide the `WriteBatchRequest` parameter that contains all the information required for you to read the batch files. Here are some of the fields included in the request message: - `replace_files` is for the `upsert` operation where the rows should be inserted if they don't exist or updated if they do. Each row will always provide values for all columns. Set the `_fivetran_synced` column in the destination with the values coming in from the .CSV files. -- `update_files` is for `update` operation where modified columns have actual values whereas unmodified columns have the special value `unmodified_string` in `CsvFileParams`. Soft-deleted rows will arrive in here as well. Update the `_fivetran_synced` column in the destination with the values coming in from the csv files. +- `update_files` is for the `update` operation where modified columns have actual values whereas unmodified columns have the special value `unmodified_string` in `CsvFileParams`. Soft-deleted rows will arrive in here as well. Update the `_fivetran_synced` column in the destination with the values coming in from the .CSV files. - `delete_files` is for `hard delete` operation. Use primary key columns (or `_fivetran_id` system column for primary-keyless tables) to perform `DELETE FROM`. From 1db8605d6c4e0abac51e2636ef0939214092399f Mon Sep 17 00:00:00 2001 From: SatvikPatil <54502802+fivetran-satvikpatil@users.noreply.github.com> Date: Fri, 14 Jun 2024 10:09:19 +0530 Subject: [PATCH 09/20] Update development-guide.md Co-authored-by: Alex Ilyichov --- development-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/development-guide.md b/development-guide.md index 3fe42f5..3fd9dcf 100644 --- a/development-guide.md +++ b/development-guide.md @@ -151,7 +151,7 @@ This operation offers several additional features, as listed below: - Max value for columns: Provides an option to specify the maximum value for each data type. #### AlterTable -This operation is used to communicate updates to a table. The `SchemaDiff` message within the `AlterTableRequest` parameter provides details about these updates: +This operation is used to communicate updates to a table. The `SchemaDiff` message within the `AlterTableRequest` parameter provides the details about these updates: - Adding a column (`add_column`): Fivetran uses this field to provide information about a new column to be added in destination table. - Update Column type (`change_column_type`): This field provides information on updated type of a column in the source that needs to be reflected in a destination table. - Primary key updates (`new_primary_key`): If the primary key has changed, this field lists all the new columns used in the updated primary key. From 87a4c36f0ef03bfadb6918abe2afaa9ece765951 Mon Sep 17 00:00:00 2001 From: SatvikPatil <54502802+fivetran-satvikpatil@users.noreply.github.com> Date: Fri, 14 Jun 2024 10:09:31 +0530 Subject: [PATCH 10/20] Update development-guide.md Co-authored-by: Alex Ilyichov --- development-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/development-guide.md b/development-guide.md index 3fd9dcf..515abf3 100644 --- a/development-guide.md +++ b/development-guide.md @@ -152,7 +152,7 @@ This operation offers several additional features, as listed below: #### AlterTable This operation is used to communicate updates to a table. The `SchemaDiff` message within the `AlterTableRequest` parameter provides the details about these updates: -- Adding a column (`add_column`): Fivetran uses this field to provide information about a new column to be added in destination table. +- Adding a column (`add_column`): Fivetran uses this field to provide information about a new column to be added in a destination table. - Update Column type (`change_column_type`): This field provides information on updated type of a column in the source that needs to be reflected in a destination table. - Primary key updates (`new_primary_key`): If the primary key has changed, this field lists all the new columns used in the updated primary key. From a84d446db004c6a318552daa0cc98fdd0e852238 Mon Sep 17 00:00:00 2001 From: Satvik Patil Date: Fri, 14 Jun 2024 10:20:24 +0530 Subject: [PATCH 11/20] doc: fix field name --- development-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/development-guide.md b/development-guide.md index 515abf3..f8d20d5 100644 --- a/development-guide.md +++ b/development-guide.md @@ -154,7 +154,7 @@ This operation offers several additional features, as listed below: This operation is used to communicate updates to a table. The `SchemaDiff` message within the `AlterTableRequest` parameter provides the details about these updates: - Adding a column (`add_column`): Fivetran uses this field to provide information about a new column to be added in a destination table. - Update Column type (`change_column_type`): This field provides information on updated type of a column in the source that needs to be reflected in a destination table. -- Primary key updates (`new_primary_key`): If the primary key has changed, this field lists all the new columns used in the updated primary key. +- Primary key updates (`updated_primary_keys`): If the primary key has changed, this field lists all the columns used in the updated primary key. ### Examples of Data Types From 707a050aeb361c8135083c2403f62f4bf3f9f8d0 Mon Sep 17 00:00:00 2001 From: SatvikPatil <54502802+fivetran-satvikpatil@users.noreply.github.com> Date: Tue, 18 Jun 2024 14:25:36 +0530 Subject: [PATCH 12/20] Update development-guide.md Co-authored-by: Emrah Diril --- development-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/development-guide.md b/development-guide.md index f8d20d5..e5df160 100644 --- a/development-guide.md +++ b/development-guide.md @@ -84,7 +84,7 @@ The previous RPC call retrieves the tests that need to be executed during connec This operation retrieves the customer's schemas, tables, and columns. It also includes an optional `selection_not_supported` field that indicates whether customers can select or deselect tables and columns within the Fivetran dashboard. #### Update -This operation should retrieve data from the source. We provide a request using the `UpdateRequest` message, which includes the customer's state, credentials, and schema information. The response, delivered through the `UpdateResponse` message, should contain data records or other supported operations. +This operation should retrieve data from the source. We send a request using the `UpdateRequest` message, which includes the customer's state, credentials, and schema information. The response, streaming through the `UpdateResponse` message, can contain data records and other supported operations. ## Destination Connector Guidelines From b4941283f20907274b37fe3476ac0db943c86dff Mon Sep 17 00:00:00 2001 From: SatvikPatil <54502802+fivetran-satvikpatil@users.noreply.github.com> Date: Tue, 18 Jun 2024 14:25:48 +0530 Subject: [PATCH 13/20] Update development-guide.md Co-authored-by: Emrah Diril --- development-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/development-guide.md b/development-guide.md index e5df160..ea1cc5f 100644 --- a/development-guide.md +++ b/development-guide.md @@ -125,7 +125,7 @@ This operation should report all columns in the destination table, including Fiv - `utc_delete_before` has millisecond precision. #### WriteBatch -This operation provides details about the batch files containing the retrieved data. We provide the `WriteBatchRequest` parameter that contains all the information required for you to read the batch files. Here are some of the fields included in the request message: +This operation provides details about the batch files containing the records to be pushed to the destination. We provide the `WriteBatchRequest` parameter that contains all the information required for you to read the batch files. Here are some of the fields included in the request message: - `replace_files` is for the `upsert` operation where the rows should be inserted if they don't exist or updated if they do. Each row will always provide values for all columns. Set the `_fivetran_synced` column in the destination with the values coming in from the .CSV files. - `update_files` is for the `update` operation where modified columns have actual values whereas unmodified columns have the special value `unmodified_string` in `CsvFileParams`. Soft-deleted rows will arrive in here as well. Update the `_fivetran_synced` column in the destination with the values coming in from the .CSV files. From d93581f90f363ef73fbf4ebb2503b221a6f8e133 Mon Sep 17 00:00:00 2001 From: SatvikPatil <54502802+fivetran-satvikpatil@users.noreply.github.com> Date: Tue, 18 Jun 2024 14:26:05 +0530 Subject: [PATCH 14/20] Update development-guide.md Co-authored-by: Emrah Diril --- development-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/development-guide.md b/development-guide.md index ea1cc5f..c6c4323 100644 --- a/development-guide.md +++ b/development-guide.md @@ -132,7 +132,7 @@ This operation provides details about the batch files containing the records to - `delete_files` is for `hard delete` operation. Use primary key columns (or `_fivetran_id` system column for primary-keyless tables) to perform `DELETE FROM`. -- `keys` is a map that provides a list of secret keys that can be used to decrypt batch files. +- `keys` is a map that provides a list of secret keys, one for each batch file, that can be used to decrypt them. - `file_params` provides information about the file type and any configurations applied to it, such as encryption or compression. From c784c968c6f35f6bba70aee30a225778aa601851 Mon Sep 17 00:00:00 2001 From: SatvikPatil <54502802+fivetran-satvikpatil@users.noreply.github.com> Date: Tue, 18 Jun 2024 14:26:19 +0530 Subject: [PATCH 15/20] Update development-guide.md Co-authored-by: Emrah Diril --- development-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/development-guide.md b/development-guide.md index c6c4323..c6af4a6 100644 --- a/development-guide.md +++ b/development-guide.md @@ -145,7 +145,7 @@ Do not assume order of columns in the batch files. Always read the CSV file head - `unmodified_string` value is used to indicate columns in `update_files` where the values did not change. #### Capabilities -This operation offers several additional features, as listed below: +This operation offers the ability for the partner code to declare its choices for capabilities listed below: - Datatype Mappings: Supports adjusting partner data types for each Fivetran data type. - Max value for columns: Provides an option to specify the maximum value for each data type. From 5d931e670172a77fff958f27968032d050009f46 Mon Sep 17 00:00:00 2001 From: SatvikPatil <54502802+fivetran-satvikpatil@users.noreply.github.com> Date: Tue, 18 Jun 2024 14:26:32 +0530 Subject: [PATCH 16/20] Update development-guide.md Co-authored-by: Emrah Diril --- development-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/development-guide.md b/development-guide.md index c6af4a6..b407cc9 100644 --- a/development-guide.md +++ b/development-guide.md @@ -147,7 +147,7 @@ Do not assume order of columns in the batch files. Always read the CSV file head #### Capabilities This operation offers the ability for the partner code to declare its choices for capabilities listed below: -- Datatype Mappings: Supports adjusting partner data types for each Fivetran data type. +- Datatype Mappings: Provides the option to map destination data types to Fivetran data types. - Max value for columns: Provides an option to specify the maximum value for each data type. #### AlterTable From f4ebaa6f4307fb2a40a222ed18f37c685f8c0156 Mon Sep 17 00:00:00 2001 From: SatvikPatil <54502802+fivetran-satvikpatil@users.noreply.github.com> Date: Tue, 18 Jun 2024 14:26:54 +0530 Subject: [PATCH 17/20] Update development-guide.md Co-authored-by: Emrah Diril --- development-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/development-guide.md b/development-guide.md index b407cc9..89a1a00 100644 --- a/development-guide.md +++ b/development-guide.md @@ -151,7 +151,7 @@ This operation offers the ability for the partner code to declare its choices fo - Max value for columns: Provides an option to specify the maximum value for each data type. #### AlterTable -This operation is used to communicate updates to a table. The `SchemaDiff` message within the `AlterTableRequest` parameter provides the details about these updates: +This operation is used to communicate changes to a table as specific update operations. The `SchemaDiff` message within the `AlterTableRequest` parameter provides the details: - Adding a column (`add_column`): Fivetran uses this field to provide information about a new column to be added in a destination table. - Update Column type (`change_column_type`): This field provides information on updated type of a column in the source that needs to be reflected in a destination table. - Primary key updates (`updated_primary_keys`): If the primary key has changed, this field lists all the columns used in the updated primary key. From 3ef0388a52f478ff8987a5c812319332b8d117e3 Mon Sep 17 00:00:00 2001 From: Satvik Patil Date: Tue, 18 Jun 2024 21:23:59 +0530 Subject: [PATCH 18/20] batch file types --- development-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/development-guide.md b/development-guide.md index 89a1a00..a5bef16 100644 --- a/development-guide.md +++ b/development-guide.md @@ -107,7 +107,7 @@ Batch files are compressed using [ZSTD](https://en.wikipedia.org/wiki/Zstd) ### Batch Files - Each batch file is size limited to 100MB - Number of records in each batch file can vary depending on row size -- Currently we only support CSV file format +- Currently we support CSV and PARQUET file format #### CSV - Fivetran creates batch files using `com.fasterxml.jackson.dataformat.csv.CsvSchema` which by default doesn't consider backslash as escape character. If you are reading the batch file then make sure that you do not consider backslash as escape character. From 6da8440f99d22c7f12c04b74459b564a6ab640ef Mon Sep 17 00:00:00 2001 From: SatvikPatil <54502802+fivetran-satvikpatil@users.noreply.github.com> Date: Mon, 24 Jun 2024 14:25:43 +0530 Subject: [PATCH 19/20] Update development-guide.md Co-authored-by: Abdul Salam <104752966+fivetran-abdulsalam@users.noreply.github.com> --- development-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/development-guide.md b/development-guide.md index a5bef16..4f92f33 100644 --- a/development-guide.md +++ b/development-guide.md @@ -154,7 +154,7 @@ This operation offers the ability for the partner code to declare its choices fo This operation is used to communicate changes to a table as specific update operations. The `SchemaDiff` message within the `AlterTableRequest` parameter provides the details: - Adding a column (`add_column`): Fivetran uses this field to provide information about a new column to be added in a destination table. - Update Column type (`change_column_type`): This field provides information on updated type of a column in the source that needs to be reflected in a destination table. -- Primary key updates (`updated_primary_keys`): If the primary key has changed, this field lists all the columns used in the updated primary key. +- Primary key updates (`update_primary_keys`): If the primary key has changed, this field lists all the columns used in the updated primary key. ### Examples of Data Types From 3aec30cd273a1fb3f2ca7bb794da0b6df10c1f69 Mon Sep 17 00:00:00 2001 From: Satvik Patil Date: Mon, 24 Jun 2024 14:45:00 +0530 Subject: [PATCH 20/20] fix: requested changes --- development-guide.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/development-guide.md b/development-guide.md index 4f92f33..a9bb875 100644 --- a/development-guide.md +++ b/development-guide.md @@ -126,9 +126,9 @@ This operation should report all columns in the destination table, including Fiv #### WriteBatch This operation provides details about the batch files containing the records to be pushed to the destination. We provide the `WriteBatchRequest` parameter that contains all the information required for you to read the batch files. Here are some of the fields included in the request message: -- `replace_files` is for the `upsert` operation where the rows should be inserted if they don't exist or updated if they do. Each row will always provide values for all columns. Set the `_fivetran_synced` column in the destination with the values coming in from the .CSV files. +- `replace_files` is for the `upsert` operation where the rows should be inserted if they don't exist or updated if they do. Each row will always provide values for all columns. Set the `_fivetran_synced` column in the destination with the values coming in from the batch files. -- `update_files` is for the `update` operation where modified columns have actual values whereas unmodified columns have the special value `unmodified_string` in `CsvFileParams`. Soft-deleted rows will arrive in here as well. Update the `_fivetran_synced` column in the destination with the values coming in from the .CSV files. +- `update_files` is for the `update` operation where modified columns have actual values whereas unmodified columns have the special value `unmodified_string` in `FileParams`. Soft-deleted rows will arrive in here as well. Update the `_fivetran_synced` column in the destination with the values coming in from the batch files. - `delete_files` is for `hard delete` operation. Use primary key columns (or `_fivetran_id` system column for primary-keyless tables) to perform `DELETE FROM`.