diff --git a/docs/integrations/sources/harvest-migrations.md b/docs/integrations/sources/harvest-migrations.md index ee28f959bc1a..6fe39fb7f12f 100644 --- a/docs/integrations/sources/harvest-migrations.md +++ b/docs/integrations/sources/harvest-migrations.md @@ -2,7 +2,7 @@ ## Upgrading to 1.0.0 -This update results in a change the following streams, requiring reset: +This update results in a change the following streams, requiring them to be cleared and completely synced again: - `expenses_clients` - `expenses_categories` - `expenses_projects` @@ -17,15 +17,15 @@ This update results in a change the following streams, requiring reset: - `invoice_messages` - `project_assignments` -We're continuously striving to enhance the quality and reliability of our connectors at Airbyte. As part of our commitment to delivering exceptional service, we are transitioning Source Harvest from the Python Connector Development Kit (CDK) to our new low-code framework improving maintainability and reliability of the connector. However, due to differences between the Python and low-code CDKs, this migration constitutes a breaking change. +We're continuously striving to enhance the quality and reliability of our connectors at Airbyte. As part of our commitment to delivering exceptional service, we are transitioning the source Harvest from the Python Connector Development Kit (CDK) to our new low-code framework to improve maintainability and reliability of the connector. However, due to differences between the Python and low-code CDKs, this migration constitutes a breaking change. -## Steps to Reset Streams +## Steps to Clear Streams -To reset your data for the impacted streams, follow the steps below: +To clear your data for the impacted streams, follow the steps below: 1. Select **Connections** in the main nav bar. 1. Select the connection(s) affected by the update. 2. Select the **Status** tab. - 1. In the **Enabled streams** list, click the three dots on the right side of the stream requiring reset and select **Reset this stream**. + 1. In the **Enabled streams** list, click the three dots on the right side of the stream and select **Clear Data**. -A fresh sync will run for the stream. For more information on resetting your data in Airbyte, see [this page](https://docs.airbyte.com/operator-guides/reset). +After the clear succeeds, trigger a sync by clicking **Sync Now**. For more information on clearing your data in Airbyte, see [this page](https://docs.airbyte.com/operator-guides/reset). diff --git a/docs/integrations/sources/linkedin-ads-migrations.md b/docs/integrations/sources/linkedin-ads-migrations.md index 6816e08cfde4..f130a972c02e 100644 --- a/docs/integrations/sources/linkedin-ads-migrations.md +++ b/docs/integrations/sources/linkedin-ads-migrations.md @@ -2,7 +2,8 @@ ## Upgrading to 1.0.0 -Version 1.0.0 introduces changes in primary key for all *-analytics streams (including custom ones). +Version 1.0.0 introduces changes in the primary key selected for all *-analytics streams (including custom ones). + - "ad_campaign_analytics" - "ad_creative_analytics" - "ad_impression_device_analytics" @@ -17,24 +18,14 @@ Version 1.0.0 introduces changes in primary key for all *-analytics streams (inc ## Migration Steps -### Refresh affected schemas and reset data +Clearing your data is required for the affected streams in order to continue syncing successfully. To clear your data for the affected streams, follow the steps below: -1. Select **Connections** in the main nav bar. - 1. Select the connection(s) affected by the update. -2. Select the **Replication** tab. - 1. Select **Refresh source schema**. - 2. Select **OK**. -:::note -Any detected schema changes will be listed for your review. -::: -3. Select **Save changes** at the bottom of the page. - 1. Ensure the **Reset affected streams** option is checked. -:::note -Depending on destination type you may not be prompted to reset your data. -::: +1. Select **Connections** in the main navbar and select the connection(s) affected by the update. +2. Select the **Schema** tab. + 1. Select **Refresh source schema** to bring in any schema changes. Any detected schema changes will be listed for your review. + 2. Select **OK** to approve changes. +3. Select **Save changes** at the bottom of the page. + 1. Ensure the **Clear affected streams** option is checked to ensure your streams continue syncing successfully with the new schema. 4. Select **Save connection**. -:::note -This will reset the data in your destination and initiate a fresh sync. -::: -For more information on resetting your data in Airbyte, see [this page](https://docs.airbyte.com/operator-guides/reset). +This will clear the data in your destination for the subset of streams with schema changes. After the clear succeeds, trigger a sync by clicking **Sync Now**. For more information on clearing your data in Airbyte, see [this page](https://docs.airbyte.com/operator-guides/reset). \ No newline at end of file diff --git a/docs/integrations/sources/notion-migrations.md b/docs/integrations/sources/notion-migrations.md index ae1e8414a4c5..0f77fa22d132 100644 --- a/docs/integrations/sources/notion-migrations.md +++ b/docs/integrations/sources/notion-migrations.md @@ -4,20 +4,20 @@ We're continuously striving to enhance the quality and reliability of our connectors at Airbyte. As part of our commitment to delivering exceptional service, we are transitioning source Notion from the Python Connector Development Kit (CDK) to our innovative low-code framework. This is part of a strategic move to streamline many processes across connectors, bolstering maintainability and freeing us to focus more of our efforts on improving the performance and features of our evolving platform and growing catalog. However, due to differences between the Python and low-code CDKs, this migration constitutes a breaking change for users syncing data from the `Comments` stream. -Specifically, we’ve evolved and standardized how state is managed for incremental streams that are nested within a parent stream. This change impacts how individual states are tracked and stored for each partition, using a more structured approach to ensure the most granular and flexible state management. To gracefully handle these changes for your existing connections, we highly recommend resetting your data for the `Comments` stream before resuming your syncs with the new version. +Specifically, we’ve evolved and standardized how state is managed for incremental streams that are nested within a parent stream. This change impacts how individual states are tracked and stored for each partition, using a more structured approach to ensure the most granular and flexible state management. To gracefully handle these changes for your existing connections, we highly recommend clearing your data for the `Comments` stream before resuming your syncs with the new version. If you are not syncing data from the `Comments` stream, this change is non-breaking, and no further action is required. -### Resetting your `Comments` data +### Migration Steps -To reset your data for the `Comments` stream, follow the steps below: +Data for the `Comments` stream will need to cleared to ensure your syncs continue successfully. To clear your data for the `Comments` stream, follow the steps below: 1. Select **Connections** in the main nav bar. 1. Select the connection(s) affected by the update. 2. Select the **Status** tab. - 1. In the **Enabled streams** list, click the three dots on the right side of the **Comments** stream and select **Reset this stream**. + 1. In the **Enabled streams** list, click the three dots on the right side of the **Comments** stream and select **Clear data**. -A fresh sync will run for the `Comments` stream. For more information on resetting your data in Airbyte, see [this page](https://docs.airbyte.com/operator-guides/reset). +After the clear succeeds, trigger a sync for the `Comments` stream by clicking "Sync Now". For more information on clearing your data in Airbyte, see [this page](https://docs.airbyte.com/operator-guides/reset). ## Upgrading to 2.0.0 diff --git a/docs/integrations/sources/rss-migrations.md b/docs/integrations/sources/rss-migrations.md index 494617dbf903..212b7c3c1cee 100644 --- a/docs/integrations/sources/rss-migrations.md +++ b/docs/integrations/sources/rss-migrations.md @@ -1,4 +1,21 @@ -# Rss Migration Guide +# RSS Migration Guide ## Upgrading to 1.0.0 -The verison migrates the Rss connector to the low-code framework for greater maintainability. You may need to refresh the connection schema (with the reset), and run a sync. \ No newline at end of file +We're continuously striving to enhance the quality and reliability of our connectors at Airbyte. + +As part of our commitment to delivering exceptional service, we are transitioning our RSS source from the Python Connector Development Kit (CDK) +to our new low-code framework improving maintainability and reliability of the connector. Due to differences between the Python and low-code CDKs, this migration constitutes a breaking change. + +## Migration Steps + +Clearing your data is required for the affected streams in order to continue syncing successfully. To clear your data for the affected streams, follow the steps below: + +1. Select **Connections** in the main navbar and select the connection(s) affected by the update. +2. Select the **Schema** tab. + 1. Select **Refresh source schema** to bring in any schema changes. Any detected schema changes will be listed for your review. + 2. Select **OK** to approve changes. +3. Select **Save changes** at the bottom of the page. + 1. Ensure the **Clear affected streams** option is checked to ensure your streams continue syncing successfully with the new schema. +4. Select **Save connection**. + +This will clear the data in your destination for the subset of streams with schema changes. After the clear succeeds, trigger a sync by clicking **Sync Now**. For more information on clearing your data in Airbyte, see [this page](https://docs.airbyte.com/operator-guides/reset). \ No newline at end of file diff --git a/docs/integrations/sources/sendgrid-migrations.md b/docs/integrations/sources/sendgrid-migrations.md index 48080d1198c7..e06eac156981 100644 --- a/docs/integrations/sources/sendgrid-migrations.md +++ b/docs/integrations/sources/sendgrid-migrations.md @@ -3,18 +3,18 @@ ## Upgrading to 1.0.0 We're continuously striving to enhance the quality and reliability of our connectors at Airbyte. + As part of our commitment to delivering exceptional service, we are transitioning Source Sendgrid from the Python Connector Development Kit (CDK) -to our new low-code framework improving maintainability and reliability of the connector. -However, due to differences between the Python and low-code CDKs, this migration constitutes a breaking change. +to our new low-code framework improving maintainability and reliability of the connector. Due to differences between the Python and low-code CDKs, this migration constitutes a breaking change. * The configuration options have been renamed to `api_key` and `start_date`. -* The `unsubscribe_groups` stream has been removed. It was the same as `suppression_groups`. You can use that and get the same data. -* The `single_sends` stream has been renamed `singlesend_stats`. This is closer to the data and API. -* The `segments` stream has been upgraded to use the Sendgrid 2.0 API because the older one has been deprecated. The schema has changed as a result. +* The `unsubscribe_groups` stream has been removed as it was a duplicate of `suppression_groups`. You can use `suppression_groups` and get the same data you were previously receiving in `unsubscribe_groups`. +* The `single_sends` stream has been renamed to `singlesend_stats`. This was done to more closely match the data from the Sendgrid API. +* The `segments` stream has been upgraded to use the Sendgrid 2.0 API as the previous version of the API has been deprecated. As a result, fields within the stream have changed to reflect the new API. -To ensure a smooth upgrade, please refresh your schemas and reset your data before resuming syncs. +To ensure a smooth upgrade, please clear your streams and trigger a sync to bring in historical data. -## Connector Upgrade Guide +## Migration Steps ### For Airbyte Open Source: Update the local connector image @@ -42,46 +42,16 @@ Each instance of the connector must be updated separately. If you have created m 3. Select **Upgrade** 1. Follow the prompt to confirm you are ready to upgrade to the new version. +### For Airbyte Cloud and Open Source: Steps to Update Schema and Clear Streams -### Refresh all schemas and reset data - -1. Select **Connections** in the main navbar. -2. Select the connection(s) affected by the update. -3. Select the **Replication** tab. - 1. Select **Refresh source schema**. - 2. Select **OK**. -:::note -Any detected schema changes will be listed for your review. -::: -4. Select **Save changes** at the bottom of the page. - 1. Ensure the **Reset all streams** option is checked. -5. Select **Save connection**. -:::note -This will reset the data in your destination and initiate a fresh sync. -::: - -For more information on resetting your data in Airbyte, see [this page](https://docs.airbyte.com/operator-guides/reset). - +To clear your data for the affected streams, follow the steps below: -### Refresh affected schemas and reset data - -1. Select **Connections** in the main navbar. - 1. Select the connection(s) affected by the update. -2. Select the **Replication** tab. - 1. Select **Refresh source schema**. - 2. Select **OK**. -:::note -Any detected schema changes will be listed for your review. -::: -3. Select **Save changes** at the bottom of the page. - 1. Ensure the **Reset affected streams** option is checked. -:::note -Depending on destination type you may not be prompted to reset your data. -::: +1. Select **Connections** in the main navbar and select the connection(s) affected by the update. +2. Select the **Schema** tab. + 1. Select **Refresh source schema** to bring in any schema changes. Any detected schema changes will be listed for your review. + 2. Select **OK** to approve changes. +3. Select **Save changes** at the bottom of the page. + 1. Ensure the **Clear affected streams** option is checked to ensure your streams continue syncing successfully with the new schema. 4. Select **Save connection**. -:::note -This will reset the data in your destination and initiate a fresh sync. -::: - -For more information on resetting your data in Airbyte, see [this page](https://docs.airbyte.com/operator-guides/reset). +This will clear the data in your destination for the subset of streams with schema changes. After the clear succeeds, trigger a sync by clicking **Sync Now**. For more information on clearing your data in Airbyte, see [this page](https://docs.airbyte.com/operator-guides/reset). \ No newline at end of file diff --git a/docs/integrations/sources/sftp-bulk-migrations.md b/docs/integrations/sources/sftp-bulk-migrations.md index 2a6d0a83f499..7178311368e3 100644 --- a/docs/integrations/sources/sftp-bulk-migrations.md +++ b/docs/integrations/sources/sftp-bulk-migrations.md @@ -1,36 +1,25 @@ # SFTP Bulk Migration Guide ## Upgrading to 1.0.0 -This release upgrades the SFTP Bulk connector to file-based CDK which causes the following changes: +We're continuously striving to enhance the quality and reliability of our connectors at Airbyte. -- Configuration changes -- Stream changes +As part of our commitment to delivering exceptional service, we are transitioning our SFTP Bulk source from the Python Connector Development Kit (CDK) to our new low-code framework improving maintainability and reliability of the connector. Due to differences between the Python and low-code CDKs, this migration constitutes a breaking change for the following: -Users should: +- Changes to the source configuration +- Changes to the schema (added, removed fields and/or streams) -- Reconfigure the source -- Refresh the source schema -- Reset affected streams after upgrading to ensure uninterrupted syncs. +## Migration Steps +This version change requires you to re-verify the configuration of your source. To do this, click on "Source" in the left-hand sidebar and navigate to your SFTP Bulk source. Test the source and ensure the configuration is valid before moving forward. -### Refresh affected schemas and reset data +Clearing your data is required for the affected streams in order to continue syncing successfully. To clear your data for the affected streams, follow the steps below: -1. Select **Connections** in the main navbar. - 1. Select the connection(s) affected by the update. -2. Select the **Replication** tab. - 1. Select **Refresh source schema**. - 2. Select **OK**. -```note -Any detected schema changes will be listed for your review. -``` -3. Select **Save changes** at the bottom of the page. - 1. Ensure the **Reset affected streams** option is checked. -```note -Depending on destination type you may not be prompted to reset your data. -``` +1. Select **Connections** in the main navbar and select the connection(s) affected by the update. +2. Select the **Schema** tab. + 1. Select **Refresh source schema** to bring in any schema changes. Any detected schema changes will be listed for your review. + 2. Select **OK** to approve changes. +3. Select **Save changes** at the bottom of the page. + 1. Ensure the **Clear affected streams** option is checked to ensure your streams continue syncing successfully with the new schema. 4. Select **Save connection**. -```note -This will reset the data in your destination and initiate a fresh sync. -``` -For more information on resetting your data in Airbyte, see [this page](https://docs.airbyte.com/operator-guides/reset). \ No newline at end of file +This will clear the data in your destination for the subset of streams with schema changes. After the clear succeeds, trigger a sync by clicking **Sync Now**. For more information on clearing your data in Airbyte, see [this page](https://docs.airbyte.com/operator-guides/reset). \ No newline at end of file diff --git a/docs/integrations/sources/slack-migrations.md b/docs/integrations/sources/slack-migrations.md index 31458bc54c1c..e2d0fb2129d1 100644 --- a/docs/integrations/sources/slack-migrations.md +++ b/docs/integrations/sources/slack-migrations.md @@ -2,17 +2,18 @@ ## Upgrading to 1.0.0 -We're continuously striving to enhance the quality and reliability of our connectors at Airbyte. -As part of our commitment to delivering exceptional service, we are transitioning source Slack from the -Python Connector Development Kit (CDK) to our innovative low-code framework. -This is part of a strategic move to streamline many processes across connectors, bolstering maintainability and -freeing us to focus more of our efforts on improving the performance and features of our evolving platform and growing catalog. -However, due to differences between the Python and low-code CDKs, this migration constitutes a breaking change. - -We’ve evolved and standardized how state is managed for incremental streams that are nested within a parent stream. -This change impacts how individual states are tracked and stored for each partition, using a more structured approach -to ensure the most granular and flexible state management. -This change will affect the `Channel Messages` stream. +We're continuously striving to enhance the quality and reliability of our connectors at Airbyte. As part of our commitment to delivering exceptional service, we are transitioning source Slack from the Python Connector Development Kit (CDK) to our innovative low-code framework. This is part of a strategic move to streamline many processes across connectors, bolstering maintainability and freeing us to focus more of our efforts on improving the performance and features of our evolving platform and growing catalog. However, due to differences between the Python and low-code CDKs, this migration constitutes a breaking change. + +We’ve evolved and standardized how state is managed for incremental streams that are nested within a parent stream. This change impacts how individual states are tracked and stored for each partition, using a more structured approach to ensure the most granular and flexible state management. This change will affect the `Channel Messages` stream. ## Migration Steps -* A `reset` for `Channel Messages` stream is required after upgrading to this version. + +Clearing your data is required in order to continue syncing `Channel Messages` successfully. To clear your data for the `Channel Messages` stream, follow the steps below: + +1. Select **Connections** in the main nav bar. + 1. Select the connection(s) affected by the update. +2. Select the **Status** tab. + 1. In the **Enabled streams** list, click the three dots on the right side of the `Channel Messages` and select **Clear Data**. + +After the clear succeeds, trigger a sync by clicking **Sync Now**. For more information on clearing your data in Airbyte, see [this page](https://docs.airbyte.com/operator-guides/reset). + diff --git a/docs/operator-guides/reset.md b/docs/operator-guides/reset.md index e68bbbc877be..565c43571317 100644 --- a/docs/operator-guides/reset.md +++ b/docs/operator-guides/reset.md @@ -2,30 +2,38 @@ products: all --- -# Resetting your data +# Clearing your data -Resetting your data allows you to drop all previously synced data so that any ensuing sync can start syncing fresh. This is useful if you don't require the data replicated to your destination to be saved permanently or are just testing Airbyte. +From time-to-time, you may want to erase all of the data that Airbyte has created in your destination. This can be accomplished by clearing your data. In order to backfill all historical data, a sync should be initiated after your clear succeeds. -Airbyte allows you to reset all streams in the connection, some, or only a single stream (when the connector support per-stream operations). +Note that there is no way to recover from a clear sync, so please be certain that you wish to erase all the data in your destination. -A sync will automatically start after a completed reset, which commonly backfills all historical data. +:::warning +Not all sources keep their history forever. If you clear your data, and your source does not retain all of its records, this may lead to data loss. +::: + +A Clear can be triggered either from the UI or Airbyte API. Airbyte allows you to clear all streams in the connection or only a single stream through the UI. You may also be prompted to clear some streams when making configuration changes that apply to multiple streams. Airbyte additionally supports the clearing of multiple streams through the API. + +## Steps to Clear Data +To perform a full removal of the data for all your streams, navigate to a connection's `Settings` tab and click "Clear data". Confirm the selection to remove all previously synced data from the destination for that connection. + +To clear data for a single stream, navigate to a Connection's status page, click the three grey dots next to any stream, and select "Clear data". This will clear the data for just that stream. You will then need to sync the connection again in order to reload data for that stream. -## Performing a Reset -To perform a full reset that resets all your streams, select `Reset your data` in the UI on a connection's status or job history tabs by selecting the three grey dots next to "Sync now". +You will also automatically be prompted to clear affected streams if you edit any stream settings or approve any non-breaking schema changes. To ensure data continues to sync accurately, Airbyte recommends doing a clear of those streams as your streams could sync incorrectly if a clear is not performed. -To reset a single stream, navigate to a Connection's status page, click the three grey dots next to any stream, and select "Reset this stream". This will perform a reset of only that stream. You will then need to sync the connection again in order to reload data for that stream. +Similarly to a sync, a clear can be completed as successful, failed, or cancelled. To resolve a failed clearing of data, you should manually drop the tables in the destination so that Airbyte can continue syncing accurately into the destination. + +In order to backfill all historical data, a sync should be initiated after your clear succeeds. :::note -A single stream reset will sync all enabled streams on the next sync. +A single stream clear will sync all enabled streams on the next sync. ::: -You will also automatically be prompted to reset affected streams if you edit any stream settings or approve any non-breaking schema changes. To ensure data continues to sync accurately, Airbyte recommends doing a reset of those streams as your streams could sync incorrectly if a reset is not performed. - -Similarly to a sync job, a reset can be completed as successful, failed, or cancelled. To resolve a failed reset, you should manually drop the tables in the destination so that Airbyte can continue syncing accurately into the destination. +## Clear behavior +When clearing data is successfully completed, all the records are deleted from your destination tables (and files, if using local JSON or local CSV as the destination). The tables or files are not removed, they will only be emptied. -## Reset behavior -When a reset is successfully completed, all the records are deleted from your destination tables (and files, if using local JSON or local CSV as the destination), and then the next sync will begin. +Clearing your data causes data downtime, meaning that your final tables will be empty once the Clear is complete. Clearing your data also blocks the running of regularly-scheduled syncs until they are complete. If you choose to clear your data while another sync is running, it will enqueue, and start at the end of the currently running sync. :::tip If you have any orphaned tables or files that are no longer being synced to, they should be cleaned up separately, as Airbyte will not clean them up for you. This can occur when the `Destination Namespace` or `Stream Prefix` connection configuration is changed for an existing connection. -::: +::: \ No newline at end of file diff --git a/docs/release_notes/upgrading_to_destinations_v2.md b/docs/release_notes/upgrading_to_destinations_v2.md index 0d6e9407df05..77fe4b878a5b 100644 --- a/docs/release_notes/upgrading_to_destinations_v2.md +++ b/docs/release_notes/upgrading_to_destinations_v2.md @@ -182,7 +182,7 @@ Note that legacy normalization will be deprecated for ClickHouse, DuckDB, MSSQL, If you upgrade to Destinations V2 and start encountering issues, as an Open Source user you can optionally roll back. If you are running an outdated Airbyte Platform version (prior to `v0.50.24`), this may occur more frequently by accidentally upgrading to Destinations V2. However: -- Rolling back will require resetting each of your upgraded connections. +- Rolling back will require clearing each of your upgraded connections. - If you are hoping to receive support from the Airbyte team, you will need to re-upgrade to Destinations V2 by the upgrade deadline. To roll back, follow these steps: @@ -190,9 +190,9 @@ To roll back, follow these steps: 1. In the Airbyte UI, go to the 'Settings page, then to 'Destinations'. 2. Manually type in the previous destination version you were running, or one of the versions listed in the table above. 3. Enter this older version to roll back to the previous connector version. -4. Reset all connections which synced at least once to a previously upgraded destination. To be safe, you may reset all connections sending data to a previously upgraded destination. +4. Clear all connections which synced at least once to a previously upgraded destination. To be safe, you may clear all connections sending data to a previously upgraded destination. -If you are an Airbyte Cloud customer, and encounter errors while upgrading from a V1 to a V2 destination, please reach out to support. We do not always recommend doing a full reset, depending on the type of error. +If you are an Airbyte Cloud customer, and encounter errors while upgrading from a V1 to a V2 destination, please reach out to support. We do not always recommend doing a full clear of your entire connection, depending on the type of error. ## Destinations V2 Implementation Differences