diff --git a/docs/cloud/managing-airbyte-cloud/review-connection-timeline.md b/docs/cloud/managing-airbyte-cloud/review-connection-timeline.md index 57f338172c76..27b9803d0832 100644 --- a/docs/cloud/managing-airbyte-cloud/review-connection-timeline.md +++ b/docs/cloud/managing-airbyte-cloud/review-connection-timeline.md @@ -8,9 +8,10 @@ The Connection Timeline displays information about several types of relevant con - Sync - Refresh - Clear -- Connector Version changes (coming soon) +- Connection configuration changes - Schema changes (coming soon) -- Connection configuration changes (coming soon) +- Connector Version changes (coming soon) +- Connector configuration changes (coming soon) To review the Connection Timeline, click a connection in the list and navigate to the "Timeline" tab to view its event history. The page displays recent events and their statuses. Where relevant, the event will also show the user who took the action. diff --git a/docs/readme.md b/docs/readme.md index a6577192d924..654aa22074e9 100644 --- a/docs/readme.md +++ b/docs/readme.md @@ -4,38 +4,41 @@ displayed_sidebar: docs # Welcome to Airbyte Docs -## What is Airbyte? + -Airbyte is an open-source data movement infrastructure for building extract and load (EL) data pipelines. It is designed for versatility, scalability, and ease-of-use. + -There are three major components to know in Airbyte: + -1. **The connector catalog** - - **350+ pre-built connectors**: Airbyte’s connector catalog comes “out-of-the-box” with over 350 pre-built connectors. These connectors can be used to start replicating data from a source to a destination in just a few minutes. - - **No-Code Connector Builder**: You can easily extend Airbyte’s functionality to support your custom use cases through tools like the [No-Code Connector Builder](./connector-development/connector-builder-ui/overview). -2. **The platform:** Airbyte’s platform provides all the horizontal services required to configure and scale data movement operations, available as [cloud-managed](https://airbyte.com/product/airbyte-cloud) or [self-managed](https://airbyte.com/product/airbyte-enterprise). -3. **The user interface:** Airbyte features a UI, [**PyAirbyte**](./using-airbyte/pyairbyte/getting-started) (Python library), [**API**](./api-documentation), and [**Terraform Provider**](./terraform-documentation) to integrate with your preferred tooling and approach to infrastructure management. + -Airbyte is suitable for a wide range of data integration use cases, including AI data infrastructure and EL(T) workloads. Airbyte is also [embeddable](https://airbyte.com/product/powered-by-airbyte) within your own application or platform to power your product. +### Why Airbyte? + +Today, teams and organizations require efficient and timely data access to an ever-growing list of data sources. In-house data pipelines are brittle and costly to build and maintain. Airbyte's unique open-source approach enables your data stack to adapt as your data needs evolve. + +- **Wide connector availability:** Airbyte’s connector catalog comes “out-of-the-box” with over 350 pre-built connectors. These connectors can be used to start replicating data from a source to a destination in just a few minutes. +- **Long-tail connector coverage:** You can easily extend Airbyte’s functionality to support your custom use cases through Airbyte's [No-Code Connector Builder](./connector-development/connector-builder-ui/overview). -## Airbyte Demo: Creating Connections +- **Robust platform** provides horizontal scaling required for large-scale data movement operations, available as [Cloud-managed](https://airbyte.com/product/airbyte-cloud) or [Self-managed](https://airbyte.com/product/airbyte-enterprise). - +- **Accessible User Interfaces** through the UI, [**PyAirbyte**](./using-airbyte/pyairbyte/getting-started) (Python library), [**API**](./api-documentation), and [**Terraform Provider**](./terraform-documentation) to integrate with your preferred tooling and approach to infrastructure management. + +Airbyte is suitable for a wide range of data integration use cases, including AI data infrastructure and EL(T) workloads. Airbyte is also [embeddable](https://airbyte.com/product/powered-by-airbyte) within your own application or platform to power your product. -## For Airbyte Cloud users -Browse the [connector catalog](./integrations/) to find the connector you want. In case the connector is not yet supported on Airbyte Cloud, consider using [Airbyte Open Source](#for-airbyte-open-source-users). +## Get Started -Next, check out the [step-by-step tutorial](./using-airbyte/getting-started) to sign up for Airbyte Cloud, understand Airbyte [concepts](./using-airbyte/core-concepts), and run your first sync. + -## For Airbyte Open Source users + -Browse the [connector catalog](./integrations/) to find the connector you want. If the connector is not yet supported on Airbyte Open Source, [build your own connector](./connector-development/). + -Next, check out the [Airbyte Open Source QuickStart](./deploying-airbyte/quickstart). Then learn how to [deploy](./deploying-airbyte/quickstart) and [manage](./operator-guides/upgrading-airbyte) Airbyte Open Source in your cloud infrastructure. + -## For Airbyte contributors + -To contribute to Airbyte code, connectors, and documentation, refer to our [Contributing Guide](./contributing-to-airbyte/). +

[![GitHub stars](https://img.shields.io/github/stars/airbytehq/airbyte?style=social&label=Star&maxAge=2592000)](https://GitHub.com/airbytehq/airbyte/stargazers/) [![License](https://img.shields.io/static/v1?label=license&message=MIT&color=brightgreen)](https://github.com/airbytehq/airbyte/tree/a9b1c6c0420550ad5069aca66c295223e0d05e27/LICENSE/README.md) [![License](https://img.shields.io/static/v1?label=license&message=ELv2&color=brightgreen)](https://github.com/airbytehq/airbyte/tree/a9b1c6c0420550ad5069aca66c295223e0d05e27/LICENSE/README.md) diff --git a/docs/release_notes/aug_2024.md b/docs/release_notes/aug_2024.md index adcdc0ec3f2e..a49e1bb58995 100644 --- a/docs/release_notes/aug_2024.md +++ b/docs/release_notes/aug_2024.md @@ -7,7 +7,7 @@ This page includes new features and improvements to the Airbyte Cloud and Airbyt ## ✨ Highlights Destination S3 ([v1.0.0](https://github.com/airbytehq/airbyte/pull/42409)) was released! Experience faster sync speeds, checkpointing, and modernization to our Destinations V2 framework, along with many bug fixes around Avro and Parquet file formats. -Databricks destination ([v3.1.0[(https://github.com/airbytehq/airbyte/pull/40692)]) was also released with several new features, including typing & deduping, refreshes, and resumable full refresh! We are actively looking for beta testers. Please reach out to us [here](https://github.com/airbytehq/airbyte/discussions/43997) with your feedback. +Databricks destination ([v3.1.0](https://github.com/airbytehq/airbyte/pull/40692)) was also released with several new features, including typing & deduping, refreshes, and resumable full refresh! We are actively looking for beta testers. Please reach out to us [here](https://github.com/airbytehq/airbyte/discussions/43997) with your feedback. ## Platform Releases @@ -29,7 +29,7 @@ We also released a few notable improvements for our connectors: - We now share SBOM for our connectors in our connector registry to enable vulnerability scanning and supply chain attack prevention. -- PyAirbyte now supports writing to destination connectors and the latest version contains significant performance improvements for record processing and Snowflake-based cache loads. . PyAirbyte has been updated to support Python 3.9 in the latest v0.17.0 release. This allows us to move faster by reducing support footprint, and it allows us to keep up with the latest version of the Python CDK. +- PyAirbyte now supports writing to destination connectors and the latest version contains significant performance improvements for record processing and Snowflake-based cache loads. PyAirbyte has been updated to support Python 3.9 in the latest v0.17.0 release. This allows us to move faster by reducing support footprint, and it allows us to keep up with the latest version of the Python CDK. ## Announcements diff --git a/docs/using-airbyte/assets/enabled-streams.png b/docs/using-airbyte/assets/enabled-streams.png deleted file mode 100644 index 18cc4689c756..000000000000 Binary files a/docs/using-airbyte/assets/enabled-streams.png and /dev/null differ diff --git a/docs/using-airbyte/assets/field-selection.png b/docs/using-airbyte/assets/field-selection.png index db9e0c0a30c2..0192f74704d8 100644 Binary files a/docs/using-airbyte/assets/field-selection.png and b/docs/using-airbyte/assets/field-selection.png differ diff --git a/docs/using-airbyte/assets/refresh-modal.png b/docs/using-airbyte/assets/refresh-modal.png index 54ded2698c2f..8db5158b8178 100644 Binary files a/docs/using-airbyte/assets/refresh-modal.png and b/docs/using-airbyte/assets/refresh-modal.png differ diff --git a/docs/using-airbyte/assets/schema-tab-streams.png b/docs/using-airbyte/assets/schema-tab-streams.png new file mode 100644 index 000000000000..9b76b6014897 Binary files /dev/null and b/docs/using-airbyte/assets/schema-tab-streams.png differ diff --git a/docs/using-airbyte/assets/select-sync-mode.png b/docs/using-airbyte/assets/select-sync-mode.png index 6ef1f5a36b4a..907cb2851e62 100644 Binary files a/docs/using-airbyte/assets/select-sync-mode.png and b/docs/using-airbyte/assets/select-sync-mode.png differ diff --git a/docs/using-airbyte/configuring-schema.md b/docs/using-airbyte/configuring-schema.md index 3f06409c7d67..777a8ac6954f 100644 --- a/docs/using-airbyte/configuring-schema.md +++ b/docs/using-airbyte/configuring-schema.md @@ -12,7 +12,7 @@ To learn more about streams and fields, see our [Core Concepts](/using-airbyte/c On the "Schema" tab of a connection, you choose which streams to sync and how they are loaded to the destination. -![Enabled Streams](./assets/enabled-streams.png) +![Enabled Streams](./assets/schema-tab-streams.png) To modify which streams are enabled: @@ -20,10 +20,10 @@ To modify which streams are enabled: 2. Click the **Schema** tab. All the streams Airbyte has discovered from your source will appear. -3. Toggle **Sync** on or off for your selected stream. To select or deselect all streams at once, use the summary toggle in the table header. To select or deselect an individual stream, use the toggle in its row. +3. Toggle the checkbox on or off for your selected stream. To select or deselect all streams at once, use the summary toggle for all the streams in the namespace. Most sources only have one namespace, so the summary toggle will be located in the header. To select or deselect an individual stream, use the toggle in its row. :::tip -Use the toggle "Hide disabled streams" to quickly view all enabled streams or all disabled streams. +Use the tabs to show only enabled streams or all disabled streams. ::: ## Modify Stream Names @@ -33,8 +33,8 @@ By default, Airbyte does not add any text to the **Stream prefix**. Streams with | Source stream name | Stream Prefix | Destination stream name | |--|--|--| -| Accounts | | Accounts | -| Accounts | salesforce_ | salesforce_accounts | +| accounts | | accounts | +| accounts | salesforce_ | salesforce_accounts | Airbyte does not allow for the renaming of individual streams at this time. @@ -58,9 +58,9 @@ Cursors and Primary Keys that are defined by the source are automatically select To modify a cursor or primary key for a stream: -1. Click on a specific stream that is enabled. +1. Click on the toggle for a stream that is enabled to show all it's fields. -2. Select a sync mode specific to the stream. +2. Select the cursor or primary key from the fields displayed. 3. Save your changes. You may need to `Refresh` or `Clear` your data to ensure data continues syncing successfully. @@ -80,13 +80,11 @@ When you refresh the schema, newly added fields will be selected by default, eve To select a partial set of fields: -1. Click on a stream to display the stream details panel. A new pane showing all the fields in the stream will open. You'll see each field we detect from the source. +1. Toggle a stream to display the fields in that stream. The fields will appear nested under the stream name. You'll see each field we detect from the source. -2. Toggle individual fields to include or exclude them in the sync, or use the toggle in the table header to select or deselect all fields at once. +2. Toggle individual fields to include or exclude them in the sync. -5. Click the **X** to close the stream details panel. - -6. Click **Save changes**, or click **Cancel** to discard the changes. +3. Click **Save changes**, or click **Cancel** to discard the changes. 7. The **Stream configuration changed** dialog may display. This gives you the option to `Refresh` the edited streams when you save your changes. If your destination does not support `Refreshes`, you will need to `Clear` your data instead. diff --git a/docs/using-airbyte/getting-started/add-a-source.md b/docs/using-airbyte/getting-started/add-a-source.md index ac60406e1c45..33cb624e8241 100644 --- a/docs/using-airbyte/getting-started/add-a-source.md +++ b/docs/using-airbyte/getting-started/add-a-source.md @@ -6,7 +6,7 @@ products: all Setting up a new source in Airbyte is a quick and simple process! When viewing the Airbyte UI, you'll see the main navigation bar on the left side of your screen. Click the **Sources** tab to bring up a list of all available sources. - + You can use the provided search bar, or simply scroll down the list to find the source you want to replicate data from. Let's use a demo source, Faker, as an example. Clicking on the **Sample Data (Faker)** card will bring us to its setup page. diff --git a/docs/using-airbyte/getting-started/assets/getting-started-column-selection.png b/docs/using-airbyte/getting-started/assets/getting-started-column-selection.png deleted file mode 100644 index 3ba866dbad71..000000000000 Binary files a/docs/using-airbyte/getting-started/assets/getting-started-column-selection.png and /dev/null differ diff --git a/docs/using-airbyte/getting-started/assets/getting-started-destination-catalog.png b/docs/using-airbyte/getting-started/assets/getting-started-destination-catalog.png index 58b99051c65d..0526c636457c 100644 Binary files a/docs/using-airbyte/getting-started/assets/getting-started-destination-catalog.png and b/docs/using-airbyte/getting-started/assets/getting-started-destination-catalog.png differ diff --git a/docs/using-airbyte/getting-started/assets/getting-started-field-selection.png b/docs/using-airbyte/getting-started/assets/getting-started-field-selection.png new file mode 100644 index 000000000000..1d29b02e9cfd Binary files /dev/null and b/docs/using-airbyte/getting-started/assets/getting-started-field-selection.png differ diff --git a/docs/using-airbyte/getting-started/assets/getting-started-select-streams.png b/docs/using-airbyte/getting-started/assets/getting-started-select-streams.png deleted file mode 100644 index f8eed27c0f89..000000000000 Binary files a/docs/using-airbyte/getting-started/assets/getting-started-select-streams.png and /dev/null differ diff --git a/docs/using-airbyte/getting-started/assets/getting-started-stream-selection.png b/docs/using-airbyte/getting-started/assets/getting-started-stream-selection.png new file mode 100644 index 000000000000..74a90a58d0d2 Binary files /dev/null and b/docs/using-airbyte/getting-started/assets/getting-started-stream-selection.png differ diff --git a/docs/using-airbyte/getting-started/readme.md b/docs/using-airbyte/getting-started/readme.md index 26305c31337d..2fdba4fa0bf4 100644 --- a/docs/using-airbyte/getting-started/readme.md +++ b/docs/using-airbyte/getting-started/readme.md @@ -2,42 +2,38 @@ products: all --- +import Tabs from "@theme/Tabs"; +import TabItem from "@theme/TabItem"; + # Airbyte Overview Airbyte is an open-source data integration platform designed to help you consolidate data from various sources into your data warehouses, lakes, and databases. -Whether you're part of a large organization managing complex data pipelines or an individual analyst consolidating data for a personal project, Airbyte can work for you! We offer flexibility and scalability that's easy to tailor to your specific needs, from one-off jobs to enterprise-grade solutions. +Whether you're part of a large organization managing complex data pipelines or an individual analyst consolidating data for a personal project, Airbyte can work for you! We offer flexibility and scalability that's easy to tailor to your specific needs, from one-off jobs to enterprise-grade solutions. + -There are two options to consider to run Airbyte: -- **Airbyte Cloud** (recommended): allows you to focus on moving data while we take care of managing the infrastructure. -- **Self-Managed Airbyte**: can be deployed either locally or in an infrastructure you've set up. Our Self-Managed solution can be used as a free Community product or implemented with support of Enterprise operations. +## Airbyte Products -:::tip -If you have already deployed Airbyte or you've signed up to use Airbyte Cloud, and you're familiar with [Airbyte's core concepts](../../using-airbyte/core-concepts/), jump ahead to [Building Connections](../../cloud/managing-airbyte-cloud/configuring-connections.md). -::: + -## Airbyte Cloud + -To use Airbyte Cloud, [sign up](https://cloud.airbyte.io/signup) with your email address, Google login, or GitHub login. Upon signing up, you'll be taken to your workspace. There, you can collaborate with team members and share resources across your team under a single billing account. + -Airbyte Cloud offers a 14-day free trial that begins after your first successful sync. You can find more details about Airbyte Cloud for practitioners and teams, as well as a tool for evaluating costs on our [pricing page](https://www.airbyte.com/pricing). + -To start setting up a data pipeline, see how to [set up a source](./add-a-source.md). + -## Self-Managed Community (OSS) + -When self-managing Airbyte, your data never leaves your premises. You can use our [OSS Quickstart](oss-quickstart.md) to get started immediately by deploying locally using abctl. + -:::tip -If you're trying Airbyte out for the first time, we'd recommend you sign up for the [Airbyte Cloud trial](https://cloud.airbyte.io/signup) or deploy OSS locally. -::: -To deploy Airbyte in production, Airbyte recommends deploying on [Kubernetes via Helm](../../deploying-airbyte/on-kubernetes-via-helm.md). + -## Self-Managed Enterprise -Airbyte Self-Managed Enterprise is the most robust way to run Airbyte yourself. With this option, you'll have access to all 300+ pre-built connectors and data will never need to leave your environment. Enterprise setup means that Airbyte becomes self-serve in your organization with new tools to manage multiple users, and the ability to set up multiple teams using Airbyte all in one place. +## Additional Resources +- [Airbyte's Core Concepts](../../using-airbyte/core-concepts/) +- [Building Connections](../../cloud/managing-airbyte-cloud/configuring-connections.md) -:::tip -To start with Self-Managed Enterprise, you'll need a license key. [Talk to sales](https://airbyte.com/company/talk-to-sales) to get started with Enterprise. For more details, review our [Enterprise setup guide](/enterprise-setup/README.md). -::: \ No newline at end of file diff --git a/docs/using-airbyte/getting-started/set-up-a-connection.md b/docs/using-airbyte/getting-started/set-up-a-connection.md index e05076a86d31..c4ce0aaf7fe6 100644 --- a/docs/using-airbyte/getting-started/set-up-a-connection.md +++ b/docs/using-airbyte/getting-started/set-up-a-connection.md @@ -17,20 +17,18 @@ Once you've chosen your source and destination you can configure the connection. Most users select "Mirror Source", which will simply copy the data from the source to the destination where you'll see one row in the destination for each row in the source. If you prefer to Append Historical Changes or take a Full Snapshot with each sync, you can optionally select those options, but keep in mind those will create duplicate records in your destination. The sync mode we choose for all the enabled streams will reflect your selection here. - + Next, you can toggle which streams you want to replicate. Our test data consists of three streams, which we've enabled and set to `Incremental - Append + Deduped` sync mode. -![Setup streams](./assets/getting-started-select-streams.png) +![Setup streams](./assets/getting-started-stream-selection.png) Your sync mode is already determined by your selection above, but you can change the sync mode for an individual stream. You can also select a cursor or primary key to enable incremental and/or deduplication. For more information on the nature of each sync mode supported by Airbyte, see [this page](/using-airbyte/core-concepts/sync-modes). -You can also select individual fields to sync on this page. Expand the fields available by clicking any stream. This is helpful when you have security concerns or don't want to sync all the data from the source. -![Column Selection](./assets/getting-started-column-selection.png) +You can also select individual fields to sync on this page. Expand the fields available by expanding all fields or expanding an individual stream. This is helpful when you have security concerns or don't want to sync all the data from the source. +![Column Selection](./assets/getting-started-field-selection.png) -Click **Next** to complete your stream setup and move to the connection configuration. This is where you'll set up how often your data will sync and where it will live in the destination. For this demo, we'll set the connection to run at 8 AM every day and sync the connection to a custom namespace with a stream prefix. - - +Click **Next** to complete your stream setup and move to the connection configuration. This is where you'll set up how often your data will sync and where it will live in the destination. For this demo, we'll set the connection to run every 24 hours. :::note To ensure your data is synced to the correct place, see our examples for [Destination Namespace](/using-airbyte/core-concepts/namespaces.md) @@ -47,7 +45,7 @@ Once you've finished setting up the connection, you will be automatically redire Here's a basic overview of the tabs and their use: 1. The **Status** tab shows you an overview of your connector's sync health. -2. The **Job History** tab allows you to check the logs for each sync. If you encounter any errors or unexpected behaviors during a sync, checking the logs is always a good first step to finding the cause and solution. +2. The **Timeline** tab allows you to track connection events. If you encounter any errors or unexpected behaviors during a sync, checking the errors and related logs is a good first step to finding the cause and solution. 3. The **Schema** tab allows you to modify the streams you chose during the connection setup. 4. The **Transformation** tab allows you to set up a custom post-sync transformations using dbt. 5. The **Settings** tab contains the connection settings, and the option to delete the connection if you no longer wish to use it. diff --git a/docusaurus/sidebars.js b/docusaurus/sidebars.js index dcf0c125aafc..592b6719d037 100644 --- a/docusaurus/sidebars.js +++ b/docusaurus/sidebars.js @@ -518,12 +518,12 @@ module.exports = { }, { type: "doc", - id: "using-airbyte/core-concepts/readme", + label: "Quickstart", + id: "using-airbyte/getting-started/oss-quickstart", }, { type: "doc", - label: "Quickstart", - id: "using-airbyte/getting-started/oss-quickstart", + id: "using-airbyte/core-concepts/readme", }, { type: "category", diff --git a/docusaurus/src/components/Card/Card.jsx b/docusaurus/src/components/Card/Card.jsx index 41e4be42cc6b..3f3757e521f8 100644 --- a/docusaurus/src/components/Card/Card.jsx +++ b/docusaurus/src/components/Card/Card.jsx @@ -32,7 +32,7 @@ const Icon = ({ name }) => { const IconComponent = FA_ICONS[name] || CUSTOM_ICONS[name]; if (name in FA_ICONS) { return ( - + ); } if (name in CUSTOM_ICONS) {