diff --git a/docs/website/docs/dlt-ecosystem/destinations/duckdb.md b/docs/website/docs/dlt-ecosystem/destinations/duckdb.md index f167db6bc6..40bfa8f5ef 100644 --- a/docs/website/docs/dlt-ecosystem/destinations/duckdb.md +++ b/docs/website/docs/dlt-ecosystem/destinations/duckdb.md @@ -84,7 +84,7 @@ p = dlt.pipeline(pipeline_name='chess', destination='duckdb', dataset_name='ches This destination accepts database connection strings in format used by [duckdb-engine](https://github.com/Mause/duckdb_engine#configuration). -You can configure a DuckDB destination with [secret / config values](../../general-usage/configuration/credentials.md) (e.g. using a `secrets.toml` file) +You can configure a DuckDB destination with [secret / config values](../../general-usage/credentials) (e.g. using a `secrets.toml` file) ```toml destination.duckdb.credentials=duckdb:///_storage/test_quack.duckdb ``` diff --git a/docs/website/docs/dlt-ecosystem/verified-sources/airtable.md b/docs/website/docs/dlt-ecosystem/verified-sources/airtable.md index 6597022ba9..6718ff15c2 100644 --- a/docs/website/docs/dlt-ecosystem/verified-sources/airtable.md +++ b/docs/website/docs/dlt-ecosystem/verified-sources/airtable.md @@ -104,7 +104,7 @@ For more information, read the > Optionally, you can also input "base_id" and "table_names" in the script, as in the pipeline > example. -For more information, read the [General Usage: Credentials.](../../general-usage/configuration/credentials.md) +For more information, read the [General Usage: Credentials.](../../general-usage/credentials) ## Run the pipeline diff --git a/docs/website/docs/dlt-ecosystem/verified-sources/asana.md b/docs/website/docs/dlt-ecosystem/verified-sources/asana.md index e2e58361c3..6a66f9c739 100644 --- a/docs/website/docs/dlt-ecosystem/verified-sources/asana.md +++ b/docs/website/docs/dlt-ecosystem/verified-sources/asana.md @@ -89,7 +89,7 @@ For more information, read the [destination documentation](../../dlt-ecosystem/destinations) to add credentials for your chosen destination. This will ensure that your data is properly routed to its final destination. -For more information, read the [General Usage: Credentials.](../../general-usage/configuration/credentials.md) +For more information, read the [General Usage: Credentials.](../../general-usage/credentials) ## Run the pipeline diff --git a/docs/website/docs/dlt-ecosystem/verified-sources/chess.md b/docs/website/docs/dlt-ecosystem/verified-sources/chess.md index 269e7fcb3a..e528f57d87 100644 --- a/docs/website/docs/dlt-ecosystem/verified-sources/chess.md +++ b/docs/website/docs/dlt-ecosystem/verified-sources/chess.md @@ -60,7 +60,7 @@ To add credentials to your destination, follow the instructions in the [destination](../../dlt-ecosystem/destinations) documentation. This will ensure that your data is properly routed to its final destination. -For more information, read the [General Usage: Credentials.](../../general-usage/configuration/credentials.md) +For more information, read the [General Usage: Credentials.](../../general-usage/credentials) ## Run the pipeline diff --git a/docs/website/docs/dlt-ecosystem/verified-sources/facebook_ads.md b/docs/website/docs/dlt-ecosystem/verified-sources/facebook_ads.md index a71c17466f..db3c6e0b81 100644 --- a/docs/website/docs/dlt-ecosystem/verified-sources/facebook_ads.md +++ b/docs/website/docs/dlt-ecosystem/verified-sources/facebook_ads.md @@ -154,7 +154,7 @@ For more information, read the 1. Replace the value of the "account id" with the one [copied above](#grab-account-id). -For more information, read the [General Usage: Credentials.](../../general-usage/configuration/credentials.md) +For more information, read the [General Usage: Credentials.](../../general-usage/credentials) ## Run the pipeline diff --git a/docs/website/docs/dlt-ecosystem/verified-sources/github.md b/docs/website/docs/dlt-ecosystem/verified-sources/github.md index d5d01ffba9..539d6131ae 100644 --- a/docs/website/docs/dlt-ecosystem/verified-sources/github.md +++ b/docs/website/docs/dlt-ecosystem/verified-sources/github.md @@ -105,7 +105,7 @@ For more information, read the add credentials for your chosen destination, ensuring proper routing of your data to the final destination. -For more information, read the [General Usage: Credentials.](../../general-usage/configuration/credentials.md) +For more information, read the [General Usage: Credentials.](../../general-usage/credentials) ## Run the pipeline diff --git a/docs/website/docs/dlt-ecosystem/verified-sources/google_analytics.md b/docs/website/docs/dlt-ecosystem/verified-sources/google_analytics.md index a46d08fbd9..cf2a6c7a4a 100644 --- a/docs/website/docs/dlt-ecosystem/verified-sources/google_analytics.md +++ b/docs/website/docs/dlt-ecosystem/verified-sources/google_analytics.md @@ -209,7 +209,7 @@ For more information, read the 1. To use queries from `.dlt/config.toml`, run the `simple_load_config()` function in [pipeline example](https://github.com/dlt-hub/verified-sources/blob/master/sources/google_analytics_pipeline.py). -For more information, read the [General Usage: Credentials.](../../general-usage/configuration/credentials.md) +For more information, read the [General Usage: Credentials.](../../general-usage/credentials) ## Run the pipeline diff --git a/docs/website/docs/dlt-ecosystem/verified-sources/google_sheets.md b/docs/website/docs/dlt-ecosystem/verified-sources/google_sheets.md index 35cbf8a331..bddfcd3e9e 100644 --- a/docs/website/docs/dlt-ecosystem/verified-sources/google_sheets.md +++ b/docs/website/docs/dlt-ecosystem/verified-sources/google_sheets.md @@ -292,7 +292,7 @@ For more information, read the > Note: You have an option to pass "range_names" and "spreadsheet_identifier" directly to the > google_spreadsheet function or in ".dlt/config.toml" -For more information, read the [General Usage: Credentials.](../../general-usage/configuration/credentials.md) +For more information, read the [General Usage: Credentials.](../../general-usage/credentials) ## Run the pipeline diff --git a/docs/website/docs/dlt-ecosystem/verified-sources/hubspot.md b/docs/website/docs/dlt-ecosystem/verified-sources/hubspot.md index ddf3e92067..c305b5b842 100644 --- a/docs/website/docs/dlt-ecosystem/verified-sources/hubspot.md +++ b/docs/website/docs/dlt-ecosystem/verified-sources/hubspot.md @@ -110,7 +110,7 @@ For more information, read the 1. Enter credentials for your chosen destination as per the [docs](../destinations/). -For more information, read the [General Usage: Credentials.](../../general-usage/configuration/credentials.md) +For more information, read the [General Usage: Credentials.](../../general-usage/credentials) ## Run the pipeline diff --git a/docs/website/docs/dlt-ecosystem/verified-sources/jira.md b/docs/website/docs/dlt-ecosystem/verified-sources/jira.md index a20c150a27..cbc24d2056 100644 --- a/docs/website/docs/dlt-ecosystem/verified-sources/jira.md +++ b/docs/website/docs/dlt-ecosystem/verified-sources/jira.md @@ -97,7 +97,7 @@ For more information, read the add credentials for your chosen destination, ensuring proper routing of your data to the final destination. -For more information, read the [General Usage: Credentials.](../../general-usage/configuration/credentials.md) +For more information, read the [General Usage: Credentials.](../../general-usage/credentials) ## Run the pipeline diff --git a/docs/website/docs/dlt-ecosystem/verified-sources/matomo.md b/docs/website/docs/dlt-ecosystem/verified-sources/matomo.md index a8debbbf80..374ec92ab9 100644 --- a/docs/website/docs/dlt-ecosystem/verified-sources/matomo.md +++ b/docs/website/docs/dlt-ecosystem/verified-sources/matomo.md @@ -98,7 +98,7 @@ For more information, read the 1. To monitor live events on a website, enter the `live_event_site_id` (usually it is same as `site_id`). -For more information, read the [General Usage: Credentials.](../../general-usage/configuration/credentials.md) +For more information, read the [General Usage: Credentials.](../../general-usage/credentials) ## Run the pipeline diff --git a/docs/website/docs/dlt-ecosystem/verified-sources/mongodb.md b/docs/website/docs/dlt-ecosystem/verified-sources/mongodb.md index 83bef0f878..35f042f969 100644 --- a/docs/website/docs/dlt-ecosystem/verified-sources/mongodb.md +++ b/docs/website/docs/dlt-ecosystem/verified-sources/mongodb.md @@ -169,7 +169,7 @@ For more information, read the 1. Replace the value of the "database" and "collections_names" with the ones [copied above](#grab-database-and-collections). -For more information, read the [General Usage: Credentials.](../../general-usage/configuration/credentials.md) +For more information, read the [General Usage: Credentials.](../../general-usage/credentials) ## Run the pipeline diff --git a/docs/website/docs/dlt-ecosystem/verified-sources/mux.md b/docs/website/docs/dlt-ecosystem/verified-sources/mux.md index d900213309..5fb794628e 100644 --- a/docs/website/docs/dlt-ecosystem/verified-sources/mux.md +++ b/docs/website/docs/dlt-ecosystem/verified-sources/mux.md @@ -83,7 +83,7 @@ For more information, read the 1. Finally, enter credentials for your chosen destination as per the [docs](../destinations/). -For more information, read the [General Usage: Credentials.](../../general-usage/configuration/credentials.md) +For more information, read the [General Usage: Credentials.](../../general-usage/credentials) ## Run the pipeline diff --git a/docs/website/docs/dlt-ecosystem/verified-sources/notion.md b/docs/website/docs/dlt-ecosystem/verified-sources/notion.md index 1ee1f73373..358ba8c547 100644 --- a/docs/website/docs/dlt-ecosystem/verified-sources/notion.md +++ b/docs/website/docs/dlt-ecosystem/verified-sources/notion.md @@ -88,7 +88,7 @@ For more information, read the your chosen destination. This will ensure that your data is properly routed to its final destination. -For more information, read the [General Usage: Credentials.](../../general-usage/configuration/credentials.md) +For more information, read the [General Usage: Credentials.](../../general-usage/credentials) ## Run the pipeline diff --git a/docs/website/docs/dlt-ecosystem/verified-sources/pipedrive.md b/docs/website/docs/dlt-ecosystem/verified-sources/pipedrive.md index 2eaefa23ca..a714186c7d 100644 --- a/docs/website/docs/dlt-ecosystem/verified-sources/pipedrive.md +++ b/docs/website/docs/dlt-ecosystem/verified-sources/pipedrive.md @@ -88,7 +88,7 @@ For more information, read the 1. Finally, enter credentials for your chosen destination as per the [docs](../destinations/). -For more information, read the [General Usage: Credentials.](../../general-usage/configuration/credentials.md) +For more information, read the [General Usage: Credentials.](../../general-usage/credentials) ## Run the pipeline @@ -266,7 +266,7 @@ verified source. ```python load_data = pipedrive_source() # calls the source function - load_info = pipeline.run(load_info) #runs the pipeline with selected source configuration + load_info = pipeline.run(load_info) #runs the pipeline with selected source credentials print(load_info) ``` diff --git a/docs/website/docs/dlt-ecosystem/verified-sources/salesforce.md b/docs/website/docs/dlt-ecosystem/verified-sources/salesforce.md index b9b618108e..c819c8120b 100644 --- a/docs/website/docs/dlt-ecosystem/verified-sources/salesforce.md +++ b/docs/website/docs/dlt-ecosystem/verified-sources/salesforce.md @@ -106,7 +106,7 @@ For more information, read the add credentials for your chosen destination, ensuring proper routing of your data to the final destination. -For more information, read the [General Usage: Credentials.](../../general-usage/configuration/credentials.md) +For more information, read the [General Usage: Credentials.](../../general-usage/credentials) ## Run the pipeline diff --git a/docs/website/docs/dlt-ecosystem/verified-sources/shopify.md b/docs/website/docs/dlt-ecosystem/verified-sources/shopify.md index efeb47e71b..ee91cdc0ea 100644 --- a/docs/website/docs/dlt-ecosystem/verified-sources/shopify.md +++ b/docs/website/docs/dlt-ecosystem/verified-sources/shopify.md @@ -98,7 +98,7 @@ For more information, read the add credentials for your chosen destination, ensuring proper routing of your data to the final destination. -For more information, read the [General Usage: Credentials.](../../general-usage/configuration/credentials.md) +For more information, read the [General Usage: Credentials.](../../general-usage/credentials) ## Run the pipeline diff --git a/docs/website/docs/dlt-ecosystem/verified-sources/slack.md b/docs/website/docs/dlt-ecosystem/verified-sources/slack.md index e43f0ffb61..fd25d7818b 100644 --- a/docs/website/docs/dlt-ecosystem/verified-sources/slack.md +++ b/docs/website/docs/dlt-ecosystem/verified-sources/slack.md @@ -101,7 +101,7 @@ For more information, read the 1. Finally, enter credentials for your chosen destination as per the [docs](../destinations/). -For more information, read the [General Usage: Credentials.](../../general-usage/configuration/credentials.md) +For more information, read the [General Usage: Credentials.](../../general-usage/credentials) ## Run the pipeline diff --git a/docs/website/docs/dlt-ecosystem/verified-sources/sql_database.md b/docs/website/docs/dlt-ecosystem/verified-sources/sql_database.md index e7fb2337f3..f219d7c0be 100644 --- a/docs/website/docs/dlt-ecosystem/verified-sources/sql_database.md +++ b/docs/website/docs/dlt-ecosystem/verified-sources/sql_database.md @@ -146,7 +146,7 @@ For more information, read the 1. Finally, follow the instructions in [Destinations](../destinations/) to add credentials for your chosen destination. This will ensure that your data is properly routed to its final destination. -For more information, read the [General Usage: Credentials.](../../general-usage/configuration/credentials.md) +For more information, read the [General Usage: Credentials.](../../general-usage/credentials) ## Run the pipeline diff --git a/docs/website/docs/dlt-ecosystem/verified-sources/strapi.md b/docs/website/docs/dlt-ecosystem/verified-sources/strapi.md index 3947d2784e..7a9161d380 100644 --- a/docs/website/docs/dlt-ecosystem/verified-sources/strapi.md +++ b/docs/website/docs/dlt-ecosystem/verified-sources/strapi.md @@ -90,7 +90,7 @@ For more information, read the 1. Finally, enter credentials for your chosen destination as per the [docs](../destinations/). -For more information, read the [General Usage: Credentials.](../../general-usage/configuration/credentials.md) +For more information, read the [General Usage: Credentials.](../../general-usage/credentials) ## Run the pipeline diff --git a/docs/website/docs/dlt-ecosystem/verified-sources/stripe.md b/docs/website/docs/dlt-ecosystem/verified-sources/stripe.md index b52f06a367..a25768981f 100644 --- a/docs/website/docs/dlt-ecosystem/verified-sources/stripe.md +++ b/docs/website/docs/dlt-ecosystem/verified-sources/stripe.md @@ -90,7 +90,7 @@ For more information, read the 1. Finally, enter credentials for your chosen destination as per the [docs](../destinations/). -For more information, read the [General Usage: Credentials.](../../general-usage/configuration/credentials.md) +For more information, read the [General Usage: Credentials.](../../general-usage/credentials) ## Run the pipeline diff --git a/docs/website/docs/dlt-ecosystem/verified-sources/workable.md b/docs/website/docs/dlt-ecosystem/verified-sources/workable.md index 66ee74ba8b..14530e081e 100644 --- a/docs/website/docs/dlt-ecosystem/verified-sources/workable.md +++ b/docs/website/docs/dlt-ecosystem/verified-sources/workable.md @@ -111,7 +111,7 @@ For more information, read the 1. Finally, enter credentials for your chosen destination as per the [docs](../destinations/). -For more information, read the [General Usage: Credentials.](../../general-usage/configuration/credentials.md) +For more information, read the [General Usage: Credentials.](../../general-usage/credentials) ## Run the pipeline diff --git a/docs/website/docs/dlt-ecosystem/verified-sources/zendesk.md b/docs/website/docs/dlt-ecosystem/verified-sources/zendesk.md index 13dce0b898..17fb366371 100644 --- a/docs/website/docs/dlt-ecosystem/verified-sources/zendesk.md +++ b/docs/website/docs/dlt-ecosystem/verified-sources/zendesk.md @@ -209,7 +209,7 @@ For more information, read the 1. Finally, enter credentials for your chosen destination as per the [docs](../destinations/). -For more information, read the [General Usage: Credentials.](../../general-usage/configuration/credentials.md) +For more information, read the [General Usage: Credentials.](../../general-usage/credentials) ## Run the pipeline diff --git a/docs/website/docs/examples/incremental_loading/index.md b/docs/website/docs/examples/incremental_loading/index.md index 81efd47bb2..629138dc11 100644 --- a/docs/website/docs/examples/incremental_loading/index.md +++ b/docs/website/docs/examples/incremental_loading/index.md @@ -20,7 +20,7 @@ In this example, you'll find a Python script that interacts with the Zendesk Sup We'll learn: -- How to pass [credentials](../../general-usage/configuration/credentials.md) as dict and how to type the `@dlt.source` function arguments. +- How to pass [credentials](../../general-usage/credentials) as dict and how to type the `@dlt.source` function arguments. - How to set [the nesting level](../../general-usage/source#reduce-the-nesting-level-of-generated-tables). - How to enable [incremental loading](../../general-usage/incremental-loading) for efficient data extraction. - How to specify [the start and end dates](../../general-usage/incremental-loading#using-dltsourcesincremental-for-backfill) for the data loading and how to [opt-in to Airflow scheduler](../../general-usage/incremental-loading#using-airflow-schedule-for-backfill-and-incremental-loading) by setting `allow_external_schedulers` to `True`. diff --git a/docs/website/docs/general-usage/configuration/credentials.md b/docs/website/docs/general-usage/credentials/add_credentials.md similarity index 100% rename from docs/website/docs/general-usage/configuration/credentials.md rename to docs/website/docs/general-usage/credentials/add_credentials.md diff --git a/docs/website/docs/general-usage/configuration/config_providers.md b/docs/website/docs/general-usage/credentials/config_providers.md similarity index 100% rename from docs/website/docs/general-usage/configuration/config_providers.md rename to docs/website/docs/general-usage/credentials/config_providers.md diff --git a/docs/website/docs/general-usage/configuration/config_specs.md b/docs/website/docs/general-usage/credentials/config_specs.md similarity index 100% rename from docs/website/docs/general-usage/configuration/config_specs.md rename to docs/website/docs/general-usage/credentials/config_specs.md diff --git a/docs/website/docs/general-usage/configuration/configuration.md b/docs/website/docs/general-usage/credentials/configuration.md similarity index 100% rename from docs/website/docs/general-usage/configuration/configuration.md rename to docs/website/docs/general-usage/credentials/configuration.md diff --git a/docs/website/docs/general-usage/glossary.md b/docs/website/docs/general-usage/glossary.md index 240807e6fa..ff7045d0e4 100644 --- a/docs/website/docs/general-usage/glossary.md +++ b/docs/website/docs/general-usage/glossary.md @@ -53,11 +53,11 @@ Describes the structure of normalized data (e.g. unpacked tables, column types, instructions on how the data should be processed and loaded (i.e. it tells `dlt` about the content of the data and how to load it into the destination). -## [Config](configuration/configuration.md) +## [Config](credentials/configuration.md) A set of values that are passed to the pipeline at run time (e.g. to change its behavior locally vs. in production). -## [Credentials](configuration/credentials.md) +## [Credentials](credentials/add_credentials.md) A subset of configuration whose elements are kept secret and never shared in plain text. diff --git a/docs/website/docs/general-usage/resource.md b/docs/website/docs/general-usage/resource.md index eedd93a12c..d327bb53d6 100644 --- a/docs/website/docs/general-usage/resource.md +++ b/docs/website/docs/general-usage/resource.md @@ -70,7 +70,7 @@ accepts following arguments: > hint value. This let's you create table and column schemas depending on the data. See example in > next section. -> 💡 You can mark some resource arguments as configuration and [credentials](configuration/credentials.md) +> 💡 You can mark some resource arguments as [configuration and credentials](credentials) > values so `dlt` can pass them automatically to your functions. ### Define a schema with Pydantic @@ -174,7 +174,7 @@ for row in generate_rows(20): print(row) ``` -You can mark some resource arguments as configuration and [credentials](configuration/credentials.md) values +You can mark some resource arguments as configuration and [credentials](credentials/add_credentials.md) values so `dlt` can pass them automatically to your functions. ### Process resources with `dlt.transformer` diff --git a/docs/website/docs/general-usage/schema.md b/docs/website/docs/general-usage/schema.md index 30f0a5e285..37cba7277f 100644 --- a/docs/website/docs/general-usage/schema.md +++ b/docs/website/docs/general-usage/schema.md @@ -64,7 +64,7 @@ The default naming convention: > 💡 Use simple, short small caps identifiers for everything! -The naming convention is [configurable](configuration/configuration.md) and users can easily create their own +The naming convention is [configurable](credentials) and users can easily create their own conventions that i.e. pass all the identifiers unchanged if the destination accepts that (i.e. DuckDB). diff --git a/docs/website/docs/getting-started.md b/docs/website/docs/getting-started.md index 97464b690f..cd3f2cc69d 100644 --- a/docs/website/docs/getting-started.md +++ b/docs/website/docs/getting-started.md @@ -637,7 +637,7 @@ If you want to take full advantage of the `dlt` library, then we strongly sugges - [Transform your data before loading](general-usage/resource#customize-resources) and see some [examples of customizations like column renames and anonymization](general-usage/customising-pipelines/renaming_columns). - [Set up "last value" incremental loading](general-usage/incremental-loading#incremental_loading-with-last-value). - [Set primary and merge keys, define the columns nullability and data types](general-usage/resource#define-schema). -- [Pass config and credentials into your sources and resources](general-usage/configuration/credentials.md). +- [Pass config and credentials into your sources and resources](general-usage/credentials). - [Use built-in requests client](reference/performance#using-the-built-in-requests-client). - [Run in production: inspecting, tracing, retry policies and cleaning up](running-in-production/running). - [Run resources in parallel, optimize buffers and local storage](reference/performance.md) diff --git a/docs/website/docs/running-in-production/running.md b/docs/website/docs/running-in-production/running.md index ed441a44f3..4d8cc581d7 100644 --- a/docs/website/docs/running-in-production/running.md +++ b/docs/website/docs/running-in-production/running.md @@ -111,7 +111,7 @@ load.delete_completed_jobs=true ## Using slack to send messages `dlt` provides basic support for sending slack messages. You can configure Slack incoming hook via -[secrets.toml or environment variables](../general-usage/configuration/credentials.md). Please note that **Slack +[secrets.toml or environment variables](../general-usage/credentials/config_providers). Please note that **Slack incoming hook is considered a secret and will be immediately blocked when pushed to github repository**. In `secrets.toml`: diff --git a/docs/website/docs/walkthroughs/add-a-verified-source.md b/docs/website/docs/walkthroughs/add-a-verified-source.md index d565aa3a60..3c2a604ce1 100644 --- a/docs/website/docs/walkthroughs/add-a-verified-source.md +++ b/docs/website/docs/walkthroughs/add-a-verified-source.md @@ -76,7 +76,7 @@ the supported locations. ## 2. Adding credentials For adding them locally or on your orchestrator, please see the following guide -[credentials](../general-usage/configuration/credentials.md). +[credentials](../general-usage/credentials). ## 3. Customize or write a pipeline script diff --git a/docs/website/sidebars.js b/docs/website/sidebars.js index a76c01291e..06fb761d30 100644 --- a/docs/website/sidebars.js +++ b/docs/website/sidebars.js @@ -110,13 +110,13 @@ const sidebars = { link: { type: 'generated-index', title: 'Configuration', - slug: 'dlt-ecosystem/configuration', + slug: 'general-usage/credentials', }, items: [ - 'general-usage/configuration/configuration', - 'general-usage/configuration/credentials', - 'general-usage/configuration/config_providers', - 'general-usage/configuration/config_specs', + 'general-usage/credentials/configuration', + 'general-usage/credentials/add_credentials', + 'general-usage/credentials/config_providers', + 'general-usage/credentials/config_specs', ] }, 'reference/performance',