diff --git a/docs/examples/custom_destination_bigquery/custom_destination_bigquery.py b/docs/examples/custom_destination_bigquery/custom_destination_bigquery.py index 64c65dc0b7..48a16f15c0 100644 --- a/docs/examples/custom_destination_bigquery/custom_destination_bigquery.py +++ b/docs/examples/custom_destination_bigquery/custom_destination_bigquery.py @@ -8,7 +8,7 @@ In this example, you'll find a Python script that demonstrates how to load to BigQuery with the custom destination. We'll learn how to: -- Use [built-in credentials.](../general-usage/credentials/prebuilt_types#gcp-credentials) +- Use [built-in credentials.](../general-usage/credentials/complex_types#gcp-credentials) - Use the [custom destination.](../dlt-ecosystem/destinations/destination.md) - Use pyarrow tables to create complex column types on BigQuery. - Use BigQuery `autodetect=True` for schema inference from parquet files. diff --git a/docs/examples/google_sheets/google_sheets.py b/docs/examples/google_sheets/google_sheets.py index d131926c0f..fbc0686fb9 100644 --- a/docs/examples/google_sheets/google_sheets.py +++ b/docs/examples/google_sheets/google_sheets.py @@ -8,8 +8,8 @@ In this example, you'll find a Python script that demonstrates how to load Google Sheets data using the `dlt` library. We'll learn how to: -- use [built-in credentials](../general-usage/credentials/prebuilt_types#gcp-credentials); -- use [union of credentials](../general-usage/credentials/prebuilt_types#working-with-alternatives-of-credentials-union-types); +- use [built-in credentials](../general-usage/credentials/complex_types#gcp-credentials); +- use [union of credentials](../general-usage/credentials/complex_types#working-with-alternatives-of-credentials-union-types); - create [dynamically generated resources](../general-usage/source#create-resources-dynamically). :::tip diff --git a/docs/website/docs/general-usage/credentials/advanced.md b/docs/website/docs/general-usage/credentials/advanced.md index c0d9e3eefc..793f5c2a55 100644 --- a/docs/website/docs/general-usage/credentials/advanced.md +++ b/docs/website/docs/general-usage/credentials/advanced.md @@ -65,7 +65,7 @@ Doing so provides several benefits: 1. You'll never receive the invalid data types in your code. 1. `dlt` will automatically parse and coerce types for you, so you don't need to parse it yourself. 1. `dlt` can generate sample config and secret files for your source automatically. -1. You can request [built-in and custom credentials](prebuilt_types) (i.e., connection strings, AWS / GCP / Azure credentials). +1. You can request [built-in and custom credentials](complex_types) (i.e., connection strings, AWS / GCP / Azure credentials). 1. You can specify a set of possible types via `Union`, i.e., OAuth or API Key authorization. Let's consider the example: @@ -84,7 +84,7 @@ def google_sheets( Now, 1. You are sure that you get a list of strings as `tab_names`. -1. You will get actual Google credentials (see [GCP Credential Configuration](prebuilt_types#gcp-credentials)), and users can +1. You will get actual Google credentials (see [GCP Credential Configuration](complex_types#gcp-credentials)), and users can pass them in many different forms: * `service.json` as a string or dictionary (in code and via config providers). diff --git a/docs/website/docs/general-usage/credentials/prebuilt_types.md b/docs/website/docs/general-usage/credentials/complex_types.md similarity index 99% rename from docs/website/docs/general-usage/credentials/prebuilt_types.md rename to docs/website/docs/general-usage/credentials/complex_types.md index 81c722f16a..24915c1b2e 100644 --- a/docs/website/docs/general-usage/credentials/prebuilt_types.md +++ b/docs/website/docs/general-usage/credentials/complex_types.md @@ -1,6 +1,6 @@ --- -title: Prebuilt credential types -description: How to specify complex custom credentials. +title: Complex credential types +description: Instructions for credentials like DB connection string. keywords: [credentials, secrets.toml, secrets, config, configuration, environment variables, specs] --- diff --git a/docs/website/docs/general-usage/credentials/setup.md b/docs/website/docs/general-usage/credentials/setup.md index e7ff8ca1bb..4ab9149bc0 100644 --- a/docs/website/docs/general-usage/credentials/setup.md +++ b/docs/website/docs/general-usage/credentials/setup.md @@ -6,13 +6,19 @@ keywords: [credentials, secrets.toml, secrets, config, configuration, environmen --- `dlt` automatically extracts configuration settings and secrets based on flexible [naming conventions](setup/#naming-convention). + It then [injects](advanced/#injection-mechanism) these values where needed in functions decorated with `@dlt.source`, `@dlt.resource`, or `@dlt.destination`. +:::note +* **Configuration** refers to non-sensitive settings that define a data pipeline's behavior. These include file paths, database hosts, timeouts, API URLs, and performance settings. +* **Secrets** are sensitive data like passwords, API keys, and private keys. They should never be hard-coded to avoid security risks. +::: + ## Available config providers There are multiple ways to define configurations and credentials for your pipelines. `dlt` looks for these definitions in the following order during pipeline execution: -1. [Environment Variables](#env-variables): If a value for a specific argument is found in an environment variable, dlt will use it and will not proceed to search in lower-priority providers. +1. [Environment Variables](#environment-variables): If a value for a specific argument is found in an environment variable, dlt will use it and will not proceed to search in lower-priority providers. 1. [Vaults](#vaults): Credentials specified in vaults like Google Secrets Manager, Azure Key Vault, AWS Secrets Manager. @@ -80,7 +86,7 @@ project_id = "" ### Credential types -In most cases, credentials are just key-value pairs, but in some cases, the actual structure of [credentials](prebuilt_types) could be quite complex and support several ways of setting it up. +In most cases, credentials are just key-value pairs, but in some cases, the actual structure of [credentials](complex_types) could be quite complex and support several ways of setting it up. For example, to connect to a `sql_database` source, you can either set up a connection string: ```toml @@ -100,13 +106,13 @@ warehouse = "warehouse_name" role = "role" ``` -`dlt` can work with both ways and convert one to another. To learn more about which credential types are supported, visit the [prebuilt credential types](prebuilt_types) page. +`dlt` can work with both ways and convert one to another. To learn more about which credential types are supported, visit the [complex credential types](complex_types) page. -## ENV variables +## Environment variables `dlt` prioritizes security by looking in environment variables before looking into the .toml files. -The format of lookup keys is slightly different from secrets files because for ENV variables, all names are capitalized, and sections are separated with a double underscore `"__"`. For example, to specify the Facebook Ads access token through environment variables, you would need to set up: +The format of lookup keys is slightly different from secrets files because for environment variables, all names are capitalized, and sections are separated with a double underscore `"__"`. For example, to specify the Facebook Ads access token through environment variables, you would need to set up: ```sh export SOURCES__FACEBOOK_ADS__ACCESS_TOKEN="" diff --git a/docs/website/docs/general-usage/destination.md b/docs/website/docs/general-usage/destination.md index e18e502a78..d88a0b53f2 100644 --- a/docs/website/docs/general-usage/destination.md +++ b/docs/website/docs/general-usage/destination.md @@ -72,7 +72,7 @@ You can create and pass partial credentials and `dlt` will fill the missing data -Please read how to use [various built in credentials types](credentials/prebuilt_types). +Please read how to use [various built in credentials types](credentials/complex_types). ::: ### Inspect destination capabilities diff --git a/docs/website/docs/general-usage/glossary.md b/docs/website/docs/general-usage/glossary.md index 8555cc00fa..5ae256b268 100644 --- a/docs/website/docs/general-usage/glossary.md +++ b/docs/website/docs/general-usage/glossary.md @@ -58,6 +58,6 @@ of the data and how to load it into the destination). A set of values that are passed to the pipeline at run time (e.g. to change its behavior locally vs. in production). -## [Credentials](credentials/prebuilt_types) +## [Credentials](credentials/complex_types) A subset of configuration whose elements are kept secret and never shared in plain text. diff --git a/docs/website/docs/walkthroughs/add_credentials.md b/docs/website/docs/walkthroughs/add_credentials.md index 3edf273d13..bc0fb3b409 100644 --- a/docs/website/docs/walkthroughs/add_credentials.md +++ b/docs/website/docs/walkthroughs/add_credentials.md @@ -42,7 +42,7 @@ To add credentials to your deployment, - either use one of the `dlt deploy` commands; - or follow the instructions to [pass credentials via code](../general-usage/credentials/advanced#examples) -or [environment](../general-usage/credentials/setup#env-variables). +or [environment](../general-usage/credentials/setup#environment-variables). ### Reading credentials from environment variables diff --git a/docs/website/sidebars.js b/docs/website/sidebars.js index fad04288d7..7098964b4c 100644 --- a/docs/website/sidebars.js +++ b/docs/website/sidebars.js @@ -146,7 +146,7 @@ const sidebars = { items: [ 'general-usage/credentials/setup', 'general-usage/credentials/advanced', - 'general-usage/credentials/prebuilt_types', + 'general-usage/credentials/complex_types', ] }, 'general-usage/pipeline',