From 3bc87792e8f4cabc76b25785420be2472362fe4d Mon Sep 17 00:00:00 2001 From: Violetta Mishechkina Date: Wed, 7 Aug 2024 19:21:01 +0200 Subject: [PATCH] Final comments --- .../docs/general-usage/credentials/advanced.md | 4 ++-- .../{prebuilt_types.md => complex_types.md} | 4 ++-- .../docs/general-usage/credentials/setup.md | 16 +++++++++++----- docs/website/docs/general-usage/destination.md | 2 +- docs/website/docs/general-usage/glossary.md | 2 +- .../website/docs/walkthroughs/add_credentials.md | 2 +- docs/website/sidebars.js | 2 +- 7 files changed, 19 insertions(+), 13 deletions(-) rename docs/website/docs/general-usage/credentials/{prebuilt_types.md => complex_types.md} (99%) diff --git a/docs/website/docs/general-usage/credentials/advanced.md b/docs/website/docs/general-usage/credentials/advanced.md index c0d9e3eefc..793f5c2a55 100644 --- a/docs/website/docs/general-usage/credentials/advanced.md +++ b/docs/website/docs/general-usage/credentials/advanced.md @@ -65,7 +65,7 @@ Doing so provides several benefits: 1. You'll never receive the invalid data types in your code. 1. `dlt` will automatically parse and coerce types for you, so you don't need to parse it yourself. 1. `dlt` can generate sample config and secret files for your source automatically. -1. You can request [built-in and custom credentials](prebuilt_types) (i.e., connection strings, AWS / GCP / Azure credentials). +1. You can request [built-in and custom credentials](complex_types) (i.e., connection strings, AWS / GCP / Azure credentials). 1. You can specify a set of possible types via `Union`, i.e., OAuth or API Key authorization. Let's consider the example: @@ -84,7 +84,7 @@ def google_sheets( Now, 1. You are sure that you get a list of strings as `tab_names`. -1. You will get actual Google credentials (see [GCP Credential Configuration](prebuilt_types#gcp-credentials)), and users can +1. You will get actual Google credentials (see [GCP Credential Configuration](complex_types#gcp-credentials)), and users can pass them in many different forms: * `service.json` as a string or dictionary (in code and via config providers). diff --git a/docs/website/docs/general-usage/credentials/prebuilt_types.md b/docs/website/docs/general-usage/credentials/complex_types.md similarity index 99% rename from docs/website/docs/general-usage/credentials/prebuilt_types.md rename to docs/website/docs/general-usage/credentials/complex_types.md index 81c722f16a..24915c1b2e 100644 --- a/docs/website/docs/general-usage/credentials/prebuilt_types.md +++ b/docs/website/docs/general-usage/credentials/complex_types.md @@ -1,6 +1,6 @@ --- -title: Prebuilt credential types -description: How to specify complex custom credentials. +title: Complex credential types +description: Instructions for credentials like DB connection string. keywords: [credentials, secrets.toml, secrets, config, configuration, environment variables, specs] --- diff --git a/docs/website/docs/general-usage/credentials/setup.md b/docs/website/docs/general-usage/credentials/setup.md index e7ff8ca1bb..4ab9149bc0 100644 --- a/docs/website/docs/general-usage/credentials/setup.md +++ b/docs/website/docs/general-usage/credentials/setup.md @@ -6,13 +6,19 @@ keywords: [credentials, secrets.toml, secrets, config, configuration, environmen --- `dlt` automatically extracts configuration settings and secrets based on flexible [naming conventions](setup/#naming-convention). + It then [injects](advanced/#injection-mechanism) these values where needed in functions decorated with `@dlt.source`, `@dlt.resource`, or `@dlt.destination`. +:::note +* **Configuration** refers to non-sensitive settings that define a data pipeline's behavior. These include file paths, database hosts, timeouts, API URLs, and performance settings. +* **Secrets** are sensitive data like passwords, API keys, and private keys. They should never be hard-coded to avoid security risks. +::: + ## Available config providers There are multiple ways to define configurations and credentials for your pipelines. `dlt` looks for these definitions in the following order during pipeline execution: -1. [Environment Variables](#env-variables): If a value for a specific argument is found in an environment variable, dlt will use it and will not proceed to search in lower-priority providers. +1. [Environment Variables](#environment-variables): If a value for a specific argument is found in an environment variable, dlt will use it and will not proceed to search in lower-priority providers. 1. [Vaults](#vaults): Credentials specified in vaults like Google Secrets Manager, Azure Key Vault, AWS Secrets Manager. @@ -80,7 +86,7 @@ project_id = "" ### Credential types -In most cases, credentials are just key-value pairs, but in some cases, the actual structure of [credentials](prebuilt_types) could be quite complex and support several ways of setting it up. +In most cases, credentials are just key-value pairs, but in some cases, the actual structure of [credentials](complex_types) could be quite complex and support several ways of setting it up. For example, to connect to a `sql_database` source, you can either set up a connection string: ```toml @@ -100,13 +106,13 @@ warehouse = "warehouse_name" role = "role" ``` -`dlt` can work with both ways and convert one to another. To learn more about which credential types are supported, visit the [prebuilt credential types](prebuilt_types) page. +`dlt` can work with both ways and convert one to another. To learn more about which credential types are supported, visit the [complex credential types](complex_types) page. -## ENV variables +## Environment variables `dlt` prioritizes security by looking in environment variables before looking into the .toml files. -The format of lookup keys is slightly different from secrets files because for ENV variables, all names are capitalized, and sections are separated with a double underscore `"__"`. For example, to specify the Facebook Ads access token through environment variables, you would need to set up: +The format of lookup keys is slightly different from secrets files because for environment variables, all names are capitalized, and sections are separated with a double underscore `"__"`. For example, to specify the Facebook Ads access token through environment variables, you would need to set up: ```sh export SOURCES__FACEBOOK_ADS__ACCESS_TOKEN="" diff --git a/docs/website/docs/general-usage/destination.md b/docs/website/docs/general-usage/destination.md index e18e502a78..d88a0b53f2 100644 --- a/docs/website/docs/general-usage/destination.md +++ b/docs/website/docs/general-usage/destination.md @@ -72,7 +72,7 @@ You can create and pass partial credentials and `dlt` will fill the missing data -Please read how to use [various built in credentials types](credentials/prebuilt_types). +Please read how to use [various built in credentials types](credentials/complex_types). ::: ### Inspect destination capabilities diff --git a/docs/website/docs/general-usage/glossary.md b/docs/website/docs/general-usage/glossary.md index 8555cc00fa..5ae256b268 100644 --- a/docs/website/docs/general-usage/glossary.md +++ b/docs/website/docs/general-usage/glossary.md @@ -58,6 +58,6 @@ of the data and how to load it into the destination). A set of values that are passed to the pipeline at run time (e.g. to change its behavior locally vs. in production). -## [Credentials](credentials/prebuilt_types) +## [Credentials](credentials/complex_types) A subset of configuration whose elements are kept secret and never shared in plain text. diff --git a/docs/website/docs/walkthroughs/add_credentials.md b/docs/website/docs/walkthroughs/add_credentials.md index 3edf273d13..bc0fb3b409 100644 --- a/docs/website/docs/walkthroughs/add_credentials.md +++ b/docs/website/docs/walkthroughs/add_credentials.md @@ -42,7 +42,7 @@ To add credentials to your deployment, - either use one of the `dlt deploy` commands; - or follow the instructions to [pass credentials via code](../general-usage/credentials/advanced#examples) -or [environment](../general-usage/credentials/setup#env-variables). +or [environment](../general-usage/credentials/setup#environment-variables). ### Reading credentials from environment variables diff --git a/docs/website/sidebars.js b/docs/website/sidebars.js index fad04288d7..7098964b4c 100644 --- a/docs/website/sidebars.js +++ b/docs/website/sidebars.js @@ -146,7 +146,7 @@ const sidebars = { items: [ 'general-usage/credentials/setup', 'general-usage/credentials/advanced', - 'general-usage/credentials/prebuilt_types', + 'general-usage/credentials/complex_types', ] }, 'general-usage/pipeline',