Skip to content

Commit

Permalink
Final comments
Browse files Browse the repository at this point in the history
  • Loading branch information
VioletM committed Aug 7, 2024
1 parent 3701655 commit 3bc8779
Show file tree
Hide file tree
Showing 7 changed files with 19 additions and 13 deletions.
4 changes: 2 additions & 2 deletions docs/website/docs/general-usage/credentials/advanced.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ Doing so provides several benefits:
1. You'll never receive the invalid data types in your code.
1. `dlt` will automatically parse and coerce types for you, so you don't need to parse it yourself.
1. `dlt` can generate sample config and secret files for your source automatically.
1. You can request [built-in and custom credentials](prebuilt_types) (i.e., connection strings, AWS / GCP / Azure credentials).
1. You can request [built-in and custom credentials](complex_types) (i.e., connection strings, AWS / GCP / Azure credentials).
1. You can specify a set of possible types via `Union`, i.e., OAuth or API Key authorization.

Let's consider the example:
Expand All @@ -84,7 +84,7 @@ def google_sheets(
Now,

1. You are sure that you get a list of strings as `tab_names`.
1. You will get actual Google credentials (see [GCP Credential Configuration](prebuilt_types#gcp-credentials)), and users can
1. You will get actual Google credentials (see [GCP Credential Configuration](complex_types#gcp-credentials)), and users can
pass them in many different forms:

* `service.json` as a string or dictionary (in code and via config providers).
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Prebuilt credential types
description: How to specify complex custom credentials.
title: Complex credential types
description: Instructions for credentials like DB connection string.
keywords: [credentials, secrets.toml, secrets, config, configuration, environment
variables, specs]
---
Expand Down
16 changes: 11 additions & 5 deletions docs/website/docs/general-usage/credentials/setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,19 @@ keywords: [credentials, secrets.toml, secrets, config, configuration, environmen
---

`dlt` automatically extracts configuration settings and secrets based on flexible [naming conventions](setup/#naming-convention).

It then [injects](advanced/#injection-mechanism) these values where needed in functions decorated with `@dlt.source`, `@dlt.resource`, or `@dlt.destination`.

:::note
* **Configuration** refers to non-sensitive settings that define a data pipeline's behavior. These include file paths, database hosts, timeouts, API URLs, and performance settings.
* **Secrets** are sensitive data like passwords, API keys, and private keys. They should never be hard-coded to avoid security risks.
:::

## Available config providers

There are multiple ways to define configurations and credentials for your pipelines. `dlt` looks for these definitions in the following order during pipeline execution:

1. [Environment Variables](#env-variables): If a value for a specific argument is found in an environment variable, dlt will use it and will not proceed to search in lower-priority providers.
1. [Environment Variables](#environment-variables): If a value for a specific argument is found in an environment variable, dlt will use it and will not proceed to search in lower-priority providers.

1. [Vaults](#vaults): Credentials specified in vaults like Google Secrets Manager, Azure Key Vault, AWS Secrets Manager.

Expand Down Expand Up @@ -80,7 +86,7 @@ project_id = "<project_id_2>"

### Credential types

In most cases, credentials are just key-value pairs, but in some cases, the actual structure of [credentials](prebuilt_types) could be quite complex and support several ways of setting it up.
In most cases, credentials are just key-value pairs, but in some cases, the actual structure of [credentials](complex_types) could be quite complex and support several ways of setting it up.
For example, to connect to a `sql_database` source, you can either set up a connection string:

```toml
Expand All @@ -100,13 +106,13 @@ warehouse = "warehouse_name"
role = "role"
```

`dlt` can work with both ways and convert one to another. To learn more about which credential types are supported, visit the [prebuilt credential types](prebuilt_types) page.
`dlt` can work with both ways and convert one to another. To learn more about which credential types are supported, visit the [complex credential types](complex_types) page.

## ENV variables
## Environment variables

`dlt` prioritizes security by looking in environment variables before looking into the .toml files.

The format of lookup keys is slightly different from secrets files because for ENV variables, all names are capitalized, and sections are separated with a double underscore `"__"`. For example, to specify the Facebook Ads access token through environment variables, you would need to set up:
The format of lookup keys is slightly different from secrets files because for environment variables, all names are capitalized, and sections are separated with a double underscore `"__"`. For example, to specify the Facebook Ads access token through environment variables, you would need to set up:

```sh
export SOURCES__FACEBOOK_ADS__ACCESS_TOKEN="<access_token>"
Expand Down
2 changes: 1 addition & 1 deletion docs/website/docs/general-usage/destination.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ You can create and pass partial credentials and `dlt` will fill the missing data
<!--@@@DLT_SNIPPET ./snippets/destination-snippets.py::config_partial_spec-->


Please read how to use [various built in credentials types](credentials/prebuilt_types).
Please read how to use [various built in credentials types](credentials/complex_types).
:::

### Inspect destination capabilities
Expand Down
2 changes: 1 addition & 1 deletion docs/website/docs/general-usage/glossary.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,6 @@ of the data and how to load it into the destination).
A set of values that are passed to the pipeline at run time (e.g. to change its behavior locally vs.
in production).

## [Credentials](credentials/prebuilt_types)
## [Credentials](credentials/complex_types)

A subset of configuration whose elements are kept secret and never shared in plain text.
2 changes: 1 addition & 1 deletion docs/website/docs/walkthroughs/add_credentials.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ To add credentials to your deployment,

- either use one of the `dlt deploy` commands;
- or follow the instructions to [pass credentials via code](../general-usage/credentials/advanced#examples)
or [environment](../general-usage/credentials/setup#env-variables).
or [environment](../general-usage/credentials/setup#environment-variables).

### Reading credentials from environment variables

Expand Down
2 changes: 1 addition & 1 deletion docs/website/sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,7 @@ const sidebars = {
items: [
'general-usage/credentials/setup',
'general-usage/credentials/advanced',
'general-usage/credentials/prebuilt_types',
'general-usage/credentials/complex_types',
]
},
'general-usage/pipeline',
Expand Down

0 comments on commit 3bc8779

Please sign in to comment.