Skip to content

Commit

Permalink
Final comments
Browse files Browse the repository at this point in the history
  • Loading branch information
VioletM committed Aug 7, 2024
1 parent 3701655 commit f2d434b
Show file tree
Hide file tree
Showing 9 changed files with 22 additions and 16 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
In this example, you'll find a Python script that demonstrates how to load to BigQuery with the custom destination.
We'll learn how to:
- Use [built-in credentials.](../general-usage/credentials/prebuilt_types#gcp-credentials)
- Use [built-in credentials.](../general-usage/credentials/complex_types#gcp-credentials)
- Use the [custom destination.](../dlt-ecosystem/destinations/destination.md)
- Use pyarrow tables to create complex column types on BigQuery.
- Use BigQuery `autodetect=True` for schema inference from parquet files.
Expand Down
4 changes: 2 additions & 2 deletions docs/examples/google_sheets/google_sheets.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@
In this example, you'll find a Python script that demonstrates how to load Google Sheets data using the `dlt` library.
We'll learn how to:
- use [built-in credentials](../general-usage/credentials/prebuilt_types#gcp-credentials);
- use [union of credentials](../general-usage/credentials/prebuilt_types#working-with-alternatives-of-credentials-union-types);
- use [built-in credentials](../general-usage/credentials/complex_types#gcp-credentials);
- use [union of credentials](../general-usage/credentials/complex_types#working-with-alternatives-of-credentials-union-types);
- create [dynamically generated resources](../general-usage/source#create-resources-dynamically).
:::tip
Expand Down
4 changes: 2 additions & 2 deletions docs/website/docs/general-usage/credentials/advanced.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ Doing so provides several benefits:
1. You'll never receive the invalid data types in your code.
1. `dlt` will automatically parse and coerce types for you, so you don't need to parse it yourself.
1. `dlt` can generate sample config and secret files for your source automatically.
1. You can request [built-in and custom credentials](prebuilt_types) (i.e., connection strings, AWS / GCP / Azure credentials).
1. You can request [built-in and custom credentials](complex_types) (i.e., connection strings, AWS / GCP / Azure credentials).
1. You can specify a set of possible types via `Union`, i.e., OAuth or API Key authorization.

Let's consider the example:
Expand All @@ -84,7 +84,7 @@ def google_sheets(
Now,

1. You are sure that you get a list of strings as `tab_names`.
1. You will get actual Google credentials (see [GCP Credential Configuration](prebuilt_types#gcp-credentials)), and users can
1. You will get actual Google credentials (see [GCP Credential Configuration](complex_types#gcp-credentials)), and users can
pass them in many different forms:

* `service.json` as a string or dictionary (in code and via config providers).
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Prebuilt credential types
description: How to specify complex custom credentials.
title: Complex credential types
description: Instructions for credentials like DB connection string.
keywords: [credentials, secrets.toml, secrets, config, configuration, environment
variables, specs]
---
Expand Down
16 changes: 11 additions & 5 deletions docs/website/docs/general-usage/credentials/setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,19 @@ keywords: [credentials, secrets.toml, secrets, config, configuration, environmen
---

`dlt` automatically extracts configuration settings and secrets based on flexible [naming conventions](setup/#naming-convention).

It then [injects](advanced/#injection-mechanism) these values where needed in functions decorated with `@dlt.source`, `@dlt.resource`, or `@dlt.destination`.

:::note
* **Configuration** refers to non-sensitive settings that define a data pipeline's behavior. These include file paths, database hosts, timeouts, API URLs, and performance settings.
* **Secrets** are sensitive data like passwords, API keys, and private keys. They should never be hard-coded to avoid security risks.
:::

## Available config providers

There are multiple ways to define configurations and credentials for your pipelines. `dlt` looks for these definitions in the following order during pipeline execution:

1. [Environment Variables](#env-variables): If a value for a specific argument is found in an environment variable, dlt will use it and will not proceed to search in lower-priority providers.
1. [Environment Variables](#environment-variables): If a value for a specific argument is found in an environment variable, dlt will use it and will not proceed to search in lower-priority providers.

1. [Vaults](#vaults): Credentials specified in vaults like Google Secrets Manager, Azure Key Vault, AWS Secrets Manager.

Expand Down Expand Up @@ -80,7 +86,7 @@ project_id = "<project_id_2>"

### Credential types

In most cases, credentials are just key-value pairs, but in some cases, the actual structure of [credentials](prebuilt_types) could be quite complex and support several ways of setting it up.
In most cases, credentials are just key-value pairs, but in some cases, the actual structure of [credentials](complex_types) could be quite complex and support several ways of setting it up.
For example, to connect to a `sql_database` source, you can either set up a connection string:

```toml
Expand All @@ -100,13 +106,13 @@ warehouse = "warehouse_name"
role = "role"
```

`dlt` can work with both ways and convert one to another. To learn more about which credential types are supported, visit the [prebuilt credential types](prebuilt_types) page.
`dlt` can work with both ways and convert one to another. To learn more about which credential types are supported, visit the [complex credential types](complex_types) page.

## ENV variables
## Environment variables

`dlt` prioritizes security by looking in environment variables before looking into the .toml files.

The format of lookup keys is slightly different from secrets files because for ENV variables, all names are capitalized, and sections are separated with a double underscore `"__"`. For example, to specify the Facebook Ads access token through environment variables, you would need to set up:
The format of lookup keys is slightly different from secrets files because for environment variables, all names are capitalized, and sections are separated with a double underscore `"__"`. For example, to specify the Facebook Ads access token through environment variables, you would need to set up:

```sh
export SOURCES__FACEBOOK_ADS__ACCESS_TOKEN="<access_token>"
Expand Down
2 changes: 1 addition & 1 deletion docs/website/docs/general-usage/destination.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ You can create and pass partial credentials and `dlt` will fill the missing data
<!--@@@DLT_SNIPPET ./snippets/destination-snippets.py::config_partial_spec-->


Please read how to use [various built in credentials types](credentials/prebuilt_types).
Please read how to use [various built in credentials types](credentials/complex_types).
:::

### Inspect destination capabilities
Expand Down
2 changes: 1 addition & 1 deletion docs/website/docs/general-usage/glossary.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,6 @@ of the data and how to load it into the destination).
A set of values that are passed to the pipeline at run time (e.g. to change its behavior locally vs.
in production).

## [Credentials](credentials/prebuilt_types)
## [Credentials](credentials/complex_types)

A subset of configuration whose elements are kept secret and never shared in plain text.
2 changes: 1 addition & 1 deletion docs/website/docs/walkthroughs/add_credentials.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ To add credentials to your deployment,

- either use one of the `dlt deploy` commands;
- or follow the instructions to [pass credentials via code](../general-usage/credentials/advanced#examples)
or [environment](../general-usage/credentials/setup#env-variables).
or [environment](../general-usage/credentials/setup#environment-variables).

### Reading credentials from environment variables

Expand Down
2 changes: 1 addition & 1 deletion docs/website/sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,7 @@ const sidebars = {
items: [
'general-usage/credentials/setup',
'general-usage/credentials/advanced',
'general-usage/credentials/prebuilt_types',
'general-usage/credentials/complex_types',
]
},
'general-usage/pipeline',
Expand Down

0 comments on commit f2d434b

Please sign in to comment.