Skip to content

Commit

Permalink
[docs] small fixes (#682)
Browse files Browse the repository at this point in the history
  • Loading branch information
AstrakhantsevaAA authored Oct 9, 2023
1 parent 0cce23e commit bd252da
Show file tree
Hide file tree
Showing 20 changed files with 196 additions and 122 deletions.
8 changes: 4 additions & 4 deletions docs/website/docs/dlt-ecosystem/verified-sources/airtable.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ keywords: [airtable api, airtable verified source, airtable]

# Airtable

Airtable is a cloud-based platform that merges spreadsheet and database functionalities for easy
[Airtable](https://www.airtable.com/) is a cloud-based platform that merges spreadsheet and database functionalities for easy
data management and collaboration.

This Airtable `dlt` verified source and
Expand All @@ -24,8 +24,6 @@ Sources and resources that can be loaded using this verified source are:

### Grab Airtable personal access tokens



1. Click your account icon top-right.
1. Choose "Developer Hub" from the dropdown.
1. Select "Personal access token" on the left, then "Create new token".
Expand Down Expand Up @@ -106,6 +104,8 @@ For more information, read the
> Optionally, you can also input "base_id" and "table_names" in the script, as in the pipeline
> example.
For more information, read the [General Usage: Credentials.](../../general-usage/credentials)

## Run the pipeline

1. Before running the pipeline, ensure that you have installed all the necessary dependencies by
Expand All @@ -118,7 +118,7 @@ For more information, read the
1. You're now ready to run the pipeline! To get started, run the following command:

```bash
python3 airtable_pipeline.py
python airtable_pipeline.py
```

1. Once the pipeline has finished running, you can verify that everything loaded correctly by using
Expand Down
8 changes: 4 additions & 4 deletions docs/website/docs/dlt-ecosystem/verified-sources/asana.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ keywords: [asana api, verified source, asana]
or [book a call](https://calendar.app.google/kiLhuMsWKpZUpfho6) with our support engineer Adrian.
:::

Asana is a widely used web-based project management and collaboration tool that helps teams stay
[Asana](https://asana.com) is a widely used web-based project management and collaboration tool that helps teams stay
organized, focused, and productive. With Asana, team members can easily create, assign, and track
tasks, set deadlines, and communicate with each other in real-time.

Expand Down Expand Up @@ -47,8 +47,8 @@ To get a complete list of sub-endpoints that can be loaded, see
1. This token will be used to configure `.dlt/secrets.toml`, so keep it secure and don't share it
with anyone.

More information you can see in the
[Asana official documentation](https://developers.asana.com/docs/authentication).
> Note: The Asana UI, which is described here, might change.
The full guide is available at [this link.](https://developers.asana.com/docs/authentication)

### Initialize the verified source

Expand Down Expand Up @@ -100,7 +100,7 @@ For more information, read the [General Usage: Credentials.](../../general-usage
```
1. You're now ready to run the pipeline! To get started, run the following command:
```bash
python3 asana_dlt_pipeline.py
python asana_dlt_pipeline.py
```
1. Once the pipeline has finished running, you can verify that everything loaded correctly by using
the following command:
Expand Down
2 changes: 1 addition & 1 deletion docs/website/docs/dlt-ecosystem/verified-sources/chess.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ For more information, read the [General Usage: Credentials.](../../general-usage
1. You're now ready to run the pipeline! To get started, run the following command:

```bash
python3 chess_pipeline.py
python chess_pipeline.py
```

1. Once the pipeline has finished running, you can verify that everything loaded correctly by using
Expand Down
16 changes: 11 additions & 5 deletions docs/website/docs/dlt-ecosystem/verified-sources/facebook_ads.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ Facebook and its affiliated apps like Instagram and Messenger.

This Facebook `dlt` verified source and
[pipeline example](https://github.com/dlt-hub/verified-sources/blob/master/sources/facebook_ads_pipeline.py)
loads data using Facebook Marketing API to the destination of your choice.
loads data using [Facebook Marketing API](https://developers.facebook.com/products/marketing-api/) to the destination of your choice.

The endpoints that this verified source supports are:

Expand Down Expand Up @@ -85,13 +85,17 @@ debug_access_token()

We highly recommend you to add the token expiration timestamp to get notified a week before token
expiration that you need to rotate it. Right now the notifications are sent to logger with error
level. In config.toml / secrets.toml:
level. In `config.toml` / `secrets.toml`:

```toml
[sources.facebook_ads]
access_token_expires_at=1688821881
access_token_expires_at=1688821881...
```

> Note: The Facebook UI, which is described here, might change.
The full guide is available at [this link.](https://developers.facebook.com/docs/marketing-apis/overview/authentication)


### Initialize the verified source

To get started with your data pipeline, follow these steps:
Expand Down Expand Up @@ -150,6 +154,8 @@ For more information, read the

1. Replace the value of the "account id" with the one [copied above](#grab-account-id).

For more information, read the [General Usage: Credentials.](../../general-usage/credentials)

## Run the pipeline

1. Before running the pipeline, ensure that you have installed all the necessary dependencies by
Expand All @@ -159,7 +165,7 @@ For more information, read the
```
1. You're now ready to run the pipeline! To get started, run the following command:
```bash
python3 facebook_ads_pipeline.py
python facebook_ads_pipeline.py
```
1. Once the pipeline has finished running, you can verify that everything loaded correctly by using
the following command:
Expand Down Expand Up @@ -269,7 +275,7 @@ def facebook_insights_source(
) -> DltResource:
```

`account_id`: Account id associated with add manager, configured in _config.toml_.
`account_id`: Account id associated with ads manager, configured in _config.toml_.

`access_token`: Access token associated with the Business Facebook App, configured in
_secrets.toml_.
Expand Down
13 changes: 7 additions & 6 deletions docs/website/docs/dlt-ecosystem/verified-sources/github.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ or [book a call](https://calendar.app.google/kiLhuMsWKpZUpfho6) with our support
:::

This verified source can be used to load data on issues or pull requests from any GitHub repository
onto a [destination](../../dlt-ecosystem/destinations) of your choice.
onto a [destination](../../dlt-ecosystem/destinations) of your choice using [GitHub API](https://docs.github.com/en/rest?apiVersion=2022-11-28).

Resources that can be loaded using this verified source are:

Expand Down Expand Up @@ -52,14 +52,13 @@ To get the API token, sign-in to your GitHub account and follow these steps:

1. Copy the token and save it. This is to be added later in the `dlt` configuration.

> You can optionally add API access tokens to avoid making requests as an unauthorized user.\
> Note:
> You can optionally add API access tokens to avoid making requests as an unauthorized user.
> If you wish to load data using the github_reaction source, the access token is mandatory.
More information you can see in the
[GitHub authentication](https://docs.github.com/en/rest/overview/authenticating-to-the-rest-api?apiVersion=2022-11-28#basic-authentication)
and
[Github API token scopes](https://docs.github.com/en/apps/oauth-apps/building-oauth-apps/scopes-for-oauth-apps)
[GitHub API token scopes](https://docs.github.com/en/apps/oauth-apps/building-oauth-apps/scopes-for-oauth-apps)
documentations.

### Initialize the verified source
Expand Down Expand Up @@ -106,7 +105,7 @@ For more information, read the
add credentials for your chosen destination, ensuring proper routing of your data to the final
destination.

For more information, read the [Walkthrough: Run a pipeline.](../../walkthroughs/run-a-pipeline)
For more information, read the [General Usage: Credentials.](../../general-usage/credentials)

## Run the pipeline

Expand All @@ -117,7 +116,7 @@ For more information, read the [Walkthrough: Run a pipeline.](../../walkthroughs
```
1. You're now ready to run the pipeline! To get started, run the following command:
```bash
python3 github_pipeline.py
python github_pipeline.py
```
1. Once the pipeline has finished running, you can verify that everything loaded correctly by using
the following command:
Expand All @@ -127,6 +126,8 @@ For more information, read the [Walkthrough: Run a pipeline.](../../walkthroughs
For example, the `pipeline_name` for the above pipeline example is `github_reactions`, you may
also use any custom name instead.

For more information, read the [Walkthrough: Run a pipeline.](../../walkthroughs/run-a-pipeline)

## Sources and resources

`dlt` works on the principle of [sources](../../general-usage/source) and
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,18 +36,18 @@ tokens are preferred when user consent is required, while service account creden
suited for server-to-server interactions. You can choose the method of authentication as per your
requirement.

### Grab google service account credentials
### Grab Google service account credentials

You need to create a GCP service account to get API credentials, if you don't have one. To create
one follow these steps:
You need to create a GCP service account to get API credentials if you don't have one. To create
one, follow these steps:

1. Sign in to [console.cloud.google.com](http://console.cloud.google.com/).

1. [Create a service account](https://cloud.google.com/iam/docs/service-accounts-create#creating) if
1. [Create a service account](https://cloud.google.com/iam/docs/service-accounts-create#creating) if
needed.

1. Enable "Google Analytics API", refer
[google documentation](https://support.google.com/googleapi/answer/6158841?hl=en) for
[Google documentation](https://support.google.com/googleapi/answer/6158841?hl=en) for
comprehensive instructions on this process.

1. Generate credentials:
Expand All @@ -60,7 +60,7 @@ one follow these steps:

### Grab google OAuth credentials

You need to create a GCP account to get OAuth credentials, if you don't have one. To create one
You need to create a GCP account to get OAuth credentials if you don't have one. To create one,
follow these steps:

1. Ensure your email used for the GCP account has access to the GA4 property.
Expand Down Expand Up @@ -91,10 +91,10 @@ follow these steps:
1. Add your email as a test user.

After configuring "client_id", "client_secret" and "project_id" in "secrets.toml". To generate the
refresh token run the following script from the root folder:
refresh token, run the following script from the root folder:

```bash
python3 google_analytics/setup_script_gcp_oauth.py
python google_analytics/setup_script_gcp_oauth.py
```

Once you have executed the script and completed the authentication, you will receive a "refresh
Expand Down Expand Up @@ -161,10 +161,10 @@ For more information, read the

1. From the ".json" that you
[downloaded earlier](google_analytics.md#grab-google-service-account-credentials),
copy `project_id`, `private_key`,
and `client_email` under `[sources.google_analytics.credentials]`.
copy `project_id`, `private_key`,
and `client_email` under `[sources.google_analytics.credentials]`.

1. Alternatively, if you're using OAuth credentials, replace the the fields and values with those
1. Alternatively, if you're using OAuth credentials, replace the fields and values with those
you [grabbed for OAuth credentials](google_analytics.md#grab-google-oauth-credentials).

1. The secrets.toml for OAuth authentication looks like:
Expand All @@ -177,7 +177,7 @@ For more information, read the
project_id = "project_id" # please set me up!
```

1. Finally, enter credentials for your chosen destination as per the [docs](../destinations/).
1. Finally, enter credentials for your chosen destination as per the [docs](../destinations/).

#### Pass `property_id` and `request parameters`

Expand Down Expand Up @@ -209,6 +209,8 @@ For more information, read the
1. To use queries from `.dlt/config.toml`, run the `simple_load_config()` function in
[pipeline example](https://github.com/dlt-hub/verified-sources/blob/master/sources/google_analytics_pipeline.py).

For more information, read the [General Usage: Credentials.](../../general-usage/credentials)

## Run the pipeline

1. Before running the pipeline, ensure that you have installed all the necessary dependencies by
Expand All @@ -218,7 +220,7 @@ For more information, read the
```
1. You're now ready to run the pipeline! To get started, run the following command:
```bash
python3 google_analytics_pipeline.py
python google_analytics_pipeline.py
```
1. Once the pipeline has finished running, you can verify that everything loaded correctly by using
the following command:
Expand Down Expand Up @@ -295,6 +297,7 @@ def metrics_table(metadata: Metadata) -> Iterator[TDataItem]:
Similarly, there is a transformer function called `dimensions_table` that populates table called
"dimensions" with the data from each dimension.

## Customization
### Create your own pipeline

If you wish to create your own pipelines, you can leverage source and resource methods from this
Expand All @@ -304,9 +307,9 @@ verified source.

```python
pipeline = dlt.pipeline(
pipeline_name="google_analytics", # Use a custom name if desired
destination="duckdb", # Choose the appropriate destination (e.g., duckdb, redshift, post)
dataset_name="GA4_data" # Use a custom name if desired
pipeline_name="google_analytics", # Use a custom name if desired
destination="duckdb", # Choose the appropriate destination (e.g., duckdb, redshift, post)
dataset_name="GA4_data" # Use a custom name if desired
)
```

Expand Down
Loading

0 comments on commit bd252da

Please sign in to comment.