Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix snippet linting errors #1392

Merged
merged 3 commits into from
May 21, 2024
Merged

Fix snippet linting errors #1392

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
99ccdb6
fix snippets
sh-rp May 21, 2024
356b60c
fix additional mypy errors
sh-rp May 21, 2024
363b3fd
fix another auth type checker error
sh-rp May 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
84 changes: 47 additions & 37 deletions docs/website/docs/dlt-ecosystem/verified-sources/rest_api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -203,7 +203,7 @@ For example, you can set the primary key, write disposition, and other default s
```py
config = {
"client": {
...
# ...
},
"resource_defaults": {
"primary_key": "id",
Expand All @@ -216,15 +216,17 @@ config = {
},
"resources": [
"resource1",
"resource2": {
"name": "resource2_name",
"write_disposition": "append",
"endpoint": {
"params": {
"param1": "value1",
{
"resource2": {
"name": "resource2_name",
"write_disposition": "append",
"endpoint": {
"params": {
"param1": "value1",
},
},
},
},
}
}
],
}
```
Expand Down Expand Up @@ -309,7 +311,7 @@ To specify the pagination configuration, use the `paginator` field in the [clien

```py
{
...
# ...
"paginator": {
"type": "json_links",
"next_url_path": "paging.next",
Expand All @@ -321,7 +323,7 @@ Or using the paginator instance:

```py
{
...
# ...
"paginator": JSONResponsePaginator(
next_url_path="paging.next"
),
Expand Down Expand Up @@ -394,11 +396,11 @@ One of the most common method is token-based authentication. To authenticate wit
```py
{
"client": {
...
# ...
"auth": {
"token": dlt.secrets["your_api_token"],
},
...
# ...
},
}
```
Expand All @@ -424,7 +426,7 @@ To specify the authentication configuration, use the `auth` field in the [client
"type": "bearer",
"token": dlt.secrets["your_api_token"],
},
...
# ...
},
}
```
Expand All @@ -438,7 +440,7 @@ config = {
"client": {
"auth": BearTokenAuth(dlt.secrets["your_api_token"]),
},
...
# ...
}
```

Expand All @@ -455,7 +457,7 @@ In the GitHub example, the `issue_comments` resource depends on the `issues` res
"name": "issues",
"endpoint": {
"path": "issues",
...
# ...
},
},
{
Expand Down Expand Up @@ -495,10 +497,12 @@ The `issue_comments` resource will make requests to the following endpoints:
The syntax for the `resolve` field in parameter configuration is:

```py
"<parameter_name>": {
"type": "resolve",
"resource": "<parent_resource_name>",
"field": "<parent_resource_field_name>",
{
"<parameter_name>": {
"type": "resolve",
"resource": "<parent_resource_name>",
"field": "<parent_resource_field_name>",
}
}
```

Expand Down Expand Up @@ -530,35 +534,41 @@ When the API endpoint supports incremental loading, you can configure the source
1. Defining a special parameter in the `params` section of the [endpoint configuration](#endpoint-configuration):

```py
"<parameter_name>": {
"type": "incremental",
"cursor_path": "<path_to_cursor_field>",
"initial_value": "<initial_value>",
},
{
"<parameter_name>": {
"type": "incremental",
"cursor_path": "<path_to_cursor_field>",
"initial_value": "<initial_value>",
},
}
```

For example, in the `issues` resource configuration in the GitHub example, we have:

```py
"since": {
"type": "incremental",
"cursor_path": "updated_at",
"initial_value": "2024-01-25T11:21:28Z",
},
{
"since": {
"type": "incremental",
"cursor_path": "updated_at",
"initial_value": "2024-01-25T11:21:28Z",
},
}
```

This configuration tells the source to create an incremental object that will keep track of the `updated_at` field in the response and use it as a value for the `since` parameter in subsequent requests.

2. Specifying the `incremental` field in the [endpoint configuration](#endpoint-configuration):

```py
"incremental": {
"start_param": "<parameter_name>",
"end_param": "<parameter_name>",
"cursor_path": "<path_to_cursor_field>",
"initial_value": "<initial_value>",
"end_value": "<end_value>",
},
{
"incremental": {
"start_param": "<parameter_name>",
"end_param": "<parameter_name>",
"cursor_path": "<path_to_cursor_field>",
"initial_value": "<initial_value>",
"end_value": "<end_value>",
}
}
```

This configuration is more flexible and allows you to specify the start and end conditions for the incremental loading.
Expand Down
Loading