Skip to content

Commit

Permalink
Fix snippet linting errors (#1392)
Browse files Browse the repository at this point in the history
* fix snippets

* fix additional mypy errors

* fix another auth type checker error
  • Loading branch information
sh-rp authored May 21, 2024
1 parent 1ce556a commit b1e0f77
Show file tree
Hide file tree
Showing 3 changed files with 50 additions and 40 deletions.
84 changes: 47 additions & 37 deletions docs/website/docs/dlt-ecosystem/verified-sources/rest_api.md
Original file line number Diff line number Diff line change
Expand Up @@ -203,7 +203,7 @@ For example, you can set the primary key, write disposition, and other default s
```py
config = {
"client": {
...
# ...
},
"resource_defaults": {
"primary_key": "id",
Expand All @@ -216,15 +216,17 @@ config = {
},
"resources": [
"resource1",
"resource2": {
"name": "resource2_name",
"write_disposition": "append",
"endpoint": {
"params": {
"param1": "value1",
{
"resource2": {
"name": "resource2_name",
"write_disposition": "append",
"endpoint": {
"params": {
"param1": "value1",
},
},
},
},
}
}
],
}
```
Expand Down Expand Up @@ -309,7 +311,7 @@ To specify the pagination configuration, use the `paginator` field in the [clien

```py
{
...
# ...
"paginator": {
"type": "json_links",
"next_url_path": "paging.next",
Expand All @@ -321,7 +323,7 @@ Or using the paginator instance:

```py
{
...
# ...
"paginator": JSONResponsePaginator(
next_url_path="paging.next"
),
Expand Down Expand Up @@ -394,11 +396,11 @@ One of the most common method is token-based authentication. To authenticate wit
```py
{
"client": {
...
# ...
"auth": {
"token": dlt.secrets["your_api_token"],
},
...
# ...
},
}
```
Expand All @@ -424,7 +426,7 @@ To specify the authentication configuration, use the `auth` field in the [client
"type": "bearer",
"token": dlt.secrets["your_api_token"],
},
...
# ...
},
}
```
Expand All @@ -438,7 +440,7 @@ config = {
"client": {
"auth": BearTokenAuth(dlt.secrets["your_api_token"]),
},
...
# ...
}
```

Expand All @@ -455,7 +457,7 @@ In the GitHub example, the `issue_comments` resource depends on the `issues` res
"name": "issues",
"endpoint": {
"path": "issues",
...
# ...
},
},
{
Expand Down Expand Up @@ -495,10 +497,12 @@ The `issue_comments` resource will make requests to the following endpoints:
The syntax for the `resolve` field in parameter configuration is:

```py
"<parameter_name>": {
"type": "resolve",
"resource": "<parent_resource_name>",
"field": "<parent_resource_field_name>",
{
"<parameter_name>": {
"type": "resolve",
"resource": "<parent_resource_name>",
"field": "<parent_resource_field_name>",
}
}
```

Expand Down Expand Up @@ -530,35 +534,41 @@ When the API endpoint supports incremental loading, you can configure the source
1. Defining a special parameter in the `params` section of the [endpoint configuration](#endpoint-configuration):

```py
"<parameter_name>": {
"type": "incremental",
"cursor_path": "<path_to_cursor_field>",
"initial_value": "<initial_value>",
},
{
"<parameter_name>": {
"type": "incremental",
"cursor_path": "<path_to_cursor_field>",
"initial_value": "<initial_value>",
},
}
```

For example, in the `issues` resource configuration in the GitHub example, we have:

```py
"since": {
"type": "incremental",
"cursor_path": "updated_at",
"initial_value": "2024-01-25T11:21:28Z",
},
{
"since": {
"type": "incremental",
"cursor_path": "updated_at",
"initial_value": "2024-01-25T11:21:28Z",
},
}
```

This configuration tells the source to create an incremental object that will keep track of the `updated_at` field in the response and use it as a value for the `since` parameter in subsequent requests.

2. Specifying the `incremental` field in the [endpoint configuration](#endpoint-configuration):

```py
"incremental": {
"start_param": "<parameter_name>",
"end_param": "<parameter_name>",
"cursor_path": "<path_to_cursor_field>",
"initial_value": "<initial_value>",
"end_value": "<end_value>",
},
{
"incremental": {
"start_param": "<parameter_name>",
"end_param": "<parameter_name>",
"cursor_path": "<path_to_cursor_field>",
"initial_value": "<initial_value>",
"end_value": "<end_value>",
}
}
```

This configuration is more flexible and allows you to specify the start and end conditions for the incremental loading.
Expand Down
4 changes: 2 additions & 2 deletions docs/website/docs/general-usage/http/rest-client.md
Original file line number Diff line number Diff line change
Expand Up @@ -542,7 +542,7 @@ from dlt.sources.helpers.rest_client import RESTClient
from dlt.sources.helpers.rest_client.auth import BearerTokenAuth

client = RESTClient(base_url="https://api.example.com")
response = client.get("/posts", auth=BearerTokenAuth(token="your_access_token"))
response = client.get("/posts", auth=BearerTokenAuth(token="your_access_token")) # type: ignore

print(response.status_code)
print(response.headers)
Expand Down Expand Up @@ -589,7 +589,7 @@ def response_hook(response, **kwargs):

for page in client.paginate(
"/posts",
auth=BearerTokenAuth(token="your_access_token"),
auth=BearerTokenAuth(token="your_access_token"), # type: ignore
hooks={"response": [response_hook]}
):
print(page)
Expand Down
2 changes: 1 addition & 1 deletion docs/website/docs/walkthroughs/create-a-pipeline.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ def github_api_resource(api_secret_key: str = dlt.secrets.value):

for page in paginate(
url,
auth=BearerTokenAuth(api_secret_key),
auth=BearerTokenAuth(api_secret_key), # type: ignore
paginator=HeaderLinkPaginator(),
params={"state": "open"}
):
Expand Down

0 comments on commit b1e0f77

Please sign in to comment.