Skip to content

Commit

Permalink
Updated rest_api.md with configuration examples
Browse files Browse the repository at this point in the history
  • Loading branch information
dat-a-man committed May 16, 2024
1 parent 34e77ac commit 2931d32
Showing 1 changed file with 47 additions and 36 deletions.
83 changes: 47 additions & 36 deletions docs/website/docs/dlt-ecosystem/verified-sources/rest_api.md
Original file line number Diff line number Diff line change
Expand Up @@ -174,13 +174,13 @@ The configuration object passed to the REST API Generic Source has three main el
```py
config: RESTAPIConfig = {
"client": {
...
# ...
},
"resource_defaults": {
...
# ...
},
"resources": [
...
# ...
],
}
```
Expand All @@ -203,7 +203,9 @@ For example, you can set the primary key, write disposition, and other default s
```py
config = {
"client": {
...
"api_key": "your_api_key_here",
"base_url": "https://api.example.com",
# Add other client configurations here
},
"resource_defaults": {
"primary_key": "id",
Expand All @@ -216,7 +218,7 @@ config = {
},
"resources": [
"resource1",
"resource2": {
{
"name": "resource2_name",
"write_disposition": "append",
"endpoint": {
Expand Down Expand Up @@ -309,7 +311,7 @@ To specify the pagination configuration, use the `paginator` field in the [clien

```py
{
...
# ...
"paginator": {
"type": "json_links",
"next_url_path": "paging.next",
Expand All @@ -321,7 +323,7 @@ Or using the paginator instance:

```py
{
...
# ...
"paginator": JSONResponsePaginator(
next_url_path="paging.next"
),
Expand Down Expand Up @@ -394,11 +396,11 @@ One of the most common method is token-based authentication. To authenticate wit
```py
{
"client": {
...
# ...
"auth": {
"token": dlt.secrets["your_api_token"],
},
...
# ...
},
}
```
Expand All @@ -424,7 +426,7 @@ To specify the authentication configuration, use the `auth` field in the [client
"type": "bearer",
"token": dlt.secrets["your_api_token"],
},
...
# ...
},
}
```
Expand All @@ -438,7 +440,7 @@ config = {
"client": {
"auth": BearTokenAuth(dlt.secrets["your_api_token"]),
},
...
# ...
}
```

Expand All @@ -455,7 +457,7 @@ In the GitHub example, the `issue_comments` resource depends on the `issues` res
"name": "issues",
"endpoint": {
"path": "issues",
...
# ...
},
},
{
Expand Down Expand Up @@ -495,13 +497,15 @@ The `issue_comments` resource will make requests to the following endpoints:
The syntax for the `resolve` field in parameter configuration is:

```py
"<parameter_name>": {
"type": "resolve",
"resource": "<parent_resource_name>",
"field": "<parent_resource_field_name>",
}
({
"{parameter_name}" :
{
"type": "resolve",
"resource": "{parent_resource_name}",
"field": "{parent_resource_field_name}",
}
})
```

Under the hood, dlt handles this by using a [transformer resource](../../general-usage/resource.md#process-resources-with-dlttransformer).

#### Include fields from the parent resource
Expand All @@ -512,7 +516,7 @@ You can include data from the parent resource in the child resource by using the
{
"name": "issue_comments",
"endpoint": {
...
# ...
},
"include_from_parent": ["id", "title", "created_at"],
}
Expand All @@ -530,35 +534,42 @@ When the API endpoint supports incremental loading, you can configure the source
1. Defining a special parameter in the `params` section of the [endpoint configuration](#endpoint-configuration):

```py
"<parameter_name>": {
"type": "incremental",
"cursor_path": "<path_to_cursor_field>",
"initial_value": "<initial_value>",
},

({
"<parameter_name>": {
"type": "incremental",
"cursor_path": "<path_to_cursor_field>",
"initial_value": "<initial_value>",
}
})
```

For example, in the `issues` resource configuration in the GitHub example, we have:

```py
"since": {
"type": "incremental",
"cursor_path": "updated_at",
"initial_value": "2024-01-25T11:21:28Z",
},
({
"since": {
"type": "incremental",
"cursor_path": "updated_at",
"initial_value": "2024-01-25T11:21:28Z",
}
})
```

This configuration tells the source to create an incremental object that will keep track of the `updated_at` field in the response and use it as a value for the `since` parameter in subsequent requests.

2. Specifying the `incremental` field in the [endpoint configuration](#endpoint-configuration):

```py
"incremental": {
"start_param": "<parameter_name>",
"end_param": "<parameter_name>",
"cursor_path": "<path_to_cursor_field>",
"initial_value": "<initial_value>",
"end_value": "<end_value>",
},
({
"incremental": {
"start_param": "<parameter_name>",
"end_param": "<parameter_name>",
"cursor_path": "<path_to_cursor_field>",
"initial_value": "<initial_value>",
"end_value": "<end_value>",
}
})
```

This configuration is more flexible and allows you to specify the start and end conditions for the incremental loading.
Expand Down

0 comments on commit 2931d32

Please sign in to comment.