Skip to content

Commit

Permalink
Merge pull request #1557 from dlt-hub/feat/514-pluggable-paginators
Browse files Browse the repository at this point in the history
docs/514 rest_api: docs on pluggable paginators
  • Loading branch information
willi-mueller authored Jul 31, 2024
2 parents 7676e4c + 55b0922 commit 5c8f241
Show file tree
Hide file tree
Showing 2 changed files with 15 additions and 1 deletion.
14 changes: 14 additions & 0 deletions docs/website/docs/dlt-ecosystem/verified-sources/rest_api.md
Original file line number Diff line number Diff line change
Expand Up @@ -410,6 +410,20 @@ These are the available paginators:

For more complex pagination methods, you can implement a [custom paginator](../../general-usage/http/rest-client.md#implementing-a-custom-paginator), instantiate it, and use it in the configuration.

Alternatively, you can use the dictionary configuration syntax also for custom paginators. For this, you need to register your custom paginator:

```py
rest_api.config_setup.register_paginator("custom_paginator", CustomPaginator)

{
# ...
"paginator": {
"type": "custom_paginator",
"next_url_path": "paging.nextLink",
}
}
```

### Data selection

The `data_selector` field in the endpoint configuration allows you to specify a JSONPath to select the data from the response. By default, the source will try to detect locations of the data automatically.
Expand Down
2 changes: 1 addition & 1 deletion docs/website/docs/general-usage/http/rest-client.md
Original file line number Diff line number Diff line change
Expand Up @@ -306,7 +306,7 @@ client = RESTClient(

### Implementing a custom paginator

When working with APIs that use non-standard pagination schemes, or when you need more control over the pagination process, you can implement a custom paginator by subclassing the `BasePaginator` class and implementing `init_request`, `update_state` and `update_request` methods:
When working with APIs that use non-standard pagination schemes, or when you need more control over the pagination process, you can implement a custom paginator by subclassing the `BasePaginator` class and implementing the methods `init_request`, `update_state` and `update_request`.

- `init_request(request: Request) -> None`: This method is called before making the first API call in the `RESTClient.paginate` method. You can use this method to set up the initial request query parameters, headers, etc. For example, you can set the initial page number or cursor value.

Expand Down

0 comments on commit 5c8f241

Please sign in to comment.