From 76fde81423832c616e6f188a8ad70008736063bd Mon Sep 17 00:00:00 2001 From: Anton Burnashev Date: Tue, 14 May 2024 16:30:58 +0200 Subject: [PATCH] Link incremental loading --- .../docs/dlt-ecosystem/verified-sources/rest_api.md | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/docs/website/docs/dlt-ecosystem/verified-sources/rest_api.md b/docs/website/docs/dlt-ecosystem/verified-sources/rest_api.md index e1e271d34f..c6d79f7699 100644 --- a/docs/website/docs/dlt-ecosystem/verified-sources/rest_api.md +++ b/docs/website/docs/dlt-ecosystem/verified-sources/rest_api.md @@ -272,12 +272,12 @@ The fields in the endpoint configuration are: - `path`: The path to the API endpoint. - `method`: The HTTP method to be used. Default is `GET`. -- `params`: Query parameters to be sent with each request. For example, `sort` to order the results. +- `params`: Query parameters to be sent with each request. For example, `sort` to order the results or `since` to specify [incremental loading](#incremental-loading). This is also used to define [resource relationships](#define-resource-relationships). - `json`: The JSON payload to be sent with the request (for POST and PUT requests). - `paginator`: Pagination configuration for the endpoint. See the [pagination](#pagination) section for more details. - `data_selector`: A JSONPath to select the data from the response. See the [data selection](#data-selection) section for more details. - `response_actions`: A list of actions that define how to process the response data. -- `incremental`: Configuration for incremental loading. +- `incremental`: Configuration for [incremental loading](#incremental-loading). ### Pagination @@ -518,7 +518,10 @@ This will include the `id`, `title`, and `created_at` fields from the `issues` r ## Incremental loading -To set up incremental loading for a resource, you can use two options: +Some APIs provide a way to fetch only new or changed data (most often by using a timestamp field like `updated_at`, `created_at`, or incremental IDs). +This is called [incremental loading](../../general-usage/incremental-loading.md) and is very useful as it allows you to reduce the load time and the amount of data transferred. + +When the API endpoint supports incremental loading, you can configure the source to load only the new or changed data using these two methods: 1. Defining a special parameter in the `params` section of the [endpoint configuration](#endpoint-configuration): @@ -556,6 +559,8 @@ To set up incremental loading for a resource, you can use two options: This configuration is more flexible and allows you to specify the start and end conditions for the incremental loading. +See the [incremental loading](../../general-usage/incremental-loading.md#incremental-loading-with-a-cursor-field) guide for more details. + ## Advanced configuration `rest_api_source()` function creates the [dlt source](../../general-usage/source.md) and lets you configure the following parameters: