Skip to content

Commit

Permalink
Updated
Browse files Browse the repository at this point in the history
  • Loading branch information
dat-a-man committed Mar 21, 2024
1 parent 5835801 commit 4c224e1
Showing 1 changed file with 4 additions and 5 deletions.
9 changes: 4 additions & 5 deletions docs/website/docs/general-usage/incremental-loading.md
Original file line number Diff line number Diff line change
Expand Up @@ -621,7 +621,7 @@ Let's take the following example to read incremental loading parameters from the
1. In "config.toml", define the parameter `id_after` as follows:
```toml
# Configuration snippet for an incremental resource
[incs.sources.pipeline.inc_res.id_after]
[incremental_source.sources.pipeline.generate_incremental_records.id_after]
cursor_path = "idAfter"
initial_value = 10
```
Expand All @@ -630,19 +630,18 @@ Let's take the following example to read incremental loading parameters from the
1. The `id_after` parameter is defined as an incremental source, and its value is retrieved from the configuration using `dlt.config.value`.
```py
@dlt.resource(table_name="incremental_records")
def inc_res(id_after: dlt.sources.incremental = dlt.config.value):
def generate_incremental_records(id_after: dlt.sources.incremental = dlt.config.value):
for i in range(100):
yield {"id": i, "idAfter": i, "name": "name-" + str(i)}


pipeline = dlt.pipeline(
pipeline_name="pipeline_with_incremental",
destination="duckdb",
)

pipeline.run(inc_res)
pipeline.run(generate_incremental_records)
```
The `inc_res` function generates a range of data, each item being a dictionary with keys `id`, `idAfter`, and `name`. The `idAfter` key is used by the incremental resource to track progress.
The `generate_incremental_records` function generates a range of data, each item being a dictionary with keys `id`, `idAfter`, and `name`. The `idAfter` key is used by the incremental resource to track progress.

## Doing a full refresh

Expand Down

0 comments on commit 4c224e1

Please sign in to comment.