Skip to content

Commit

Permalink
Typo
Browse files Browse the repository at this point in the history
  • Loading branch information
steinitzu committed Dec 5, 2024
1 parent b9d8dbd commit ca07633
Show file tree
Hide file tree
Showing 2 changed files with 5 additions and 5 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ This ensures there are no gaps in the extracted sequence. But it does come with
both due to the deduplication processing and the cost of fetching redundant records from the database.

This is not always needed. If you know that your data does not contain overlapping cursor values then you
can optimize extraction by passing `start_range="open"` to incremental.
can optimize extraction by passing `range_start="open"` to incremental.

This both disables the deduplication process and changes the operator used in the SQL `WHERE` clause from `>=` (greater-or-equal) to `>` (greater than), so that no overlapping rows are fetched.

Expand All @@ -110,7 +110,7 @@ table = sql_table(
incremental=dlt.sources.incremental(
'last_modified', # Cursor column name
initial_value=pendulum.DateTime(2024, 1, 1, 0, 0, 0), # Initial cursor value
start_range="open", # exclude the start value
range_start="open", # exclude the start value
)
)
```
Expand Down
6 changes: 3 additions & 3 deletions docs/website/docs/general-usage/incremental-loading.md
Original file line number Diff line number Diff line change
Expand Up @@ -693,7 +693,7 @@ august_issues = repo_issues(
...
```

Note that dlt's incremental filtering considers the ranges half-closed. `initial_value` is inclusive, `end_value` is exclusive, so chaining ranges like above works without overlaps. This behaviour can be changed with the `start_range` (default `"closed"`) and `end_range` (default `"open"`) arguments.
Note that dlt's incremental filtering considers the ranges half-closed. `initial_value` is inclusive, `end_value` is exclusive, so chaining ranges like above works without overlaps. This behaviour can be changed with the `range_start` (default `"closed"`) and `range_end` (default `"open"`) arguments.

### Declare row order to not request unnecessary data

Expand Down Expand Up @@ -793,8 +793,8 @@ def some_data(last_timestamp=dlt.sources.incremental("item.ts", primary_key=()))
yield {"delta": i, "item": {"ts": pendulum.now().timestamp()}}
```

This deduplication process is always enabled when `start_range` is set to `"closed"` (default).
When you pass `start_range="open"` no deduplication is done as it is not needed as rows with the previous cursor value are excluded. This can be a useful optimization to avoid the performance overhead of deduplication if the cursor field is guaranteed to be unique.
This deduplication process is always enabled when `range_start` is set to `"closed"` (default).
When you pass `range_start="open"` no deduplication is done as it is not needed as rows with the previous cursor value are excluded. This can be a useful optimization to avoid the performance overhead of deduplication if the cursor field is guaranteed to be unique.

### Using `dlt.sources.incremental` with dynamically created resources

Expand Down

0 comments on commit ca07633

Please sign in to comment.