Add open/closed range arguments for incremental #1991
Conversation
✅ Deploy Preview for dlt-hub-docs canceled.
|
| @@ -111,6 +112,8 @@ class Incremental(ItemTransform[TDataItem], BaseConfiguration, Generic[TCursorVa | |||
| row_order: Optional[TSortOrder] = None | |||
| allow_external_schedulers: bool = False | |||
| on_cursor_value_missing: OnCursorValueMissing = "raise" | |||
| range_start: TIncrementalRange = "closed" | |||
There was a problem hiding this comment.
do not forget docstrings and docs
There was a problem hiding this comment.
also add this to TIncrementalArgs (so we can define those in REST API)
There was a problem hiding this comment.
also when start_range is open - disable boundary deduplication. there's no reason to deduplicate. there's no boundary overlap
| filter_op = operator.le | ||
| filter_op_end = operator.gt | ||
| filter_op = operator.le if self.range_start == "closed" else operator.lt | ||
| filter_op_end = operator.gt if self.range_end == "open" else operator.ge |
There was a problem hiding this comment.
we should document this in sql_database docs. we have separate chapter for incremental loading.
for example if we load incrementally by id or high resolution timestamp or we do not expect stuff to be added (ie. if we have a cursor on day) it is better to keep range open (that disables deduplication and produces faster code)
There was a problem hiding this comment.
Wrote a section on the sql docs, hope it's clear.
steinitzu
force-pushed
the
incremental-open-closed-ranges
branch
from
762d7cf to
10e0770
Compare
* Add open/closed range arguments for incremental * Docs for incremental range args * Docstring * Typo * Ensure deduplication is disabled when range_start=='open' * Cache transformer settings
Description
This allows configuring whether the incremental range is open or closed on both sides (start/end value).
Translates to changing the operators between
> | >=/< | <=in sql database source and in incremental filtering.Default is
start_range=closedandend_range=open, meaning the exact initial value is included and the exact end value is excluded.With
start_range=openyou getWHERE cursor > last_valueinstead of>=.With
end_range=closedyou get... cursor <= end_valueso non-overlapping chunks are still possible without the start_value deduplication logic.Related Issues
Additional Context