You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In #10624 we're building a new type of incremental model strategy, microbatch. To support that we need a concept of lookback. The lookback is essentially how far to look back in multiples of the batch_size when automatically setting the event_time_start. So if event_time_end="2024-09-04 13:44:00", batch_size="day", and lookback=4, then the event_start_time would be set to 2024-08-31 00:00:00.
If defining it in the model sql it'd look like
-- my_model.sql
{{ config(
materialization='incremental',
incremental_strategy='microbatch',
event_time='my_time_field',
batch_size='day',
lookback=0, -- int, defaults to 0 if not specified
)
}}
...
If defining it in the model yaml it'd look like:
models:
- name: my_modelconfig:
event_time: my_time_fieldincremental_strategy: microbatchbatch_size: daylookback: 0# int, defaults to 0 if not specified
Acceptance criteria
a lookback value can be specified in a model's config
the lookback value should default to 0 if not specified
the lookback value alters the generatedevent_time_start by a multiple of the batch_size
Suggested Tests
a lookback value can be specified in a model's config (via yaml and sql)
the lookback value is default to 0 if not specified
the lookback value alters the generatedevent_time_start by a multiple of the batch_size
Impact to Other Teams
Cloud artifacts, new property in the config of models
Will backports be required?
No
Context
No response
The text was updated successfully, but these errors were encountered:
QMalcolm
changed the title
Allow setting of a lookback_period to effect incremental building
[Feature] Allow setting of a lookback as top level model property
Sep 4, 2024
Housekeeping
Short description
In #10624 we're building a new type of incremental model strategy,
microbatch
. To support that we need a concept oflookback
. Thelookback
is essentially how far to look back in multiples of thebatch_size
when automatically setting theevent_time_start
. So ifevent_time_end="2024-09-04 13:44:00"
,batch_size="day"
, andlookback=4
, then theevent_start_time
would be set to2024-08-31 00:00:00
.If defining it in the model sql it'd look like
If defining it in the model yaml it'd look like:
Acceptance criteria
lookback
value can be specified in a model's configlookback
value should default to0
if not specifiedlookback
value alters the generatedevent_time_start
by a multiple of thebatch_size
Suggested Tests
lookback
value can be specified in a model's config (via yaml and sql)lookback
value is default to0
if not specifiedlookback
value alters the generatedevent_time_start
by a multiple of thebatch_size
Impact to Other Teams
Cloud artifacts, new property in the config of models
Will backports be required?
No
Context
No response
The text was updated successfully, but these errors were encountered: