You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In a6a1ef8 of #10751 we added some edge case handling logic such that if every batch for a microbatch model failed, the batches would be regenerated on retry and true/false on full-refresh would be properly applied. This was to address the following edge case:
a microbatch model had a begin of 2020-01-01 and has been running this way for a while
the begin config has changed to 2024-01-01 and dbt run --full-refresh gets run
every batch for a microbatch model fails
on dbt retry the relation for the model is said to exist, and the now out of range data (2020-01-01 through 2023-12-31) is never purged
The solution in a6a1ef8 did solve this edge case, however it also created another:
On dbt run a microbatch model has 2 batches that succeed and 1 batch that fails
On the first invocation of dbt retry the 1 previously failed batch is retried and fails again
dbt retry is run a second time and now 3 batches are run (the initial two that succeeded, and the failing one)
At step (3) only the failing batch should be run. However, presently our logic is that if a prior run has only failed batches, then we rerun the microbatch model as it did on dbt run. We need on the second retry to know that we had previously successful batches for the initial run being retried. However, we also must continue to solve the edge case we resolved in a6a1ef8.
The text was updated successfully, but these errors were encountered:
In a6a1ef8 of #10751 we added some edge case handling logic such that if every batch for a microbatch model failed, the batches would be regenerated on retry and true/false on
full-refresh
would be properly applied. This was to address the following edge case:begin
of2020-01-01
and has been running this way for a whilebegin
config has changed to2024-01-01
anddbt run --full-refresh
gets rundbt retry
the relation for the model is said to exist, and the now out of range data (2020-01-01 through 2023-12-31) is never purgedThe solution in a6a1ef8 did solve this edge case, however it also created another:
dbt run
a microbatch model has 2 batches that succeed and 1 batch that failsdbt retry
the 1 previously failed batch is retried and fails againdbt retry
is run a second time and now 3 batches are run (the initial two that succeeded, and the failing one)At step (3) only the failing batch should be run. However, presently our logic is that if a prior run has only failed batches, then we rerun the microbatch model as it did on
dbt run
. We need on the second retry to know that we had previously successful batches for the initial run being retried. However, we also must continue to solve the edge case we resolved in a6a1ef8.
The text was updated successfully, but these errors were encountered: