You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We're experiencing an issue with incremental models created using a JavaScript publish method. Even when we trigger a run with the "Run with full refresh" option enabled, the model still seems to execute incrementally.
Is this the expected behavior, a potential bug, or perhaps a misconfiguration on our end?
Due to business constraints, we can't share the actual model code. However, here's a simplified example demonstrating our approach:
constraw_tables=["table1","table2","table3"];constconfig={type: "incremental",bigquery: {partitionBy: "DATE(Timestamp)",},};publish("model_name",config).preOps(ctx=>` DECLARE event_timestamp_checkpoint DEFAULT (${ctx.when(ctx.incremental(),`SELECT MAX(Timestamp) FROM ${ctx.self()}`,`SELECT TIMESTAMP("2000-01-01")`)});${ctx.when(ctx.incremental(),`DELETE FROM ${ctx.self()} WHERE Timestamp >= event_timestamp_checkpoint;`)} SET @@query_label = "${query_label.from_object(labels)}";`).query(ctx=>`${raw_tables.map((table,i,tables)=>`SELECT Timestamp, Column1, Column2, Column3,FROM ${ctx.ref("project","dataset",table)}WHERE Timestamp > event_timestamp_checkpoint${tables.length-1===i ? "" : "UNION ALL"} `).join("")}`);
Any insights or suggestions for a workaround would be greatly appreciated.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Hi everyone,
We're experiencing an issue with incremental models created using a JavaScript
publish
method. Even when we trigger a run with the "Run with full refresh" option enabled, the model still seems to execute incrementally.Is this the expected behavior, a potential bug, or perhaps a misconfiguration on our end?
Due to business constraints, we can't share the actual model code. However, here's a simplified example demonstrating our approach:
Any insights or suggestions for a workaround would be greatly appreciated.
Thanks in advance!
Beta Was this translation helpful? Give feedback.
All reactions