Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Custom incremental strategies supported on BigQuery #1406

Closed
2 tasks done
jackodes opened this issue Nov 14, 2024 · 1 comment
Closed
2 tasks done

[Feature] Custom incremental strategies supported on BigQuery #1406

jackodes opened this issue Nov 14, 2024 · 1 comment
Labels
type:enhancement New feature or request

Comments

@jackodes
Copy link

Is this a new bug in dbt-bigquery?

  • I believe this is a new bug in dbt-bigquery
  • I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

I'm trying to implement a custom incremental strategy based on this documentation.

But there's a specific macro on dbt-bigquery that checks for the name of the incremental strategy and raises an error in case it's none of the predefined ones:

{% macro dbt_bigquery_validate_get_incremental_strategy(config) %}
  {#-- Find and validate the incremental strategy #}
  {%- set strategy = config.get("incremental_strategy") or 'merge' -%}

  {% set invalid_strategy_msg -%}
    Invalid incremental strategy provided: {{ strategy }}
    Expected one of: 'merge', 'insert_overwrite', 'microbatch'
  {%- endset %}
  {% if strategy not in ['merge', 'insert_overwrite', 'microbatch'] %}
    {% do exceptions.raise_compiler_error(invalid_strategy_msg) %}
  {% endif %}

  {% if strategy == 'microbatch' %}
    {% do bq_validate_microbatch_config(config) %}
  {% endif %}

  {% do return(strategy) %}
{% endmacro %}

Expected Behavior

The expected behavior is that it checks for custom incremental strategies as in the doc I shared and set it as valid in case it exists.

Steps To Reproduce

  1. dbt cloud (versionless env)
  2. Use an environment connected to a BigQuery project
  3. Create a custom incremental strategy macro (mine was named get_incremental_merge_upsert_sql)
  4. Create an incremental model based on that, as an example:
{{ config(
    materialized='incremental',
    incremental_strategy='merge_upsert',
    unique_key='id'
) }}

  1. Build the model

Relevant log output

Invalid incremental strategy provided: merge_upsert
      Expected one of: 'merge', 'insert_overwrite', 'microbatch'
  
  > in macro dbt_bigquery_validate_get_incremental_strategy (macros/materializations/incremental.sql)
  > called by macro materialization_incremental_bigquery (macros/materializations/incremental.sql)
  > called by model stg_mymodel (models/stg/stg_mymodel.sql)


### Environment

```markdown
- dbt cloud versionless

Additional Context

No response

@jackodes jackodes added type:bug Something isn't working triage:product labels Nov 14, 2024
@amychen1776 amychen1776 added type:enhancement New feature or request and removed triage:product type:bug Something isn't working labels Nov 18, 2024
@amychen1776 amychen1776 changed the title [Bug] Custom incremental strategies do not work with BigQuery [Feature] Custom incremental strategies supported on BigQuery Nov 18, 2024
@amychen1776
Copy link

Hello @jackodes! Unfortunately we actually don't support Custom Incremental Stratergies on BQ at this time. Please see this issue that covers the remaining work dbt-labs/dbt-core#9290

What I'm going to do is actually close this issue for the time (since we are tracking on that dbt-core issue and this is a duplicate) and get our docs updated accordingly to call out this.

@amychen1776 amychen1776 closed this as not planned Won't fix, can't repro, duplicate, stale Nov 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type:enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants