Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unit Tests in 1.8.1 do not honor quoting on the descriptors for columns #10269

Closed
2 tasks done
SlobberingHorde opened this issue Jun 6, 2024 · 1 comment
Closed
2 tasks done
Labels
bug Something isn't working duplicate This issue or pull request already exists unit tests Issues related to built-in dbt unit testing functionality

Comments

@SlobberingHorde
Copy link

SlobberingHorde commented Jun 6, 2024

Is this a new bug in dbt-core?

  • I believe this is a new bug in dbt-core
  • I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

I have a pipeline using dbt-core 1.8.1 and dbt-postgres 1.8.1. The source data is imported from a SQL Server system so the column and table names have cased names.

I created a simple unit tests for a coalesce and it fails with

13:18:17.594720 [debug] [Thread-4 (]: Postgres adapter: Postgres error: column i.InterestCalculationMethodId does not exist
LINE 48:         JOIN __dbt__cte__Contract c ON i."InterestCalculatio...

The rest of the pipeline works fine, just the unit test fails.

Expected Behavior

The expectation is that if you have quoted identifiers configured that the unit test would properly handle those columns.

Steps To Reproduce

dbt_project.yml contains

quoting:
  identifier: true

The model is defined as such

-- depends_on: {{ ref('Contract') }}

{{
    config(
        materialized='incremental',
        unique_key='"InterestCalculationMethodId"',
    )
}}


SELECT i."InterestCalculationMethodId", 
        i."AccrualMethod", 
        COALESCE(i."BasisPointsOverRate", 0) AS "BasisPointsOverRate", 
        true AS "ConsiderLeapYear", --TODO: where is this in contracts
        i."InterestCalculationType"
	FROM {{ ref('int_interest') }} i
        JOIN {{ ref('Contract') }} c ON i."InterestCalculationMethodId" = c."ContractId"

The yaml descriptor is

version: 2
      
models:
  - name: InterestCalculationMethod
    description: this model set the interest calculation method for each contract.
    columns:
      - name: "InterestCalculationMethodId"
        quote: true
        description: interest calculation method unique id.
        data_tests:
          - not_null
          - unique
      - name: "AccrualMethod"
        quote: true
        data_tests:
          - not_null
      - name: "BasisPointsOverRate"
        quote: true
        data_tests:
          - not_null
      - name: "InterestCalculationType"
        quote: true
        data_tests:
          - not_null                
          
unit_tests:
  - name: test_basis_point_defaults_to_zero_when_blank
    model: InterestCalculationMethod
    given:
      - input: ref('int_interest')
        rows:
          - {"InterestCalculationMethodId": 255, "AccrualMethod": 360, "BasisPointsOverRate": null, "InterestCalculationType": 0}
      - input: ref('Contract')
        rows:
         - {"ContractId": 255}
    expect:
      rows:
        - {"InterestCalculationMethodId": 255, "AccrualMethod": 360, "BasisPointsOverRate": 0, "ConsiderLeapYear": true, "InterestCalculationType": 0}

Relevant log output

13:18:17.383422 [debug] [Thread-4 (]: Began running node unit_test.bootstrapping.InterestCalculationMethod.test_basis_point_defaults_to_zero_when_blank
13:18:17.384675 [info ] [Thread-4 (]: 153 of 202 START unit_test InterestCalculationMethod::test_basis_point_defaults_to_zero_when_blank  [RUN]
13:18:17.385697 [debug] [Thread-4 (]: Re-using an available connection from the pool (formerly model.common_ingestion.int_loans_hfs_and_lthfs, now unit_test.bootstrapping.InterestCalculationMethod.test_basis_point_defaults_to_zero_when_blank)
13:18:17.386600 [debug] [Thread-4 (]: Began compiling node unit_test.bootstrapping.InterestCalculationMethod.test_basis_point_defaults_to_zero_when_blank
13:18:17.387264 [debug] [Thread-4 (]: Began executing node unit_test.bootstrapping.InterestCalculationMethod.test_basis_point_defaults_to_zero_when_blank
13:18:17.449107 [debug] [Thread-4 (]: Using postgres connection "unit_test.bootstrapping.InterestCalculationMethod.test_basis_point_defaults_to_zero_when_blank"
13:18:17.450138 [debug] [Thread-4 (]: On unit_test.bootstrapping.InterestCalculationMethod.test_basis_point_defaults_to_zero_when_blank: BEGIN
13:18:17.451027 [debug] [Thread-4 (]: Opening a new connection, currently in state closed
13:18:17.477692 [debug] [Thread-4 (]: SQL status: BEGIN in 0.0 seconds
13:18:17.478352 [debug] [Thread-4 (]: Using postgres connection "unit_test.bootstrapping.InterestCalculationMethod.test_basis_point_defaults_to_zero_when_blank"
13:18:17.479019 [debug] [Thread-4 (]: On unit_test.bootstrapping.InterestCalculationMethod.test_basis_point_defaults_to_zero_when_blank: /* {"app": "dbt", "dbt_version": "1.8.1", "profile_name": "bootstrapping", "target_name": "dev", "node_id": "unit_test.bootstrapping.InterestCalculationMethod.test_basis_point_defaults_to_zero_when_blank"} */

      select
          column_name,
          data_type,
          character_maximum_length,
          numeric_precision,
          numeric_scale

      from "warehouse".INFORMATION_SCHEMA.columns
      where table_name = 'Contract'
        
        and table_schema = 'app'
        
      order by ordinal_position

  
13:18:17.491857 [debug] [Thread-4 (]: SQL status: SELECT 7 in 0.0 seconds
13:18:17.510688 [debug] [Thread-4 (]: Writing injected SQL for node "unit_test.bootstrapping.InterestCalculationMethod.test_basis_point_defaults_to_zero_when_blank"
13:18:17.520955 [debug] [Thread-4 (]: Using postgres connection "unit_test.bootstrapping.InterestCalculationMethod.test_basis_point_defaults_to_zero_when_blank"
13:18:17.521676 [debug] [Thread-4 (]: On unit_test.bootstrapping.InterestCalculationMethod.test_basis_point_defaults_to_zero_when_blank: /* {"app": "dbt", "dbt_version": "1.8.1", "profile_name": "bootstrapping", "target_name": "dev", "node_id": "unit_test.bootstrapping.InterestCalculationMethod.test_basis_point_defaults_to_zero_when_blank"} */

      select
          column_name,
          data_type,
          character_maximum_length,
          numeric_precision,
          numeric_scale

      from "warehouse".INFORMATION_SCHEMA.columns
      where table_name = 'int_interest'
        
        and table_schema = 'app'
        
      order by ordinal_position

  
13:18:17.526828 [debug] [Thread-4 (]: SQL status: SELECT 4 in 0.0 seconds
13:18:17.530539 [debug] [Thread-4 (]: Writing injected SQL for node "unit_test.bootstrapping.InterestCalculationMethod.test_basis_point_defaults_to_zero_when_blank"
13:18:17.554970 [debug] [Thread-4 (]: Writing injected SQL for node "unit_test.bootstrapping.InterestCalculationMethod.test_basis_point_defaults_to_zero_when_blank"
13:18:17.592226 [debug] [Thread-4 (]: Using postgres connection "unit_test.bootstrapping.InterestCalculationMethod.test_basis_point_defaults_to_zero_when_blank"
13:18:17.593086 [debug] [Thread-4 (]: On unit_test.bootstrapping.InterestCalculationMethod.test_basis_point_defaults_to_zero_when_blank: /* {"app": "dbt", "dbt_version": "1.8.1", "profile_name": "bootstrapping", "target_name": "dev", "node_id": "unit_test.bootstrapping.InterestCalculationMethod.test_basis_point_defaults_to_zero_when_blank"} */

    
  
    

  create temporary table "test_basis_point_defaults_to_zero_when_bla__dbt_tmp131817586734"
  
  
    as
  
  (
    select * from (
        with __dbt__cte__Contract as (

-- Fixture for Contract
select 
    
    cast(255 as bigint)
 as contractid, cast(null as character varying(200)) as name, cast(null as timestamp with time zone) as effectivedate, cast(null as timestamp with time zone) as expirationdate, cast(null as character varying(5)) as mplid, cast(null as integer) as hfsseasoningid, cast(null as integer) as lthfsseasoningid
),  __dbt__cte__int_interest as (

-- Fixture for int_interest
select 
    
    cast(255 as bigint)
 as interestcalculationmethodid, 
    
    cast(0 as integer)
 as interestcalculationtype, 
    
    cast(null as numeric)
 as basispointsoverrate, 
    
    cast(360 as integer)
 as accrualmethod
) -- depends_on: __dbt__cte__Contract




SELECT i."InterestCalculationMethodId", 
        i."AccrualMethod", 
        COALESCE(i."BasisPointsOverRate", 0) AS "BasisPointsOverRate", 
        true AS "ConsiderLeapYear", --TODO: where is this in contracts
        i."InterestCalculationType"
	FROM __dbt__cte__int_interest i
        JOIN __dbt__cte__Contract c ON i."InterestCalculationMethodId" = c."ContractId"
    ) as __dbt_sbq
    where false
    limit 0

  );
  
  
13:18:17.594720 [debug] [Thread-4 (]: Postgres adapter: Postgres error: column i.InterestCalculationMethodId does not exist
LINE 48:         JOIN __dbt__cte__Contract c ON i."InterestCalculatio...
                                                ^

13:18:17.595409 [debug] [Thread-4 (]: On unit_test.bootstrapping.InterestCalculationMethod.test_basis_point_defaults_to_zero_when_blank: ROLLBACK
13:18:17.596499 [debug] [Thread-4 (]: On unit_test.bootstrapping.InterestCalculationMethod.test_basis_point_defaults_to_zero_when_blank: Close
13:18:17.611307 [debug] [Thread-4 (]: Runtime Error in unit_test test_basis_point_defaults_to_zero_when_blank (models/final/InterestCalculationMethod.yml)
  An error occurred during execution of unit test 'test_basis_point_defaults_to_zero_when_blank'. There may be an error in the unit test definition: check the data types.
   Database Error
    column i.InterestCalculationMethodId does not exist
    LINE 48:         JOIN __dbt__cte__Contract c ON i."InterestCalculatio...
                                                    ^
13:18:17.612220 [error] [Thread-4 (]: 153 of 202 ERROR InterestCalculationMethod::test_basis_point_defaults_to_zero_when_blank  [ERROR in 0.23s]
13:18:17.613760 [debug] [Thread-4 (]: Finished running node unit_test.bootstrapping.InterestCalculationMethod.test_basis_point_defaults_to_zero_when_blank
13:18:17.615562 [debug] [Thread-4 (]: Began running node model.bootstrapping.InterestCalculationMethod
13:18:17.616768 [info ] [Thread-4 (]: 155 of 202 SKIP relation app.InterestCalculationMethod ......................... [SKIP]
13:18:17.617823 [debug] [Thread-4 (]: Finished running node model.bootstrapping.InterestCalculationMethod
13:18:17.618658 [debug] [Thread-4 (]: Began running node model.bootstrapping.VolumeFeeCalculationMethod
13:18:17.619907 [info ] [Thread-4 (]: 152 of 202 START sql incremental model app.VolumeFeeCalculationMethod .......... [RUN]
13:18:17.620760 [debug] [Thread-4 (]: Re-using an available connection from the pool (formerly unit_test.bootstrapping.InterestCalculationMethod.test_basis_point_defaults_to_zero_when_blank, now model.bootstrapping.VolumeFeeCalculationMethod)

Environment

- OS: Debian 12
- Python: 3.11.8
- dbt: 1.8.1

Which database adapter are you using with dbt?

postgres

Additional Context

No response

@SlobberingHorde SlobberingHorde added bug Something isn't working triage labels Jun 6, 2024
@dbeatty10 dbeatty10 added the unit tests Issues related to built-in dbt unit testing functionality label Jun 6, 2024
@dbeatty10
Copy link
Contributor

This looks the same as dbt-labs/dbt-adapters#205, so closing as a duplicate.

@dbeatty10 dbeatty10 closed this as not planned Won't fix, can't repro, duplicate, stale Jun 6, 2024
@dbeatty10 dbeatty10 added duplicate This issue or pull request already exists and removed triage labels Jun 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working duplicate This issue or pull request already exists unit tests Issues related to built-in dbt unit testing functionality
Projects
None yet
Development

No branches or pull requests

2 participants