Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] dbt run against spark profile checks for database even in unrelated sources #849

Open
3 of 11 tasks
soham-dasgupta opened this issue Feb 25, 2025 · 0 comments
Open
3 of 11 tasks
Labels
triage:product In Product's queue type:bug Something isn't working as documented

Comments

@soham-dasgupta
Copy link

Is this a new bug?

  • I believe this is a new bug
  • I have searched the existing issues, and I could not find an existing issue for this bug

Which packages are affected?

  • dbt-adapters
  • dbt-tests-adapter
  • dbt-athena
  • dbt-athena-community
  • dbt-bigquery
  • dbt-postgres
  • dbt-redshift
  • dbt-snowflake
  • dbt-spark

Current Behavior

sources.yml

  - name: spark
    description: glue default database
    schema: default
    tables:
      - name: fact_customer_item_dly
        description: fact_customer_item_dly

profiles.yml

sparkprofile:
  target: dev
  outputs:
    dev:
      type: spark
      method: thrift
      host: localhost
      port: 10001
      schema: dev

spark_test.sql

{{ config(
    materialized='table',
    file_format='parquet'
) }}


select count(*) from {{ source('spark' ,'fact_customer_item_dly') }}

.venv/bin/dbt run --select spark_test.sql --profile sparkprofile errors with Cannot set database in spark!

Expected Behavior

With the above sources.yml, profiles.yml .venv/bin/dbt run --select spark_test.sql --profile sparkprofile should run without any error as source with name spark do not have any database specified

Steps To Reproduce

Files sources.yml and profiles.yml should be as above

Relevant log output

N/A

Environment

- OS: MACOS
- Python: 3.11.11
- dbt-spark: 1.8.0

Additional Context

No response

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
triage:product In Product's queue type:bug Something isn't working as documented
Projects
None yet
Development

No branches or pull requests

1 participant