Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Specifying compute for Python models #890

Open
vzhyruk opened this issue Dec 23, 2024 · 0 comments
Open

Specifying compute for Python models #890

vzhyruk opened this issue Dec 23, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@vzhyruk
Copy link

vzhyruk commented Dec 23, 2024

Describe the bug

https://docs.getdbt.com/reference/resource-configs/databricks-configs#specifying-compute-for-python-models
Documentation says:
"When you specify your databricks_compute for a python model, you are currently only specifying which compute to use when running the model-specific SQL. If you wish to use a different compute for executing the python itself, you must specify an alternate compute in the config for the model."

This behaviour is misleading and not intuitive. And also requires to additionally specify same computes several times in profiles and model configurations which breaks the usage of different profiles and targets in your project.

Steps To Reproduce

  1. Secondary databricks compute is set as all_purpose compute
    profiles.yml:
<profile>:
  target: <target>
  outputs:
    <target>:
      type: databricks
      ..........
      http_path: <default_compute>
      compute:
        secondary_compute:
          http_path: <secondary_all_purpose_compute>
  1. Compute is set for subfolder with models which include python models as well.
    dbt_project.yml:
<subfolder>:
      .........
      +databricks_compute: secondary_compute
  1. Run models
  2. Python models are executed against default compute regardless of the 'databricks_compute' config

Expected behavior

When 'databricks_compute' is set for python models and it is all purpose compute they should use it for both Python and SQL parts.

@vzhyruk vzhyruk added the bug Something isn't working label Dec 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant