feat(dbt): load dbt scaffold using package_data
#15502
Conversation
|
Current dependencies on/for this PR: This comment was auto-generated by Graphite. |
rexledesma
force-pushed
the
rl/install-dbt-project-as-part-of-package
branch
2 times, most recently
from
dbbb5c1
to
1f508a6
Compare
rexledesma
force-pushed
the
rl/install-dbt-project-as-part-of-package
branch
from
1f508a6
to
509c470
Compare
| {% if use_dbt_project_package_data_dir -%} | ||
| package_data={ | ||
| "{{ project_name }}": [ | ||
| "dbt-project/*", |
There was a problem hiding this comment.
fyi not sure if you expected nested directories here but this might only do one level by default.
There was a problem hiding this comment.
I did expect nested directories here, but looks like /* worked as I expected, since it built in cloud. But I'll use a recursive glob here, since that seems to be supported as well, and makes this behavior explicit: https://setuptools.pypa.io/en/latest/history.html#v62-3-0
| @@ -89,6 +90,9 @@ def copy_scaffold( | |||
| for target in profile["outputs"].values() | |||
| ] | |||
|
|
|||
| if use_dbt_project_package_data_dir: | |||
There was a problem hiding this comment.
I think this should just use the passed in dbt_project_dir and just error if the dbt_project_dir is not under the start path
The scaffold can run locally without package data. However, it is required when you deploy. I was planning on adding documentation to explain the changes required to deploy a Dagster + dbt project in a Python package to cloud. But to shield the user from this complexity, by default we don't enable this in the open source CLI. |
rexledesma
force-pushed
the
rl/install-dbt-project-as-part-of-package
branch
from
509c470
to
6c3d6b2
Compare
## Summary & Motivation Allow `dagster-dbt project scaffold` to load dbt projects from their path in `package_data`. This will be invoked in the dbt NUX. We hardcode `dbt-project` as the dbt project directory name for the `package_data`. This restraint could be loosened in the future. ## How I Tested These Changes - pytest - dbt NUX creates a pull request that builds to branch deployment: https://github.com/dagster-io/test-dagster-dbt-scaffold/pull/53 - Github actions that build a Dagster package utilizing `package_data` dagster-io/dagster-cloud-action#147
Summary & Motivation
Allow
dagster-dbt project scaffoldto load dbt projects from their path inpackage_data. This will be invoked in the dbt NUX.We hardcode
dbt-projectas the dbt project directory name for thepackage_data. This restraint could be loosened in the future.How I Tested These Changes
package_datachore: add steps to build Python package with dbt project dagster-cloud-action#147