-
Notifications
You must be signed in to change notification settings - Fork 15
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add storage_kind tag #95
Conversation
Your pull request at commit
|
…esn't necessarily have to match the storage
Your pull request at commit
|
I updated all assets (with a couple exceptions) to include storage_kind since the compute_kind doesn't necessarily have to match the storage_kind. That said, dagstermill_assets (model_nb) aren't yet supported and I'm still unsure how we want to represent the k8s storage_kind. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Lgtm!
Your pull request at commit
|
Adding
storage_kind
to non-dbt assets in thehooli_data_eng
code location.Summary of changes:
config_utils.py
to dynamically generate storage_kind (duckdb or snowflake) based on deployment (local or branch/prod)compute_kind
didn't make this obviousOne thing to note, I defaulted the storage_kind to S3 for assets using the
model_io_manager
since it uses theFilesystemIOManager()
locally and S3 in branch and prod deployments.