-
Notifications
You must be signed in to change notification settings - Fork 160
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add staging CD jobs #23
Conversation
…or BQ (#18) Testing CI jobs, we want both BQ and SF to run here. --------- Co-authored-by: Winnie Winship <[email protected]> Co-authored-by: dave-connors-3 <[email protected]>
There is a race condition using multiple warehouses for CI jobs on the same repo. This adds GitHub Actions to run the CI jobs so we can check against all of them. It also adds slim CD jobs that run on merge to main.
Closes #1 This turned into pulling a sweater thread that unravelled some issues in the source data, so I had to dig in to quite a bit! This updates some issues in the source data seed files in addition to fixing logic.
The Semantic Layer has an awesome feature to allow the construction and materialization of queries based on your metrics. We've added those to the Jaffle Shop and expanded the SL implementation a bit.
Is this because there are models whose dependencies are not being forced? Wouldn't it be better to force the dependency than babysit which nodes get built outside of dbt itself? |
so @joellabes seeds run in parallel with everything else, if a source is pointed at a seed's output, and the seed is...seeding into the db still, bad stuff happens. there are no models to force the deps on, as they're sources rather than models. the Big Fix would be if seeds were treated as a proper part of the DAG, which there's an issue for and i've brought up with Doug and Jerco. i think that would make a lot of sense. i have an alternative fix i'm toying with i'll send you as a Loom, would be interested in your thoughts on it, but it's more of a tradeoff than a fix. |
Wouldn't this work?
#models/sources.yml
sources:
- name: seeds
tables:
- name: my_seed
# etc -- models/stg_my_seed.sql
-- depends_on: {{ ref('my_seed') }}
with seed as (
select * from {{ source('seeds', 'my_seed') }}
)
-- etc Maybe not a good idea as I write it out if this is meant to be a best practice demo |
@joellabes oh yea i think you're right that should work, but yea maybe not ideal for beginners 🤔 hadn't thought of that though |
Adds GHAs to call the Staging Deploy jobs for BQ and SF on merge to
staging
.This supports the change to make Staging the default branch in Jaffle Shop (already done), and completes the process of getting us to the following state: