Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add schema checks #89

Merged
merged 9 commits into from
May 14, 2024
Merged

Add schema checks #89

merged 9 commits into from
May 14, 2024

Conversation

cnolanminich
Copy link
Contributor

@cnolanminich cnolanminich commented May 1, 2024 •
edited
Loading

This PR does a two things:

  • boring upkeep
    • bumps Dagster version to 1.7.5
    • fixes freshness checks to use the new list return value present in 1.7.5
    • pins dbt-core to 1.7.x since 1.8 was released yesterday and breaks our build until dagster-dbt is upgraded
  • (the main event!) adds schema checks

Schema checks:

This PR currently adds schema checks to two raw assets and all dbt assets. Points for feedback:

  • is it bad to add a build_column_schema_change_checks to the definitions.py module? Is there any weird overhead / issues to think about?
  • this works locally, will mark as ready for review once next Dagster release is out.

Pre:
image

After casting order_date to a string 🎉

image

@github-actions GitHub Actions
Copy link

github-actions bot commented May 1, 2024 •
edited
Loading

Your pull request at commit e180a8ffdd69d6ac73b7ed3217d7233e5519b631 is automatically being deployed to Dagster Cloud.

Location Status Link Updated
demo_assets View in Cloud May 01, 2024 at 02:18 PM (UTC)
batch_enrichment View in Cloud May 01, 2024 at 02:18 PM (UTC)
data-eng-pipeline View in Cloud May 01, 2024 at 02:18 PM (UTC)
basics View in Cloud May 01, 2024 at 02:18 PM (UTC)
snowflake_insights View in Cloud May 01, 2024 at 02:18 PM (UTC)

@cnolanminich cnolanminich changed the title Add schema checks -- draft not working Add schema checks May 1, 2024
@github-actions GitHub Actions
Copy link

github-actions bot commented May 1, 2024 •
edited
Loading

Your pull request at commit 23f6e3e1fb79b9e30f34015909725358084ba6c8 is automatically being deployed to Dagster Cloud.

Location Status Link Updated
demo_assets Deploy failed May 01, 2024 at 09:17 PM (UTC)
batch_enrichment Deploy failed May 01, 2024 at 09:17 PM (UTC)
data-eng-pipeline Deploy failed May 01, 2024 at 09:17 PM (UTC)
basics Deploy failed May 01, 2024 at 09:17 PM (UTC)
snowflake_insights Deploy failed May 01, 2024 at 09:17 PM (UTC)

@github-actions GitHub Actions
Copy link

github-actions bot commented May 2, 2024 •
edited
Loading

Your pull request at commit 1487d57666dc08f84f6b4d3e84c96145c5dc22a2 is automatically being deployed to Dagster Cloud.

Location Status Link Updated
demo_assets Deploy failed May 03, 2024 at 12:04 AM (UTC)
batch_enrichment Deploy failed May 03, 2024 at 12:04 AM (UTC)
data-eng-pipeline Deploy failed May 03, 2024 at 12:04 AM (UTC)
basics Deploy failed May 03, 2024 at 12:04 AM (UTC)
snowflake_insights Deploy failed May 03, 2024 at 12:04 AM (UTC)

@github-actions GitHub Actions
Copy link

github-actions bot commented May 9, 2024

Your pull request at commit 589ef3d0a5cd65e2e6085e6073d4d47a37b3c63e is automatically being deployed to Dagster Cloud.

Location Status Link Updated
demo_assets Building... May 09, 2024 at 06:56 PM (UTC)
batch_enrichment Building... May 09, 2024 at 06:56 PM (UTC)
data-eng-pipeline Building... May 09, 2024 at 06:56 PM (UTC)
basics Building... May 09, 2024 at 06:56 PM (UTC)
snowflake_insights Building... May 09, 2024 at 06:56 PM (UTC)

@github-actions GitHub Actions
Copy link

github-actions bot commented May 9, 2024

Your pull request at commit 18b9bb61504e16e849be403e7bf8d2b1143bb374 is automatically being deployed to Dagster Cloud.

Location Status Link Updated
demo_assets Building... May 09, 2024 at 07:04 PM (UTC)
batch_enrichment Building... May 09, 2024 at 07:04 PM (UTC)
data-eng-pipeline Building... May 09, 2024 at 07:04 PM (UTC)
basics Building... May 09, 2024 at 07:04 PM (UTC)
snowflake_insights Building... May 09, 2024 at 07:04 PM (UTC)

@github-actions GitHub Actions
Copy link

github-actions bot commented May 9, 2024

Your pull request at commit b2b2d094478740bec1e26bdf8019211f926198bc is automatically being deployed to Dagster Cloud.

Location Status Link Updated
demo_assets Building... May 09, 2024 at 07:12 PM (UTC)
batch_enrichment Building... May 09, 2024 at 07:12 PM (UTC)
data-eng-pipeline Building... May 09, 2024 at 07:12 PM (UTC)
basics Building... May 09, 2024 at 07:12 PM (UTC)
snowflake_insights Building... May 09, 2024 at 07:12 PM (UTC)

@github-actions GitHub Actions
Copy link

github-actions bot commented May 9, 2024 •
edited
Loading

Your pull request at commit 089a93e8fd8810d1b75c1b37b95f27dd9cb5e3de is automatically being deployed to Dagster Cloud.

Location Status Link Updated
demo_assets Deploy failed May 09, 2024 at 07:24 PM (UTC)
batch_enrichment Deploy failed May 09, 2024 at 07:24 PM (UTC)
data-eng-pipeline Deploy failed May 09, 2024 at 07:24 PM (UTC)
basics Deploy failed May 09, 2024 at 07:24 PM (UTC)
snowflake_insights Deploy failed May 09, 2024 at 07:24 PM (UTC)

@github-actions GitHub Actions
Copy link

github-actions bot commented May 10, 2024 •
edited
Loading

Your pull request at commit b886fe62623e3fb5bddbcb28e47f4a01f205569b is automatically being deployed to Dagster Cloud.

Location Status Link Updated
demo_assets View in Cloud May 10, 2024 at 01:51 PM (UTC)
batch_enrichment View in Cloud May 10, 2024 at 01:51 PM (UTC)
data-eng-pipeline View in Cloud May 10, 2024 at 01:51 PM (UTC)
basics View in Cloud May 10, 2024 at 01:51 PM (UTC)
snowflake_insights View in Cloud May 10, 2024 at 01:51 PM (UTC)

@cnolanminich cnolanminich marked this pull request as ready for review May 10, 2024 14:22
@cnolanminich cnolanminich merged commit 61abddf into master May 14, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@slopp slopp slopp approved these changes

@izzye84 izzye84 Awaiting requested review from izzye84

@thomasganka thomasganka Awaiting requested review from thomasganka

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@cnolanminich @slopp