Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update sling assets #92

Merged
merged 5 commits into from
May 24, 2024
Merged

Update sling assets #92

merged 5 commits into from
May 24, 2024

Conversation

izzye84
Copy link
Collaborator

@izzye84 izzye84 commented May 17, 2024 •
edited
Loading

Updating the Sling demo to the new APIs.

Summary of updates:

  • added replication python config
  • updated SlingResource to use SlingConnectionResource instead of SlingSourceConnection/SlingTargetConnection (soon to be deprecated)
  • add custom sling translator to modify the default asset key (prefix with "RAW_DATA")
  • updated sling assets to use the sling_assets decorator

There seems to be a bug with local dev when including a target schema with duckdb — "Catalog Error: Schema with name "raw_data" already exists!"

duck_db_target_schema_error

@github-actions GitHub Actions
Copy link

github-actions bot commented May 17, 2024 •
edited
Loading

Your pull request at commit 6d047e86ce1d41e132992cc40c1c8f581170fd11 is automatically being deployed to Dagster Cloud.

Location Status Link Updated
demo_assets View in Cloud May 17, 2024 at 01:10 AM (UTC)
basics View in Cloud May 17, 2024 at 01:10 AM (UTC)
batch_enrichment View in Cloud May 17, 2024 at 01:10 AM (UTC)
data-eng-pipeline View in Cloud May 17, 2024 at 01:10 AM (UTC)
snowflake_insights View in Cloud May 17, 2024 at 01:10 AM (UTC)

@github-actions GitHub Actions
Copy link

Your pull request at commit e01c16d41ce1101b4abcdd1ebcdeb1fb81b8f907 is automatically being deployed to Dagster Cloud.

Location Status Link Updated
demo_assets Building... May 17, 2024 at 03:28 AM (UTC)
basics Building... May 17, 2024 at 03:28 AM (UTC)
batch_enrichment Building... May 17, 2024 at 03:28 AM (UTC)
data-eng-pipeline Building... May 17, 2024 at 03:28 AM (UTC)
snowflake_insights Building... May 17, 2024 at 03:28 AM (UTC)

@github-actions GitHub Actions
Copy link

github-actions bot commented May 17, 2024 •
edited
Loading

Your pull request at commit 50569fd991e0e8433046e872d2b685063ad1e676 is automatically being deployed to Dagster Cloud.

Location Status Link Updated
demo_assets View in Cloud May 17, 2024 at 03:40 AM (UTC)
basics View in Cloud May 17, 2024 at 03:40 AM (UTC)
batch_enrichment View in Cloud May 17, 2024 at 03:40 AM (UTC)
data-eng-pipeline View in Cloud May 17, 2024 at 03:40 AM (UTC)
snowflake_insights View in Cloud May 17, 2024 at 03:40 AM (UTC)

cnolanminich
cnolanminich approved these changes May 21, 2024
View reviewed changes
Copy link
Contributor

@cnolanminich cnolanminich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm! Love keeping our demo up to date with the latest and greatest Dagster APIs. That duckdb issue with sling is driving me nuts, I'll try to take a crack at it at some point

izzye84
izzye84 commented May 21, 2024
View reviewed changes
hooli-demo-assets/hooli_demo_assets/resources/__init__.py
type="duckdb",
instance=f"{DUCKDB_PATH}",
schema="RAW_DATA",
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Commenting out the schema will allow this to run locally—this is the bug I mentioned in the PR description.

@izzye84 @cnolanminich
Update hooli-demo-assets/hooli_demo_assets/resources/__init__.py
3024396 
Adding database param to be consistent with other environments—bug still exists though.

Co-authored-by: Christian Minich <[email protected]>
@github-actions GitHub Actions
Copy link

github-actions bot commented May 24, 2024 •
edited
Loading

Your pull request at commit 3024396e28bc0bde640771e4110d36bf5562c43c is automatically being deployed to Dagster Cloud.

Location Status Link Updated
demo_assets View in Cloud May 24, 2024 at 03:22 PM (UTC)
basics View in Cloud May 24, 2024 at 03:22 PM (UTC)
batch_enrichment View in Cloud May 24, 2024 at 03:22 PM (UTC)
data-eng-pipeline View in Cloud May 24, 2024 at 03:22 PM (UTC)
snowflake_insights View in Cloud May 24, 2024 at 03:22 PM (UTC)

@izzye84 izzye84 merged commit 44694dd into master May 24, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@cnolanminich cnolanminich cnolanminich approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@izzye84 @cnolanminich