Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sort DBs Fly Datasette; don't distribute Datasette's metadata.yml #3106

Merged
merged 3 commits into from
Dec 1, 2023
Merged

Sort DBs Fly Datasette; don't distribute Datasette's metadata.yml #3106

merged 3 commits into from
Dec 1, 2023

Conversation

zaneselvans
Copy link
Member

PR Overview

  • Remove the Datasette-specific metadata.yml from the outputs folder before uploading to GCS/S3 for distribution.
  • Manually sort SQLite DBs in Fly.io datasette script so that PUDL comes first.

PR Checklist

  • Merge the most recent version of the branch you are merging into (probably dev).
  • All CI checks are passing. Run tests locally to debug failures
  • Make sure you've included good docstrings.
  • For major data coverage & analysis changes, run data validation tests
  • Include unit tests for new functions and classes.
  • Defensive data quality/sanity checks in analyses & data processing functions.
  • Update the release notes and reference reference the PR and related issues.
  • Do your own explanatory review of the PR to help the reviewer understand what's going on and identify issues preemptively.

@zaneselvans zaneselvans self-assigned this Dec 1, 2023
@zaneselvans zaneselvans added the datasette Issues related the accessing PUDL data via Datasette. label Dec 1, 2023
@zaneselvans zaneselvans linked an issue Dec 1, 2023 that may be closed by this pull request
Make nightly build outputs easier to download and access remotely #3079
Closed
@zaneselvans zaneselvans marked this pull request as ready for review December 1, 2023 16:00
jdangerx
jdangerx reviewed Dec 1, 2023
View reviewed changes
devtools/datasette/fly/run.sh
@@ -7,4 +7,4 @@ find /data/ -name '*.sqlite' -delete
mv all_dbs.tar.zst /data
zstd -f -d /data/all_dbs.tar.zst -o /data/all_dbs.tar
tar -xf /data/all_dbs.tar --directory /data
datasette serve --host 0.0.0.0 /data/*.sqlite --cors --inspect-file inspect-data.json --metadata metadata.yml --setting sql_time_limit_ms 5000 --port $PORT
datasette serve --host 0.0.0.0 /data/pudl.sqlite /data/ferc*.sqlite /data/censusdp1tract.sqlite --cors --inspect-file inspect-data.json --metadata metadata.yml --setting sql_time_limit_ms 5000 --port $PORT
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎉

@zaneselvans zaneselvans merged commit 29a7443 into dev Dec 1, 2023
6 checks passed
@zaneselvans zaneselvans deleted the better-build-outputs branch December 25, 2023 17:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jdangerx jdangerx jdangerx approved these changes

Assignees

@zaneselvans zaneselvans

Labels
datasette Issues related the accessing PUDL data via Datasette.
Projects
Catalyst Megaproject
Archived in project
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Make nightly build outputs easier to download and access remotely
2 participants
@zaneselvans @jdangerx