Use Google Batch for full ETL runs #3211

jdangerx · 2024-01-03T22:57:15Z

Overview

Closes #3208 .

This lets us kick off nightly builds via Google Batch.

Opens up:

prompter/searchable/filterable logging
scaling up/down the instances easily
kicking off multiple builds at once without stepping on each other's toes
trying to use spot instances - we'd probably want to set this job up in a more "restartable" way, so that we don't end up having to re-run the whole dang thing when our VM gets interrupted

What did you change?

add script to create batch configuration, based on the same CLI args we were using before for VM configuration
run Postgres within the container, so we can stop running the Cloud SQL instance
slightly nicer Slack message outputs
exit 1 on failure so that the batch job dashboard correctly identifies a failure

Testing

How did you make sure this worked? How can a reviewer verify this?

Ran lots of nightly builds using Batch :) You can kick off your own via workflow_dispatch.

To-do list

Give feedback

Make sure full ETL runs & make pytest-integration-full passes locally
Review the PR yourself and call out any questions or issues you have
remove the Cloud SQL instances + Compute Engine instances once this gets merged in
Options

devtools/generate_batch_config.py

review-notebook-app · 2024-01-23T16:51:10Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

jdangerx · 2024-01-26T20:57:56Z

.github/workflows/build-deploy-pudl.yml

-            --container-env DAGSTER_PG_USERNAME="postgres" \
-            --container-env DAGSTER_PG_PASSWORD="$DAGSTER_PG_PASSWORD" \
-            --container-env DAGSTER_PG_HOST="104.154.182.24" \
-            --container-env DAGSTER_PG_DB="dagster-storage" \


These guys got moved into docker/dagster.yaml because:

I wanted all the dagster postgres configs in one place

dagster postgres port configuration has to be in dagster.yaml, otherwise we run into "port is a string, not an int" issues

.github/workflows/build-deploy-pudl.yml

jdangerx · 2024-01-26T21:07:47Z

docker/gcp_pudl_etl.sh


    send_slack_msg "$message"
+    upload_file_to_slack "$LOGFILE" "pudl_etl logs for $BUILD_ID:"


Do we want to keep this here? It's kind of nice to be able to click and download logs right from the failure notification. But we can already just click that Google Batch Console link...

Maybe I just don't know how to use it correctly but I have found the logs on the console annoying to find anything in and they only show a few lines no matter how bit the window is. And I can't just click on the link because then it opens in the wrong browser window, or under the wrong Google ID, so I have to copy link, open a tab in the right window in the right user container, paste the URL and then get an interface I don't like anyway.

The thing I'd like to have is direct links / URLs for both the output directory and the logfile, with the $BUILD_ID visible rather than hidden in a link:

Outputs: gs://builds.catalyst.coop/${BUILD_ID}/

Logfile: gs://builds.catalyst.coop/${BUILD_ID}/${BUILD_ID}-pudl-etl.log

so I can pull the logfile down with gsutil cp and open it in my editor / grep through it, and know without having to hover over it what build it's from. I find the "download file" dialog in Slack a bit involved, but it's probably more accessible to the less cloudy folks.

Another potential improvement, especially with our newfound ability to run an arbitrary number of builds all at the same time, would be to make sure that every slack message includes the $BUILD_ID so that we know which build the messages pertain to, since if there's more than one going at a time they get interleaved with each other and that can be very confusing. Or if there were some way to group all of the messages that pertain to a given build in a single thread that would be even better.

I guess what I'm saying with the links to logs & outputs is different people will prefer different access methods so maybe we should just provide them all.

jdangerx · 2024-01-26T21:08:36Z

devtools/generate_batch_config.py

+
+    # NOTE (daz): the best documentation of the actual data structure I've found is at
+    # https://cloud.google.com/python/docs/reference/batch/latest/google.cloud.batch_v1.types.Job
+    config = {


I briefly thought about doing some dataclass thing here instead of a dictionary, but that seemed super verbose so I abandoned that idea.

zaneselvans

Looks great to the extent that I understand what it's doing :)

I have some desires around what shows up in the Slack notifications so we can easily tell which build a message is coming from and directly access the outputs.

devtools/generate_batch_config.py

zaneselvans · 2024-01-26T21:58:29Z

docker/gcp_pudl_etl.sh


    send_slack_msg "$message"
+    upload_file_to_slack "$LOGFILE" "pudl_etl logs for $BUILD_ID:"


Maybe I just don't know how to use it correctly but I have found the logs on the console annoying to find anything in and they only show a few lines no matter how bit the window is. And I can't just click on the link because then it opens in the wrong browser window, or under the wrong Google ID, so I have to copy link, open a tab in the right window in the right user container, paste the URL and then get an interface I don't like anyway.

The thing I'd like to have is direct links / URLs for both the output directory and the logfile, with the $BUILD_ID visible rather than hidden in a link:

Outputs: gs://builds.catalyst.coop/${BUILD_ID}/

Logfile: gs://builds.catalyst.coop/${BUILD_ID}/${BUILD_ID}-pudl-etl.log

so I can pull the logfile down with gsutil cp and open it in my editor / grep through it, and know without having to hover over it what build it's from. I find the "download file" dialog in Slack a bit involved, but it's probably more accessible to the less cloudy folks.

Another potential improvement, especially with our newfound ability to run an arbitrary number of builds all at the same time, would be to make sure that every slack message includes the $BUILD_ID so that we know which build the messages pertain to, since if there's more than one going at a time they get interleaved with each other and that can be very confusing. Or if there were some way to group all of the messages that pertain to a given build in a single thread that would be even better.

jdangerx · 2024-01-29T16:02:26Z

I think I know how to get the Slack notifications working even better but I figured that was out of scope for this PR. Just decided to update the wording & make sure that you can (a) click the links and go straight to the correct console page (b) copy the links directly into gsutil or gcloud storage commands.

zaneselvans · 2024-01-29T16:33:44Z

I note that since we we distribute the log file associated with the nightly builds, it's also possible to provide a clickable HTTP download link, though that would only work if the build has gotten as far as distributing the outputs. In which case I guess we'll usually not need to look at the logs.

https://s3.us-west-2.amazonaws.com/pudl.catalyst.coop/nightly/2024-01-27-0602-52a2741ff-main-pudl-etl.log

Another nice bonus of giving the logfile a readable and unique name is we can now at a glance see what build the nightlies came from.

jdangerx · 2024-01-29T17:23:23Z

Yeah, I didn't realize what I didn't have until I had it, but being able to see both the date & the git hash is 🎆

jdangerx linked an issue Jan 3, 2024 that may be closed by this pull request

Port nightly build process to Google Batch #3208

Closed

jdangerx force-pushed the try-google-batch branch 3 times, most recently from 2b6ab6c to da11dc9 Compare January 9, 2024 18:40

zaneselvans added cloud Stuff that has to do with adapting PUDL to work in cloud computing context. nightly-builds Anything having to do with nightly builds or continuous deployment. labels Jan 9, 2024

jdangerx force-pushed the try-google-batch branch 2 times, most recently from 9670eac to 0399687 Compare January 10, 2024 17:42

zaneselvans added this to the v2024.01 milestone Jan 12, 2024

jdangerx force-pushed the try-google-batch branch 3 times, most recently from c23a4ac to ee423f1 Compare January 16, 2024 20:54

zaneselvans reviewed Jan 17, 2024

View reviewed changes

devtools/generate_batch_config.py Outdated Show resolved Hide resolved

jdangerx force-pushed the try-google-batch branch 2 times, most recently from 7196e2a to 7c26216 Compare January 23, 2024 16:26

jdangerx force-pushed the try-google-batch branch 6 times, most recently from 96e4479 to e842d7b Compare January 26, 2024 20:58

jdangerx commented Jan 26, 2024

View reviewed changes

jdangerx force-pushed the try-google-batch branch from e842d7b to 2c8bb16 Compare January 26, 2024 21:09

jdangerx requested review from bendnorman and zaneselvans January 26, 2024 21:16

jdangerx force-pushed the try-google-batch branch from 2c8bb16 to 39569c2 Compare January 26, 2024 21:21

jdangerx changed the title ~~WIP: Try google batch~~ Use Google Batch for full ETL runs Jan 26, 2024

zaneselvans requested changes Jan 26, 2024

View reviewed changes

zaneselvans assigned jdangerx Jan 26, 2024

jdangerx added 6 commits January 29, 2024 10:55

Create batch job on push to try-google-batch branch

2c135ba

Add debugging output for docker image.

31a1bb7

Attempt to use in-container postgres

69ec683

More time, less disk space

672961e

Terminate Postgres instance so container can exit.

8a337f1

Update slack language

13416cf

jdangerx force-pushed the try-google-batch branch from 39569c2 to 13416cf Compare January 29, 2024 15:59

jdangerx marked this pull request as ready for review January 29, 2024 16:00

jdangerx requested a review from zaneselvans January 29, 2024 16:02

zaneselvans approved these changes Jan 29, 2024

View reviewed changes

jdangerx enabled auto-merge (squash) January 29, 2024 17:23

jdangerx merged commit c48766d into main Jan 29, 2024
17 checks passed

jdangerx deleted the try-google-batch branch January 29, 2024 17:35

bendnorman mentioned this pull request Jan 31, 2024

Spin up a postgres database within our nightly build container #3003

Closed

bendnorman mentioned this pull request May 13, 2024

Deploy multiple images to multiple VMs #1649

Closed

zaneselvans mentioned this pull request Sep 26, 2024

Dagster SQLite EventLog failures #3868

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use Google Batch for full ETL runs #3211

Use Google Batch for full ETL runs #3211

jdangerx commented Jan 3, 2024 •

edited

Loading

To-do list

review-notebook-app bot commented Jan 23, 2024

jdangerx Jan 26, 2024

jdangerx Jan 26, 2024

zaneselvans Jan 26, 2024 •

edited

Loading

zaneselvans Jan 26, 2024

jdangerx Jan 26, 2024

zaneselvans left a comment

zaneselvans Jan 26, 2024 •

edited

Loading

jdangerx commented Jan 29, 2024

zaneselvans commented Jan 29, 2024

jdangerx commented Jan 29, 2024


		send_slack_msg "$message"
		upload_file_to_slack "$LOGFILE" "pudl_etl logs for $BUILD_ID:"

Use Google Batch for full ETL runs #3211

Use Google Batch for full ETL runs #3211

Conversation

jdangerx commented Jan 3, 2024 • edited Loading

Overview

Testing

To-do list

review-notebook-app bot commented Jan 23, 2024

jdangerx Jan 26, 2024

Choose a reason for hiding this comment

jdangerx Jan 26, 2024

Choose a reason for hiding this comment

zaneselvans Jan 26, 2024 • edited Loading

Choose a reason for hiding this comment

zaneselvans Jan 26, 2024

Choose a reason for hiding this comment

jdangerx Jan 26, 2024

Choose a reason for hiding this comment

zaneselvans left a comment

Choose a reason for hiding this comment

zaneselvans Jan 26, 2024 • edited Loading

Choose a reason for hiding this comment

jdangerx commented Jan 29, 2024

zaneselvans commented Jan 29, 2024

jdangerx commented Jan 29, 2024

jdangerx commented Jan 3, 2024 •

edited

Loading

zaneselvans Jan 26, 2024 •

edited

Loading

zaneselvans Jan 26, 2024 •

edited

Loading