Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Port nightly build process to Google Batch #3208

Closed
2 tasks
Tracked by #3207
jdangerx opened this issue Jan 3, 2024 · 2 comments · Fixed by #3211
Closed
2 tasks
Tracked by #3207

Port nightly build process to Google Batch #3208

jdangerx opened this issue Jan 3, 2024 · 2 comments · Fixed by #3211
Labels
cloud Stuff that has to do with adapting PUDL to work in cloud computing context. nightly-builds Anything having to do with nightly builds or continuous deployment.
Milestone

Comments

@jdangerx
Copy link
Member

jdangerx commented Jan 3, 2024

This lets us run multiple tag builds at once, and provides an environment where the dang logs show up on time.

Let's try to leave the rest of the system as untouched as possible.

Success criteria

@jdangerx jdangerx changed the title port nightly build process to Google Batch with minimal changes Port nightly build process to Google Batch Jan 3, 2024
@jdangerx jdangerx moved this from New to In progress in Catalyst Megaproject Jan 3, 2024
@jdangerx jdangerx linked a pull request Jan 3, 2024 that will close this issue
@zaneselvans zaneselvans added cloud Stuff that has to do with adapting PUDL to work in cloud computing context. nightly-builds Anything having to do with nightly builds or continuous deployment. labels Jan 8, 2024
@zaneselvans zaneselvans added this to the v2024.01 milestone Jan 12, 2024
@jdangerx
Copy link
Member Author

As of now, the batch job seems to run the ETL successfully, but doesn't cleanly exit & terminate the batch job when the ETL fails - need to investigate that more.

@bendnorman
Copy link
Member

It seems like the VM should terminate when the container exits :/ You could use our workaround from our existing set up where we make an API call from within the VM to shut down itself down. Pretty janky but an option!

@jdangerx jdangerx moved this from In progress to In review in Catalyst Megaproject Jan 26, 2024
@github-project-automation github-project-automation bot moved this from In review to Done in Catalyst Megaproject Jan 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cloud Stuff that has to do with adapting PUDL to work in cloud computing context. nightly-builds Anything having to do with nightly builds or continuous deployment.
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

3 participants