Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Minor Kibo workflow fixes #2348

Merged
merged 6 commits into from
Aug 27, 2024
Merged

Minor Kibo workflow fixes #2348

merged 6 commits into from
Aug 27, 2024

Conversation

akremin
Copy link
Member

@akremin akremin commented Aug 27, 2024

This PR implements three small changes to the workflow.

First I improved the logging of the processing table caching, which was printing a line for every file it read it. It now prints one line with a list of files it will load in the most concise way possible. This just improves the logging but doesn't impact the downstream jobs at all.

Second I changed the order of operations in desi_submit_prod so that it now checks for the existence of a processing table first before checking the number of jobs in the queue. This allows us to skip hundreds of completed nights without needing to query the queue each time. This resolves Issue #2347 .

Third, I fixed a bud in the calibration submission that arose when cleaning up nights that crashed due to Slurm issues. When some calibrations are in the processing table the code correctly skipped over them but didn't include those jobs in the joint fitting of e.g. the psfnight. This is wrong, as we want all 5 arcs to be included in the psfnight job if they are good. The same situation was happening with flats. This now fixes that by using the existing rows in the processing table and passing the complete set of jobs to the psfnight and nightlyflat job submitter.

I tested this in /global/cfs/cdirs/desi/spectro/redux/test_kibo by sym-linking the calibnight, exposures, preproc, and exposure_tables directors; copying the incomplete processing table from 20211220 currently in Kibo; and running desi_proc_night out of my branch. It identifies the processing table entries and includes them in the joint fit (this currently fails in main). The output of the processing table caching is far fewer lines and works as expected.

Finally, I copied over the config file from Kibo and ran a test of desi_submit_prod, which now checks for processing table existence before checking the queue, and is therefore much faster in getting to nights that actually need to be submitted.

@akremin akremin requested a review from sbailey August 27, 2024 22:52
Copy link
Contributor

@sbailey sbailey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks

@sbailey sbailey merged commit fdbd151 into main Aug 27, 2024
26 checks passed
@sbailey sbailey deleted the kibo_pipe_tweaks branch August 27, 2024 23:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants