Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

consolidate psf and flat jobs #2371

Open
sbailey opened this issue Sep 12, 2024 · 0 comments
Open

consolidate psf and flat jobs #2371

sbailey opened this issue Sep 12, 2024 · 0 comments
Labels

Comments

@sbailey
Copy link
Contributor

sbailey commented Sep 12, 2024

One of the challenges of running kibo was the sheer number of jobs, ~50k. This required extra queue management to avoid submit limits, made rare startup problems common to encounter, and added overhead to reservations which sometimes had ~10% of the nodes idle while turning over jobs. To reduce the number of jobs,

  • bundle arc and psfnight jobs together for a 6 -> 1 reduction for those steps. Consider also bundling the ccdcalib job into this bunch.
  • bundle flat and nightlyflat jobs together for a 13 -> 1 reduction of those steps

Although there are still more science jobs than there are arc/psf/flat/nightlyflat jobs, these calib jobs are tightly coupled and blocking factors for getting to the more genuinely independent tilenight and ztile jobs, so bundling them would streamline the workflow.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant