Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support custom groups for zproc, e.g. 1x_depth #2309

Merged
merged 2 commits into from
Aug 5, 2024
Merged

support custom groups for zproc, e.g. 1x_depth #2309

merged 2 commits into from
Aug 5, 2024

Conversation

sbailey
Copy link
Contributor

@sbailey sbailey commented Jul 25, 2024

This PR adds support for custom groups of exposures for zproc, e.g. to enable 1x_depth processing through zproc itself, instead of running the underlying commands by hand like was done for previous prods. The key update is adding the concept of a user-specified "subgroup" to findfile and zproc to be used for custom groupings. For pernight and cumulative, subgroup defaults to night, and perexp and healpix also work as before.

Examples in /global/cfs/cdirs/desi/users/sjbailey/spectro/redux/jura1x

# Examples for running 1x_depth
desi_zproc --batch --nosubmit -t 80605 \
    --groupname 1x_depth --subgroup 1 -n 20210205 20210130 -e 74781 73702
desi_zproc --batch -q debug -t 80605 \
    --groupname 1x_depth --subgroup 2 -n 20201215 20210205 -e 67975 74782

# Example to confirm that previous default cumulative grouping isn't broken
desi_zproc --batch --nosubmit -t 1000 -n 20210517

The first and last case with --nosubmit generated the slurm file, which I then sourced in an interactive node. The middle case without --nosubmit ran in the debug queue as expected.

This PR will make it much easier to run 1x_depth followup to Jura to evaluate new QuasarNET models for Kibo.

@sbailey sbailey requested a review from akremin July 25, 2024 18:41
Copy link
Member

@akremin akremin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall this looks good. I found what appears to be two typos in the unit tests. I also request a name change to the proposed new jobdesc. I'd prefer something like CUSTOMZ or CUSTOMTILE (with no underscore).

py/desispec/test/test_io.py Outdated Show resolved Hide resolved
py/desispec/test/test_io.py Outdated Show resolved Hide resolved
py/desispec/workflow/desi_proc_funcs.py Outdated Show resolved Hide resolved
py/desispec/workflow/redshifts.py Outdated Show resolved Hide resolved
@akremin akremin merged commit 8baff32 into main Aug 5, 2024
26 checks passed
@akremin akremin deleted the 1x_depth branch August 5, 2024 23:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants