Fix indexing bug in zproc expqa check #2344
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
This solves the issue identified in Issue #2342 that was introduced in PR #2306 .
zproc allows many modes to process healpix based redshifts and tile-based redshifts in per-exposure, per-night, cumulative, and custom varieties. Because of this the input structure and handling of information is complex and sometimes dependent on the type of job being requested.
Originally the code looped over expids and night to get the list of expid-night combinations, but these are unique lists of the requested variables rather than index-matched lists. However, for pernight, cumulative, and custom tile-based jobs an exposure_table is populated from the set of nights and then subselected to the requested set of exposure ID's. So this table has the index-matched information we want to use.
This PR updates the code to use the exposure_table for identifying exposure_qa files that are expected to exist. The exposure_table variable should exist for pernight and cumulative jobs, which are the only types that perform the tileqa step that we're updating here.
Test
I ran a test of the failing job (tile=80616 thrunight=20201223) mentioned in the issue. I sym-linked the
calibnight
,exposure_tables
,processing_tables
, andexposures
directories from kibo; copied the relevant script from kibo; and ran it using this branch. The job succeeded.Location:
/global/cfs/cdirs/desi/spectro/redux/test_kibo
zproc log:
/global/cfs/cdirs/desi/spectro/redux/test_kibo/ztile-80616-thru20201223-29816844.log
tileqa log:
/global/cfs/cdirs/desi/spectro/redux/test_kibo/tiles/cumulative/80616/20201223/logs/tile-qa-80616-thru20201223.log