Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix indexing bug in zproc expqa check #2344

Merged
merged 1 commit into from
Aug 26, 2024
Merged

Fix indexing bug in zproc expqa check #2344

merged 1 commit into from
Aug 26, 2024

Conversation

akremin
Copy link
Member

@akremin akremin commented Aug 26, 2024

Overview

This solves the issue identified in Issue #2342 that was introduced in PR #2306 .

zproc allows many modes to process healpix based redshifts and tile-based redshifts in per-exposure, per-night, cumulative, and custom varieties. Because of this the input structure and handling of information is complex and sometimes dependent on the type of job being requested.

Originally the code looped over expids and night to get the list of expid-night combinations, but these are unique lists of the requested variables rather than index-matched lists. However, for pernight, cumulative, and custom tile-based jobs an exposure_table is populated from the set of nights and then subselected to the requested set of exposure ID's. So this table has the index-matched information we want to use.

This PR updates the code to use the exposure_table for identifying exposure_qa files that are expected to exist. The exposure_table variable should exist for pernight and cumulative jobs, which are the only types that perform the tileqa step that we're updating here.

Test

I ran a test of the failing job (tile=80616 thrunight=20201223) mentioned in the issue. I sym-linked the calibnight, exposure_tables, processing_tables, and exposures directories from kibo; copied the relevant script from kibo; and ran it using this branch. The job succeeded.

Location: /global/cfs/cdirs/desi/spectro/redux/test_kibo
zproc log:/global/cfs/cdirs/desi/spectro/redux/test_kibo/ztile-80616-thru20201223-29816844.log
tileqa log: /global/cfs/cdirs/desi/spectro/redux/test_kibo/tiles/cumulative/80616/20201223/logs/tile-qa-80616-thru20201223.log

@akremin akremin requested a review from sbailey August 26, 2024 19:59
Copy link
Contributor

@sbailey sbailey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! That was a much cleaner fix than I was expecting. Great that you found a way to update this without having to re-define what the nights variable means and thus potentially break some other piece of the code.

@sbailey sbailey merged commit 1ec383e into main Aug 26, 2024
26 checks passed
@sbailey sbailey deleted the expqa_fname_bugfix branch August 26, 2024 21:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants