Package infrastructure update and other fixes #199

weaverba137 · 2024-09-18T21:35:50Z

This PR

Closes Search function is not working #198
Closes installed desidatamodel drops data/column_descriptions.csv #197
Updates the Python package infrastructure to recent standards (i.e. minimalist setup.py, etc.).
Makes some cosmetic changes to comply with stricter pycodestyle configuration.

Remaining work:

Definitions are needed for these columns:
- NUMEXP
- NUMTILE
- IN_COADD_(B|R|Z).

coveralls · 2024-09-18T21:38:51Z

coverage: 100.0%. remained the same
when pulling e81dcb3 on package-updates
into 44e0708 on main.

weaverba137 · 2024-09-19T00:20:32Z

I've fixed some header descriptions for SKYLEVEL, which should also close #191.

weaverba137 · 2024-10-07T17:02:42Z

@sbailey, pinging you for review. Note that there are some unresolved questions about new columns, see above.

weaverba137 · 2024-10-07T23:19:02Z

@sbailey: I think I found where IN_COADD_[BRZ] come from: desihub/desispec#2100.

So the definition would appear to be "If True, this exposure was used in the coadd of camera B,R,Z."

When I was generating the zcatalog files for daily, these columns appeared in the EXP_FIBERMAP HDU, which is fine.

I was able to find references to COADD_NUMEXP and COADD_NUMTILE, but not "bare" NUMEXP or NUMTILE. For reference these columns appeared in the ZCATALOG HDU of the daily zcatalog files, along with COADD_NUMEXP and COADD_NUMTILE. So that's potentially confusing on top of still needing a definition.

Is it possibly a bug that NUMEXP and NUMTILE are showing up?

weaverba137 · 2024-10-07T23:37:56Z

Possibly more precise: "If True, this fiber in this exposure was used in the coadd of camera B,R,Z."

sbailey · 2024-10-08T16:06:58Z

Possibly more precise: "If True, this fiber in this exposure was used in the coadd of camera B,R,Z."

Looks good for long form. For short form for FITS headers, maybe something like "fiber/exp used in coadd of camera B,R,Z"

NUMEXP vs. COADD_NUMEXP and NUMTILE vs. COADD_NUMTILE: I only see the "COADD_*" versions of these in Kibo. If you ended up with all 4 columns when making daily zcatalogs, that's likely due to a leftover format when propagating upstream files before the FIBERMAP / EXP_FIBERMAP split. Please point me to an example output file that has all for columns and the command that generated it. I'm guessing that non-zero entries of NUMEXP and NUMTILE could be traced back to old data, and we'll need to add another special case to some reader to intercept and rename that.

weaverba137 · 2024-10-08T16:26:19Z

You should look in $DESI_SPECTRO_REDUX/dailydb/zcatalog/v1. For each .fits file, there is a corresponding .txt file which is the output of fitsverify -l, so you can compare them without actually opening the files.

I hadn't done this previously, but there is some detective work to be done, because I'm seeing different survey/program having different columns.

sbailey · 2024-10-08T16:47:05Z

It appears that all flavors of ztile-*-dark-cumulative.fits have a NUMEXP column, but they also appear to be all zeros. I don't see from the code where this is appearing. Please trace upstream to the input files to see where this column first appears. Thanks.

weaverba137 · 2024-10-08T17:08:01Z

Sure, will do.

weaverba137 · 2024-10-09T17:52:25Z

I'm still working on tracing the origin of NUMEXP, but I have some interesting updates:

It appears that all flavors of ztile-*-dark-cumulative.fits have a NUMEXP column, but they also appear to be all zeros.

It turns out this isn't actually true. NUMEXP does not appear in main-backup or special-backup. I don't know why but it may be a clue.

I did a thorough analysis of all of the columns in the dailydb ztile files, and the results are interesting.

For the ZCATALOG HDU:

Column 'NUMEXP' in HDU 'ZCATALOG' is missing for these files: ['main-backup', 'special-backup'].
Column 'NUMTILE' in HDU 'ZCATALOG' is missing for these files: ['main-backup', 'special-backup'].
Column 'PLATE_RA' in HDU 'ZCATALOG' is missing for these files: ['cmx-other', 'sv1-backup', 'sv1-bright', 'sv1-dark', 'sv2-backup', 'sv2-bright', 'sv2-dark', 'sv3-backup'].
Column 'PLATE_DEC' in HDU 'ZCATALOG' is missing for these files: ['cmx-other', 'sv1-backup', 'sv1-bright', 'sv1-dark', 'sv2-backup', 'sv2-bright', 'sv2-dark', 'sv3-backup'].
Column 'MIN_MJD' in HDU 'ZCATALOG' is missing for these files: ['special-backup'].
Column 'MAX_MJD' in HDU 'ZCATALOG' is missing for these files: ['special-backup'].
Column 'MEAN_MJD' in HDU 'ZCATALOG' is missing for these files: ['special-backup'].
Column 'DESINAME' in HDU 'ZCATALOG' is missing for these files: ['cmx-other', 'special-backup', 'sv1-backup', 'sv1-bright', 'sv1-dark', 'sv1-other', 'sv2-backup', 'sv2-bright', 'sv2-dark', 'sv3-backup', 'sv3-bright', 'sv3-dark'].
Column 'FITMETHOD' in HDU 'ZCATALOG' is missing for these files: ['cmx-other', 'special-backup', 'special-bright', 'sv1-backup', 'sv1-bright', 'sv1-dark', 'sv1-other', 'sv2-backup', 'sv2-bright', 'sv2-dark', 'sv3-backup', 'sv3-bright', 'sv3-dark'].
Column 'NUMTARGET' in HDU 'ZCATALOG' is missing for these files: ['main-backup', 'main-bright', 'main-dark', 'special-backup', 'special-bright', 'special-dark', 'sv1-backup', 'sv1-bright', 'sv1-other', 'sv2-backup', 'sv2-bright', 'sv2-dark', 'sv3-backup', 'sv3-bright', 'sv3-dark'].
Column 'BLOBDIST' in HDU 'ZCATALOG' is missing for these files: ['main-backup', 'main-bright', 'main-dark', 'special-backup', 'special-bright', 'special-dark', 'sv1-backup', 'sv1-bright', 'sv1-other', 'sv2-backup', 'sv2-bright', 'sv2-dark', 'sv3-backup', 'sv3-bright', 'sv3-dark'].
Column 'FIBERFLUX_IVAR_G' in HDU 'ZCATALOG' is missing for these files: ['main-backup', 'main-bright', 'main-dark', 'special-backup', 'special-bright', 'special-dark', 'sv1-backup', 'sv1-bright', 'sv1-other', 'sv2-backup', 'sv2-bright', 'sv2-dark', 'sv3-backup', 'sv3-bright', 'sv3-dark'].
Column 'FIBERFLUX_IVAR_R' in HDU 'ZCATALOG' is missing for these files: ['main-backup', 'main-bright', 'main-dark', 'special-backup', 'special-bright', 'special-dark', 'sv1-backup', 'sv1-bright', 'sv1-other', 'sv2-backup', 'sv2-bright', 'sv2-dark', 'sv3-backup', 'sv3-bright', 'sv3-dark'].
Column 'FIBERFLUX_IVAR_Z' in HDU 'ZCATALOG' is missing for these files: ['main-backup', 'main-bright', 'main-dark', 'special-backup', 'special-bright', 'special-dark', 'sv1-backup', 'sv1-bright', 'sv1-other', 'sv2-backup', 'sv2-bright', 'sv2-dark', 'sv3-backup', 'sv3-bright', 'sv3-dark'].
Column 'HPXPIXEL' in HDU 'ZCATALOG' is missing for these files: ['main-backup', 'main-bright', 'main-dark', 'special-backup', 'special-bright', 'special-dark', 'sv1-backup', 'sv1-bright', 'sv1-other', 'sv2-backup', 'sv2-bright', 'sv2-dark', 'sv3-backup', 'sv3-bright', 'sv3-dark'].

Some of the targeting columns aren't present in all of the files, but I verified that they are missing where they are expected to be missing:

Column 'CMX_TARGET' in HDU 'ZCATALOG' is missing from some files, but that is expected.
Column 'SV1_DESI_TARGET' in HDU 'ZCATALOG' is missing from some files, but that is expected.
Column 'SV1_BGS_TARGET' in HDU 'ZCATALOG' is missing from some files, but that is expected.
Column 'SV1_MWS_TARGET' in HDU 'ZCATALOG' is missing from some files, but that is expected.
Column 'SV1_SCND_TARGET' in HDU 'ZCATALOG' is missing from some files, but that is expected.
Column 'SV2_DESI_TARGET' in HDU 'ZCATALOG' is missing from some files, but that is expected.
Column 'SV2_BGS_TARGET' in HDU 'ZCATALOG' is missing from some files, but that is expected.
Column 'SV2_MWS_TARGET' in HDU 'ZCATALOG' is missing from some files, but that is expected.
Column 'SV2_SCND_TARGET' in HDU 'ZCATALOG' is missing from some files, but that is expected.
Column 'SV3_DESI_TARGET' in HDU 'ZCATALOG' is missing from some files, but that is expected.
Column 'SV3_BGS_TARGET' in HDU 'ZCATALOG' is missing from some files, but that is expected.
Column 'SV3_MWS_TARGET' in HDU 'ZCATALOG' is missing from some files, but that is expected.
Column 'SV3_SCND_TARGET' in HDU 'ZCATALOG' is missing from some files, but that is expected.
Column 'SCND_TARGET' in HDU 'ZCATALOG' is missing from some files, but that is expected.

And there is a type mismatch for RELEASE, although it is present in all files:

Column 'RELEASE' in HDU 'ZCATALOG' with type 'I' has a different type in these files: ['cmx-other', 'sv1-dark'].
Column 'RELEASE' in HDU 'ZCATALOG' with type 'J' has a different type in these files: ['main-backup', 'main-bright', 'main-dark', 'special-backup', 'special-bright', 'special-dark', 'sv1-backup', 'sv1-bright', 'sv1-other', 'sv2-backup', 'sv2-bright', 'sv2-dark', 'sv3-backup', 'sv3-bright', 'sv3-dark'].

For the EXP_FIBERMAP HDU:

Column 'PLATE_RA' in HDU 'EXP_FIBERMAP' is missing for these files: ['cmx-other', 'sv1-backup', 'sv1-bright', 'sv1-dark', 'sv2-backup', 'sv2-bright', 'sv2-dark', 'sv3-backup'].
Column 'PLATE_DEC' in HDU 'EXP_FIBERMAP' is missing for these files: ['cmx-other', 'sv1-backup', 'sv1-bright', 'sv1-dark', 'sv2-backup', 'sv2-bright', 'sv2-dark', 'sv3-backup'].
Column 'PSF_TO_FIBER_SPECFLUX' in HDU 'EXP_FIBERMAP' is missing for these files: ['cmx-other'].
Column 'IN_COADD_B' in HDU 'EXP_FIBERMAP' is missing for these files: ['special-backup'].
Column 'IN_COADD_R' in HDU 'EXP_FIBERMAP' is missing for these files: ['special-backup'].
Column 'IN_COADD_Z' in HDU 'EXP_FIBERMAP' is missing for these files: ['special-backup'].

weaverba137 · 2024-10-10T22:14:32Z

So far, I've only found NUMEXP, NUMTILE in the ZBEST HDU of zbest files, but I'm still searching.

sbailey · 2024-10-11T17:49:30Z

It looks like zbest*.fits files have NUMEXP, and redrock*.fits files have COADD_NUMEXP. In that case I think the solution would be to intercept and rename NUMEXP as part of read_redrock. Ditto for NUMTILE.

weaverba137 · 2024-10-11T17:52:38Z

I will certainly do that, however, I am not convinced that NUMEXP appears exclusively in zbest files and COADD_NUMEXP appears exclusively in redrock files. The prevalence of NUMEXP even in main survey zcatalog files suggests otherwise, unless some tiles with zbest files are actually in the main survey.

sbailey · 2024-10-11T18:40:46Z

At minimum I think it would be save to apply something like:

if 'NUMEXP' in fibermap.colnames and 'COADD_NUMEXP' not in fibermap.colnames:
    fibermap.rename_column('NUMEXP', 'COADD_NUMEXP')
if 'NUMTILE' in fibermap.colnames and 'COADD_NUMTILE' not in fibermap.colnames:
    fibermap.rename_column('NUMTILE', 'COADD_NUMTILE')

Then rerun desi_zcatalog and see if problems remain...

weaverba137 · 2024-10-11T18:42:37Z

Sure, I'll try that.

weaverba137 · 2024-10-11T18:43:47Z

@sbailey, assuming we completely get rid of NUMEXP, NUMTILE, and I remove the placeholders for those columns in the CSV file, is there any reason not to merge this PR?

weaverba137 added 4 commits September 18, 2024 08:46

update Sphinx config

5c6bbf0

fix import error

094511c

update package infrastructure

b5b47a6

add some missing column definitions

ef5f886

weaverba137 added the WIP Work in Progress label Sep 18, 2024

weaverba137 requested a review from sbailey September 18, 2024 21:35

weaverba137 self-assigned this Sep 18, 2024

update change log

c419a90

weaverba137 mentioned this pull request Sep 19, 2024

Fiberflat header documentation #191

Closed

fix unit brackets for SKYLEVEL

06e046c

weaverba137 mentioned this pull request Sep 26, 2024

Support reading zbest redshift files desihub/desispec#2374

Merged

weaverba137 added 2 commits October 11, 2024 11:45

remove NUMEXP, NUMTILE

f77f6e6

update IN_COADD definitions

e81dcb3

sbailey merged commit dab9a68 into main Oct 11, 2024
34 checks passed

sbailey deleted the package-updates branch October 11, 2024 20:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Package infrastructure update and other fixes #199

Package infrastructure update and other fixes #199

weaverba137 commented Sep 18, 2024

coveralls commented Sep 18, 2024 •

edited

Loading

weaverba137 commented Sep 19, 2024

weaverba137 commented Oct 7, 2024

weaverba137 commented Oct 7, 2024 •

edited

Loading

weaverba137 commented Oct 7, 2024

sbailey commented Oct 8, 2024

weaverba137 commented Oct 8, 2024

sbailey commented Oct 8, 2024

weaverba137 commented Oct 8, 2024

weaverba137 commented Oct 9, 2024

weaverba137 commented Oct 10, 2024

sbailey commented Oct 11, 2024

weaverba137 commented Oct 11, 2024

sbailey commented Oct 11, 2024

weaverba137 commented Oct 11, 2024

weaverba137 commented Oct 11, 2024

Package infrastructure update and other fixes #199

Package infrastructure update and other fixes #199

Conversation

weaverba137 commented Sep 18, 2024

coveralls commented Sep 18, 2024 • edited Loading

weaverba137 commented Sep 19, 2024

weaverba137 commented Oct 7, 2024

weaverba137 commented Oct 7, 2024 • edited Loading

weaverba137 commented Oct 7, 2024

sbailey commented Oct 8, 2024

weaverba137 commented Oct 8, 2024

sbailey commented Oct 8, 2024

weaverba137 commented Oct 8, 2024

weaverba137 commented Oct 9, 2024

weaverba137 commented Oct 10, 2024

sbailey commented Oct 11, 2024

weaverba137 commented Oct 11, 2024

sbailey commented Oct 11, 2024

weaverba137 commented Oct 11, 2024

weaverba137 commented Oct 11, 2024

coveralls commented Sep 18, 2024 •

edited

Loading

weaverba137 commented Oct 7, 2024 •

edited

Loading