QuasarNet afterburner updates part I

[sbailey]

This PR requires desihub/redrock#320 and fixes:

• QuasarNET afterburner only uses HIZ template results in overlap region #2299 (now it runs both HIZ and LOZ templates in a single call to Redrock, thus treating the overlap region self-consistently)
• Quasarnet afterburner tries to write to input directory #2281 (now it writes temporary files in the output directory instead of the input directory)
• Quasarnet afterburner doesn't record QN version #2398 (now it includes header keywords to record code versions, the location of $RR_TEMPLATE_DIR, $QN_MODEL_FILE, and exactly which Redrock template files were used)

I have not addressed #2375 in this PR because that will change the data model of the QN output in more substantive ways so I want it isolated in its own PR.

Heads up @abrodze @dylanagreen @akremin

[sbailey]

Test output files are in /pscratch/sd/s/sjbailey/temp/qso_qn/pr2402, run with

cd $SCRATCH/temp/qso_qn
INDIR=$CFS/desi/spectro/redux/kibo/tiles/cumulative/100/20210505
for PETAL in $(seq 0 9); do
  desi_qso_qn_afterburner \
    --coadd $INDIR/coadd-$PETAL-100-thru20210505.fits \
    --redrock $INDIR/redrock-$PETAL-100-thru20210505.fits \
    --target_selection all --save_target all --delete_RR_output False \
    --output pr2402/qso_qn-$PETAL-thru20210505.fits
done

[coveralls]

coverage: 30.18% (-0.01%) from 30.191%
when pulling 362dbd0 on qsoqn
into fe88bbd on main.

[akremin]

I'll leave the more detailed review to Allyson, since she has more in-depth knowledge of this code and how it interplays with redrock and the templates. I have two comments, one can be ignored if too much work, and one log message that I think needs to be updated.

[abrodze]

I talked with Stephen in person about this, but I am writing it here for better tracking.
Right now, it looks like a decent percentage of the spectra being rerun through redrock are from sky fibers. I am unsure if there is a legitimate reason to keep the implementation as such, but this could easily be fixed by requiring fibermap['OBJTYPE'] == 'TGT' in the selection mask sel_index_with_QN . Though the only benefit to making such a change would be a (maybe negligible) boost in runtime, as these will still be filtered out in catalog construction.

[sbailey]

I agree with @abrodze's suggestion to not rerun Redrock on sky fibers, but would like to save that for a followup PR that makes other output changes e.g. adding missing columns. I believe the only output changes from this PR are:

• Redrock rerun files have "qn" in the name instead of "tmp" (cosmetic, but I found this cleaner in case we chose to keep them).
• Since Redrock is run with both LOZ and HIZ templates together, the redshift overlap region is handled self consistently (instead of the HIZ answer always being picked and HIZ intermediate files overwriting the LOZ ones).

[abrodze]

A few comments related to the bestchi2 section we have discussed in person. Fine to push this to future PR

[sbailey]

@abrodze raised the additional issue of whether the simultaneous run of LOZ and HIZ templates will result in more ZWARN=4 (SMALL_DELTA_CHI2) flags in the 1.4 < z < 1.6 overlap region (close redshifts, but still |dv|>1000). Previously these would not occur because LOZ and HIZ were run separately so they couldn't trigger warnings on each other. I will try to quantify this before merging.

[sbailey]

Rerun redshifts have a high ZWARN>0 rate, but they don't appear to be any higher in the 1.4 < z < 1.6 LOZ:HIZ overlap region:

This was based on spot checking the 10 main-dark healpix with the highest number of rerun QSOs in the overlap region.

I'm going to proceed with final cross checks on the outputs and then merge and deploy.

[abrodze]

Unless you explicitly removed them, I would guess the ~50% flags is from 50% of reruns that are sky fibers

[sbailey]

Good point. At the time of this PR I had not yet removed sky fibers from the reruns, so indeed many of the ZWARNs are for sky fibers. Additionally, low S/N spectra are more likely to have a Redrock / QuasarNet disagreement and also more likely to have a LOW_DELTA_CHI2 warning.