write_nightqa_html: use earliest science exposure timestamp for orange

[araichoor]

This PR changes the way night_qa decides to report calibration files in orange (in the calibnight section).
Before, it was based on comparing the calibration files timestamps to the considered night.

However, what really matters is if the calibration files timestamps are earlier than the processed science exposure files.
This PR should implement that.

For each {camera}{petal}, the code now:

• looks for all the processed files in {prod}/exposures/{night}/{expid}/*-{camera}{petal}-*fits for the list of considered expids, and retains the earliest one;
• then compares it with the calibration file timestamp;
• and it also dumps in the log those timestamps.

To do that I had to change the argument list of night_qa.write_nightqa_html(); but I guess it s fine, as this is the only place in desispec where it is called.
Hopefully, that shouldn t make the script much longer to run, as this os.path.getmtime(fn) seems to be pretty fast.

Here one example:
https://data.desi.lbl.gov/desi/users/raichoor/nightqa_dev/nightqa_v22/v0/nightqa-20230806.html
https://data.desi.lbl.gov/desi/users/raichoor/nightqa_dev/nightqa_v22/v0/nightqa-20230806.log
And here another example when a {camera}{petal} is missing (to verify that code doesn t crash):
https://data.desi.lbl.gov/desi/users/raichoor/nightqa_dev/nightqa_v22/v0/nightqa-20230504.html
https://data.desi.lbl.gov/desi/users/raichoor/nightqa_dev/nightqa_v22/v0/nightqa-20230504.log

Unfortunately, I don t have at hand examples of nights where we expect the code to return orange text, ie when the exposure files have been processed before the calibration files (because I expect that we ve fixed those).
If someone knows such an existing night, I ll be happy to make a run test on it.

[araichoor]

Ps: those examples have been run with e.g.:

PROD=/global/cfs/cdirs/desi/spectro/redux/daily
NIGHT=20230807
OUTDIR=/global/cfs/cdirs/desi/users/raichoor/nightqa_dev/nightqa_v22/v0
desi_night_qa -p $PROD -n $NIGHT -o $OUTDIR --steps html > $OUTDIR/nightqa-$NIGHT.log

[sbailey]

Looks good, thanks. I added a suggestion to limit the glob of files to reduce the metadata load on N>>1 mtime calls, even if normally it runs fast.

I also ran touch /global/cfs/cdirs/desi/spectro/redux/p1/calibnight/20220910/*.* so that SPECPROD=p1 test production will have calibrations with mtime greater than their exposures files for night 20220910. Please use that to test a nightqa turning orange.

[sbailey]

even though os.path.getmtime is fairly fast, metadata queries are historically a weak point of CFS, so I suggest that we limit this glob to reduce the number of files that will be inspected, e.g. to "frame-{}-*.format(campet)".

[araichoor]

awesome, thanks for the prompt feedback!
I ve updated the code to only query the frame files when checking timestamps.

Examples with the latest commit:

PROD=/global/cfs/cdirs/desi/spectro/redux/daily
NIGHT=20230807
OUTDIR=/global/cfs/cdirs/desi/users/raichoor/nightqa_dev/nightqa_v22/v1
desi_night_qa -p $PROD -n $NIGHT -o $OUTDIR --steps html > $OUTDIR/nightqa-$NIGHT.log

https://data.desi.lbl.gov/desi/users/raichoor/nightqa_dev/nightqa_v22/v1/nightqa-20230807.html
https://data.desi.lbl.gov/desi/users/raichoor/nightqa_dev/nightqa_v22/v1/nightqa-20230807.log

and for 20220910 for the p1 production:

PROD=/global/cfs/cdirs/desi/spectro/redux/p1
NIGHT=20220910
OUTDIR=/global/cfs/cdirs/desi/users/raichoor/nightqa_dev/nightqa_v22/v1
desi_night_qa -p $PROD -n $NIGHT -o $OUTDIR --steps html > $OUTDIR/nightqa-$NIGHT.log

https://data.desi.lbl.gov/desi/users/raichoor/nightqa_dev/nightqa_v22/v1/nightqa-20220910.html
https://data.desi.lbl.gov/desi/users/raichoor/nightqa_dev/nightqa_v22/v1/nightqa-20220910.log
=> looks good to me; one "feature": for b9, the calib files are there, but no frame science exposures are there, so it s displayed in green; I think it s ok...

[sbailey]

Looks good, thanks, including switching to using findfile for the frames instead of glob to further reduce I/O metadata load.