Night_qa: introduce new ctedet row-by-row diagnosis plot

[araichoor]

This PR introduces a new plot to diagnose possible cte occurences.

The plot was suggested by @schlafly here: https://desisurvey.slack.com/archives/C01HNN87Y7J/p1722543734755599?thread_ts=1722538227.361809&cid=C01HNN87Y7J
Copying from there: "load the 1 s flat, do a row by row extraction, plot the residuals."

In addition to @schlafly initial "recipe", I made these changes:

• subtract the median value of the residuals (res_median, reported on the plot);
• restrict the color range to (-2, 2);
• add per-amplifier infos:
  • the amplifier name, with its related region on the CCD (using 0-index convention);
  • any existing correction (OFFCOLS and/or CTECOLS): column range is reported, along with a line to visualize it.

The plots come as a new pdf file (ctedetrowbyrow-NIGHT.pdf), with one page per petal, three plots (for each camera) per page.
Here is one example for a petal for 20230822:

And here is another example for 20240730, illustrating the "newly" cte on r1:

Here are the nightqa-NIGHT.html pages for those two examples:
https://data.desi.lbl.gov/desi/users/raichoor/nightqa_dev/nightqa_v27/aug07/nightqa/20230822/nightqa-20230822.html
https://data.desi.lbl.gov/desi/users/raichoor/nightqa_dev/nightqa_v27/aug07/nightqa/20240730/nightqa-20240730.html

I think that the code now reads twice the 1s flat preproc images (and I also had to switch from a simple reading with fitsio to a more complete/time-consuming reading with desispec.io.read_image()), that may take an extra minute? but I guess it s ok.
The new plot also calls correct_cte.get_rowbyrow_image_model(), which may take 1 extra minute.

Any suggestion is welcome!

[araichoor]

I notice that the "documentation" check didn t pass...

[schlafly]

sphinx is being strict about the indentation under mydicts; I think if you align those lines it will be quiet.

[araichoor]

thanks! that did fix it.

[akremin]

Thanks Anand. I'd like @schlafly to comment on your choices of restricting the color range and removing the median. The code looks fine to me (I admittedly didn't attempt to understand every line).
How much testing has been done for this? Did you just run it on one recent night or on a few random nights to make sure it's robust? And will this gracefully skip early nights that don't have a 1s flat, or will it crash with an error?

[araichoor]

@akremin : thanks for the feedback.
I just tested on two nights...

about missing 1s flat: the approach is the same as for the previous ctedet diagnosis, ie desi_night_qa won t try to generate the pdf if no image is found:

desispec/bin/desi_night_qa

Lines 170 to 173 in 094c3be

	if "ctedet" in steps_tbd:
	if ctedet_expid is not None:
	create_ctedet_pdf(outfns["ctedet"], args.night, args.prod, ctedet_expid, args.nproc)
	create_ctedet_rowbyrow_pdf(outfns["ctedetrowbyrow"], args.night, args.prod, ctedet_expid, args.nproc)

one other thing I could think of: if some camera / petal are missing; I ve not tested that; here, it should have the same behavior as for the dark diagnosis (I adapted that piece of code), i.e. just create an empty figure, so I think it should be robust.
it s here in the code:

desispec/py/desispec/night_qa.py

Line 1115 in 094c3be

if mydict is not None:

if you can provide me an example of a 1s flat exposure with missing camera / petal, I can surely run a test (I ll try to look for such an exposure, but you re likely be faster than me, here).

about the choices of restricting the color range and removing the median: sorry, I should have mentioned, I ve already iterated with @schlafly on that.

[akremin]

20240723 had petal 7 missing from its configuration. Would that test your concern? It doesn't have a BADCAMWORD set in the pipeline exposure_table because the camera wasn't in the instance. If you need a night where we flagged a camera as bad I can dig further. Those are quite rare for calibrations.

[araichoor]

thanks that should be perfect for testing (I see /global/cfs/cdirs/desi/spectro/redux/daily/preproc/20240723/00245086/ has no petal=7 data).

[schlafly]

Anand and I chatted separately re color scales; this looks good to me.

[araichoor]

as usual, @akremin, thanks for raising those points.
I actually had a small bug in handling that case; and I also fixed minor things in desi_night_qa.

now the code runs well if one petal is missing:

full night qa files for 20240723 here:
https://data.desi.lbl.gov/desi/users/raichoor/nightqa_dev/nightqa_v27/aug07/nightqa/20240723/ctedetrowbyrow-20240723.pdf

[akremin]

Thank you, Anand. The plots in that last post look great. Are you happy with this PR now or are you still planning to make changes?

[araichoor]

as far as I m concerned, I m happy with the current PR version.
don t know if @schlafly would have any further suggestions?

[schlafly]

No, go ahead, thanks!

…

On Wed, Aug 7, 2024, 18:49 araichoor ***@***.***> wrote: as far as I m concerned, I m happy with the current PR version. don t know if @schlafly <https://github.com/schlafly> would have any further suggestions? — Reply to this email directly, view it on GitHub <#2312 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAK2E4MSHZVJ7BPCAH4GUDTZQKP53AVCNFSM6AAAAABME77ATSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENZUGQ3TONZXGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>