Errors from NaN wavelengths when expanding NIRSpec wavelength range

[stscijgbot-jp]

Issue JP-3873 was created on JIRA by Christian Hayes:

There are a few errors that arise when trying to extend the NIRSpec wavelength range, brought up by the PR here:  #8964 particularly when processing NIRSpec MOS data.

The wavelength range that the pipeline process can be expanded by modified wavelengthrange reference file (see attached for example, which modifies the CLEAR_PRISM to cover the wavelength range from 0.5 - 5.7 microns) and supplied to assign_wcs.  The above PR reported an error in extract_2d that gets thrown when using the wider wavelength range.  Issues appear due to assign_wcs.nirspec.compute_bounding_box adding a buffer region to the bounding boxes that it computes.  With an expanded wavelength range, the buffer regions can expand to pixels for which the NIRSpec WCS transforms are not valid and provide NaN wavelengths.  

I've tried to put together a more targeted catch of the initially reported error here:  https://github.com/hayescr/jwst/tree/expand_waverange, but this only rejects slits that have all NaN wavelength arrays on a given detector.  The presence of NaNs in the wavelength arrays for slits that are barely on a given detector cause later issues when performing resampling, because resample_spec.compute_spectral_pixel_scale calculates the pixel scale at the center of the bounding box.  This corresponds to pixels outside of the NIRSpec wavelength coverage for slits sufficiently close to the edge of the detector, resulting in NaN pixel scales and downstream errors.

It's not clear if this should be addressed by modifying the bounding box (and its buffer), how the pixel scales are calculated, or if we should throw out slits with sufficiently little data on the detector (I'm not sure that it is useful to keep a slit if it only has there are ~10 or so pixels in the spectral direction).

[stscijgbot-jp]

Comment by Christian Hayes on JIRA:

A good example for testing is PID 2750 obs 2 (note this requires associated MSA metadata file as well), e.g.,

strun calwebb_spec2 jw02750002001_03101_00001_nrs1_rate.fits --steps.assign_wcs.override_wavelengthrange=jwst_nirspec_wavelengthrange_0006_mod.asdf ```