Merge pull request #1665 from pypeit/staged

Merges develop into release (1.14.0 tag prep)

.github/workflows/ci_cron.yml

@@ -16,7 +16,7 @@ jobs:
       matrix:
         os: [ubuntu-latest]
         python: ['3.9', '3.10', '3.11']
-        toxenv: [test-alldeps, test-numpydev, test-linetoolsdev, test-gingadev, test-astropydev, conda]
+        toxenv: [test-alldeps, test-numpydev, test-linetoolsdev, test-gingadev, test-astropydev]
     steps:
     - name: Check out repository
       uses: actions/checkout@v3
@@ -31,4 +31,4 @@ jobs:
         python -m pip install --upgrade pip tox
     - name: Test with tox
       run: |
-        tox -e ${{ matrix.toxenv }}
+        tox -e ${{ matrix.python }}-${{ matrix.toxenv }}

.github/workflows/ci_tests.yml

@@ -18,7 +18,7 @@ jobs:
       matrix:
         os: [ubuntu-latest]
         python: ['3.9', '3.10', '3.11']
-        toxenv: [test, test-alldeps-cov, test-linetoolsdev, test-gingadev, test-astropydev, conda]
+        toxenv: [test, test-alldeps-cov, test-linetoolsdev, test-gingadev, test-astropydev]
     steps:
     - name: Check out repository
       uses: actions/checkout@v3
@@ -31,7 +31,7 @@ jobs:
         python -m pip install --upgrade pip tox
     - name: Test with tox
       run: |
-        tox -e ${{ matrix.toxenv }}
+        tox -e ${{ matrix.python }}-${{ matrix.toxenv }}
     - name: Upload coverage to codecov
       if: "contains(matrix.toxenv, '-cov')"
       uses: codecov/codecov-action@v3
@@ -62,7 +62,22 @@ jobs:
         python -m pip install --upgrade pip tox
     - name: Test with tox
       run: |
-        tox -e ${{ matrix.toxenv }}
+        tox -e ${{ matrix.python }}-${{ matrix.toxenv }}
+
+  conda:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v3
+    - name: Conda environment check
+      uses: actions/setup-python@v4
+      with:
+        python-version: '3.11'
+    - name: Install base dependencies
+      run: |
+        python -m pip install --upgrade pip tox
+    - name: Run pypeit tests from environment built with conda
+      run: |
+        tox -e conda
 
   codestyle:
     runs-on: ubuntu-latest

CHANGES.rst

@@ -1,3 +1,58 @@
+1.14.0 (18 Sep 2023)
+--------------------
+
+- Add support for Gemini/GNIRS (IFU)
+- Add support for Keck/KCRM
+- Added a script to convert a wavelength solution into something that can be placed in the reid archive.
+- Hotfix for GTC/OSIRIS lamp list
+- Hotfix for Arc1D stats annotations on the QA
+- Hotfix for metadata (correctly set config_independent frames when multiple configurations are being setup)
+- Hotfix for metadata (support lists in ``config_independent_frames()``)
+- Hotfix for rebin (speed-up and conserves flux)
+- Hotfix for skysub regions GUI that used np.bool
+- Hotfix to stop pypeit_setup from crashing on data from lbt_luci1, lbt_luci2, magellan_fire,
+  magellan_fire_long, p200_tspec, or vlt_sinfoni.
+- Hotfix to set BPM for each type of calibration file.
+- Adds Keck/ESI to PypeIt
+- Instrumental FWHM map is calculated and output in ``Calibrations`` and ``spec1d`` files.
+- Adds Keck/ESI to PypeIt
+- Add MDM/Modspec spectrograph
+- Store user-generated wavelength solution in pypeit cache
+- Improvements to wavelength grids and masking in coadd routines.
+- Fixed a bug in echelle coadding where the wrong coadded spectra were being
+  used in final stacks.
+- Sensitivity function models can now be computed relative to the blaze
+  spectrum.
+- Refactored coadding routines to work with lists to support coadding data from
+  different setups.
+- Changes to how masking is dealt with in extraction to fix a bug in how masks
+  were being treated for echelle data
+- Various fixes and changes required to add more support for Keck/HIRES and JWST
+- Fix a bug in ``spectrograph.select_detectors``, where a list of ``slitspatnum`` could not be used.
+- Improvements in 2D coaddition
+    - Fix a bug in `pypeit_setup_coadd2d` for the output file name of the .coadd2d file
+    - Added possibility to specify more than one Science folder in `pypeit_setup_coadd2d`
+    - Now ``only_slits`` parameter in `pypeit_coadd_2dspec` includes the detector number (similar to ``slitspatnum``)
+    - Added ``exclude_slits`` parameter in `pypeit_coadd_2dspec` to exclude specific slits
+    - Fix wrong RA & Dec for 2D coadded serendips
+- Changed calibration frame naming as an attempt to avoid very long names for
+  files with many calibration groups.  Sequential numbers are reduced to a
+  range; e.g., ``'0-1-2-3-4'`` becomes ``'0+4'`` and
+  ``'3-5-6-10-11-12-15-18-19'`` becomes ``'3-5+6-10+12-15-18+19'``
+- HIRES wavelength solution improvements galor
+- Added `redo_slits` option
+- Refactored ``load_line_lists()`` yet again!
+- Improvements for keck/LRIS
+    - Generated wavelength templates for all the LRIS grism & grating
+    - Added FeAr line list
+    - Improved calibration association and frame typing
+    - Improved and added documentation
+    - Changes to ``metadata.py`` including commenting out, in the pypeit file,
+      files that have frametype None (this prevent ``run_pypeit`` to crash)
+    - Added a function ``check_spectrograph()`` (currently only defined for LRIS),
+      that checks (during ``pypeit_setup``) if the selected spectrograph is the
+      corrected one for the data used. 
+
 
 1.13.0 (2 June 2023)
 --------------------
@@ -7,7 +62,6 @@
 - Allow user control of the local sky subtraction window
 - Deprecate use of python 3.8 with PypeIt, allow python 3.11
 - Make pypeit_show_2dspec (somewhat) backwards compatible.
-- Added the option to disable strict version checking for 1d coadds.
 - Hotfix for KCWI when using alignment (aka ContBars) frames for the astrometric correction.
 - Sensitivity function masking and output updates
 - Fixed a bug in the `variance_model` calculation for combined images.
@@ -27,7 +81,8 @@
 - The ``'calib'`` column is now always added to the pypeit file, regardless of
   whether or not you also request the ``'comb_id'`` and ``'bkg_id'`` columns.
 - Names of associated calibration frames now written to ``spec2d`` file headers.
-- Major quicklook updates.  ql_multislit.py deprecated.
+- Added the option to disable strict version checking for 1d coadds.
+- Major quicklook updates.  ql_multislit.py temporarily deprecated.
 - Improve speed in ginga visualization of traces and added
   `pypeit_chk_tilts`. Note that this script uses an update
   of the tilts datamodel, so it will not work on older reductions.
@@ -1523,7 +1578,7 @@ better with 2d coadds.
 - Other odds and ends including code flow doc
 - Introduce pypit/par and pypit/config directories
 - Introduce PypitPar as an initial step toward refactoring the front end
-- Move spectrograph specific code into spectographs/ folder
+- Move spectrograph specific code into spectrographs/ folder
 - Introduces the Spectrographs class
 - Introduces the Calibrations class with Notebook
 - Bug fix in view_fits script

LICENSE.rst

@@ -1,4 +1,4 @@
-Copyright (c) 2018-2019, PypeIt Developers
+Copyright (c) 2018-2023, PypeIt Developers
 All rights reserved.
 
 Redistribution and use in source and binary forms, with or without modification,

README.rst

@@ -97,7 +97,7 @@ follow our `Code of Conduct
 
 Along with our extensive `online documentation
 <https://pypeit.readthedocs.io/en/release/>`__, we encourage the PypeIt user
-base to communicate via our `PypeIt Users Slack <pypeit-users.slack.com>`__.
+base to communicate via our `PypeIt Users Slack <https://pypeit-users.slack.com>`__.
 All are welcome to join using `this invitation link <https://join.slack.com/t/pypeit-users/shared_invite/zt-1kc4rxhsj-vKU1JnUA~8PZE~tPlu~aTg>`__.
 
 If you find a bug (particularly one that is experienced by others in the Users
@@ -170,4 +170,6 @@ development of PypeIt.
 * Milan Roberson
 * Timothy Pickering
 * Timothy Ellsworth-Bowers
+* Gregory Simonian
+* Heather Martin
 

doc/Makefile

@@ -84,7 +84,7 @@ htmlonly:
 
 picky:
 	make apirst
-	$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) -n $(BUILDDIR)/html
+	$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) -n -vvv -T $(BUILDDIR)/html -W
 	@echo
 	@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."
 

doc/api/pypeit.scripts.arxiv_solution.rst

@@ -0,0 +1,8 @@
+pypeit.scripts.arxiv\_solution module
+=====================================
+
+.. automodule:: pypeit.scripts.arxiv_solution
+   :members:
+   :private-members:
+   :undoc-members:
+   :show-inheritance:

doc/api/pypeit.scripts.rst

@@ -7,6 +7,7 @@ Submodules
 .. toctree::
    :maxdepth: 4
 
+   pypeit.scripts.arxiv_solution
    pypeit.scripts.cache_github_data
    pypeit.scripts.chk_alignments
    pypeit.scripts.chk_edges
@@ -34,7 +35,6 @@ Submodules
    pypeit.scripts.parse_slits
    pypeit.scripts.qa_html
    pypeit.scripts.ql
-   pypeit.scripts.ql_multislit
    pypeit.scripts.run_pypeit
    pypeit.scripts.scriptbase
    pypeit.scripts.sensfunc

doc/api/pypeit.spectrographs.keck_esi.rst

@@ -0,0 +1,8 @@
+pypeit.spectrographs.keck\_esi module
+=====================================
+
+.. automodule:: pypeit.spectrographs.keck_esi
+   :members:
+   :private-members:
+   :undoc-members:
+   :show-inheritance:

doc/api/pypeit.spectrographs.mdm_modspec.rst

@@ -0,0 +1,8 @@
+pypeit.spectrographs.mdm\_modspec module
+========================================
+
+.. automodule:: pypeit.spectrographs.mdm_modspec
+   :members:
+   :private-members:
+   :undoc-members:
+   :show-inheritance:

doc/api/pypeit.spectrographs.rst

@@ -15,6 +15,7 @@ Submodules
    pypeit.spectrographs.jwst_nircam
    pypeit.spectrographs.jwst_nirspec
    pypeit.spectrographs.keck_deimos
+   pypeit.spectrographs.keck_esi
    pypeit.spectrographs.keck_hires
    pypeit.spectrographs.keck_kcwi
    pypeit.spectrographs.keck_lris
@@ -26,6 +27,7 @@ Submodules
    pypeit.spectrographs.ldt_deveny
    pypeit.spectrographs.magellan_fire
    pypeit.spectrographs.magellan_mage
+   pypeit.spectrographs.mdm_modspec
    pypeit.spectrographs.mdm_osmos
    pypeit.spectrographs.mmt_binospec
    pypeit.spectrographs.mmt_bluechannel

doc/calibrations/calibrations.rst

@@ -1,4 +1,6 @@
 
+.. include:: ../include/links.rst
+
 .. _calibrations:
 
 ============
@@ -175,15 +177,22 @@ All reduced calibration frames are named according to their primary calibration
 type (e.g., ``Arc``).  They are also assigned a unique identifier that is a
 combination of:
 
-    - the instrument configuration (setup) identifier (e.g., ``A``),
+    #. the instrument configuration (setup) identifier (e.g., ``A``),
 
-    - the list of associated calibration groups (e.g., ``1-2`` or ``all``), and
+    #. a compressed list of associated calibration groups (e.g., ``1+2`` or ``all``), and
 
-    - the detector or mosaic identifier (e.g., ``DET01`` or ``MSC01``).
+    #. the detector or mosaic identifier (e.g., ``DET01`` or ``MSC01``).
 
-.. note::
+For the second component, sequential numbers are reduced to a range; e.g.,
+``'0-1-2-3-4'`` becomes ``'0+4'`` and ``'3-5-6-10-11-12-15-18-19'`` becomes
+``'3-5+6-10+12-15-18+19'``.
+
+.. warning::
 
     If you have a lot of calibration groups in your pypeit file, you may end up
-    with very long file names!
+    with very long file names!  This may cause a fault when the file name is
+    included in the header of the output fits files.  If using the calibration
+    group ``all`` doesn't solve the problem or isn't possible given your
+    application, please `Submit an issue`_.
 
 

doc/calibrations/image_proc.rst

@@ -198,7 +198,7 @@ in our documentation as ``(nspec,nspat)``.  The operations required to
 flip/transpose the image arrays to match the PypeIt convention are dictated by
 instrument-specific :class:`~pypeit.images.detector_container.DetectorContainer`
 parameters and performed by
-:func:`~pypeit.spectrograph.spectrographs.Spectrograph.orient_image`.  Image
+:func:`~pypeit.spectrographs.spectrograph.Spectrograph.orient_image`.  Image
 orientation will be performed if the ``orient`` parameter is true.
 
 .. warning::

doc/calibrations/slit_tracing.rst

@@ -19,12 +19,40 @@ task owing to the wide variety in:
 Developing a single algorithm to handle all of these edge cases (pun
 intended) is challenging if not impossible. Therefore, there are a number of
 user-input parameters that one may need to consider when running PypeIt (see
-below).
+below :ref:`slit_tracing_issues` and :ref:`slit_tracing_customizing`).
 
 Underlying the effort is the :class:`~pypeit.edgetrace.EdgeTraceSet` class; see
 :func:`~pypeit.edgetrace.EdgeTraceSet.auto_trace` for a description of the
 algorithm.
 
+Slit-mask design matching
+-------------------------
+
+PypeIt can incorporate information about the slit-mask design into the
+slit-tracing process. This is primarily performed by
+:func:`pypeit.edgetrace.EdgeTraceSet.maskdesign_matching`, which matches
+the slit edges traced by PypeIt to the slit edges predicted
+using the slit-mask design information stored in the observations.
+Moreover, :func:`~pypeit.edgetrace.EdgeTraceSet.maskdesign_matching`
+uses the predicted slit edges positions to add slit traces that have
+not been detected in the image.
+
+
+This functionality at the moment is implemented only for these
+:ref:`slitmask_info_instruments` and is switched on by setting
+``use_maskdesign`` flag in :ref:`edgetracepar` to True. Other parameters
+may need to be adjusted as well, depending on the instrument (see
+:ref:`slitmask_ids_report` and the relevant instrument documentation pages).
+
+.. _slitmask_info_instruments:
+
+Slit-mask design Spectrographs
+++++++++++++++++++++++++++++++
+- :doc:`../spectrographs/deimos`
+- :doc:`../spectrographs/mosfire`
+- :doc:`../spectrographs/lris` (limited)
+- :doc:`../spectrographs/gemini_gmos` (limited)
+
 Viewing
 =======
 
@@ -61,6 +89,8 @@ additional output that can be used to diagnose the parameterized fits to the
 edge traces and the PCA decomposition.  Fair warning that, for images with many
 slits, these plots can be laborious to wade through...
 
+.. _slit_tracing_issues:
+
 Known Slit Tracing Issues
 =========================
 
@@ -189,6 +219,8 @@ For example:
 This will remove any slit on detector 2 that contains ``x_spat=2121``
 at ``y_spec=2000`` and similarly for the slit on ``det=3``.
 
+.. _slit_tracing_customizing:
+
 Slit Tracing Customizing
 ========================
 

doc/calibrations/slits.rst

@@ -45,10 +45,10 @@ This will show a table that looks like this:
         322     699091   248.21048545837402 .. 278.352352142334   345.8145122528076 .. 376.4430561065674   251.7797389884793 .. 281.9216056724392   345.8145122528076 .. 376.4430561065674   297.0124988555908 .. 327.3977041244507         0     0    -inf     inf
         457     699080  350.76593017578125 .. 379.9788398742676   514.4993572235107 .. 544.0507583618164  355.0206685112119 .. 384.23357820969824    513.3126446738094 .. 542.864045812115     432.632643699646 .. 462.014799118042         0     0    -inf     inf
 
-In addition, if reducing :doc:`../spectrographs/deimos` or
-:doc:`../spectrographs/mosfire` data and slit-mask design matching is performed
-(see :ref:`deimos-mask-matching` for DEIMOS and :ref:`mosfire-edge-tracing` for
-MOSFIRE), a second `astropy.io.fits.BinTableHDU`_ is written to disk.
+In addition, if reducing data from these :ref:`slitmask_info_instruments`
+and slit-mask design matching is performed (see e.g., :ref:`deimos-mask-matching`
+for DEIMOS and :ref:`mosfire-edge-tracing` for MOSFIRE), a second
+`astropy.io.fits.BinTableHDU`_ is written to disk.
 
 .. code-block:: console
 

doc/calibrations/tilt.rst

@@ -58,7 +58,7 @@ Current TiltImage Data Model
 ============================
 
 Internally, the image is held in
-:class:`pypeit.tiltimage.TiltImage`
+:class:`pypeit.images.buildimage.tiltimage.TiltImage`
 which subclasses from :class:`pypeit.images.pypeitimage.PypeItImage` and
 :class:`pypeit.datamodel.DataContainer`.