Merge pull request #1723 from pypeit/adap2019_collate_test

Adap2019 collate test

doc/dev/build_archived_sensfuncs.rst

@@ -3,26 +3,63 @@
 
 .. _build_archived_sensfuncs:
 
-How to build archived sensitivity functions
-===========================================
+
+Creating archival sensitivity functions
+=======================================
+
+Archival sensitivity functions for instruments can be created by combining 
+multiple sensivity functions from different exposures at different 
+grating tilts to get a wider wavelength coverage. The steps in this
+process are typically:
+
+#. Choose sensitivity functions for grating. The functions should have good wavelength coverage,
+   and low airmass.
+
+#. Start with the bluest sensitivity function and translate the next function up/down to roughly match.
+   One or both function may be truncated to fit.
+
+#. If the gap between sensitivity function isn't smooth (for example on a detector boundary) 
+   a polynomial fit can be used to join them.
+
+There are helper functions in $PYPEIT_DEV/sensfunc_archive/stitcutils.py intended to make this easier. See
+``$PYPEIT_DEV/sensfunc_archive/create_deimos_sensfuncs.py`` and ``$PYPEIT_DEV/sensfunc_archive/stitchdeimos.py`` for examples of how this was done
+for DEIMOS.
+
+The DEIMOS archived sensitivity functions
+-----------------------------------------
+
+*Disclaimer*:
+The DEIMOS archival sensitivity functions do not provide an absolute flux 
+calibration.  Instead, they are only intended to remove the instrumental 
+response, providing a relative flux calibration up to some unknown 
+normalization.
+
+
+Creating DEIMOS stitched sensitivity functions for every grating
+****************************************************************
 
 1. Make sure you have up to date versions of the `PypeIt
    <https://github.com/pypeit/PypeIt>`_ and `PypeIt Development Suite`_ 
    repositories from GitHub, and that PypeIt :ref:`installing` is complete.
 
 2. Download the reduced slitless source images used to create Keck DEIMOS sensitivity functions from
-   the `PypeIt dev-suite Google Drive`_. The files are located under ``DEIMOS_Dev/Throughput/throughput_gdw/data_products/extract``.
-
-   .. code-block:: bash
+   the `PypeIt dev-suite Google Drive`_. The files are located under ``DEIMOS_Dev/Throughput/throughput_gdw/data_products/``.
 
-        rclone -P copy remote:/PypeIt-development-suite/DEIMOS_Dev/Throughput/throughput_gdw/data_products/extract/ source_files/
-
-3. Run the ``create_deimos_sensfuncs.py`` script.
+3. Run the ``create_deimos_sensfuncs.py`` script. The below examples assumes the files
+   were downloaded to a ``data_products`` directory and will create the archival 
+   sensitivity function files under ``sensfunc_files``.
 
    .. code-block:: bash
 
         mkdir sensfunc_files
-        PypeIt-development-suite/sensfunc_archive/create_deimos_sensfuncs.py all source_files sensfunc_files
+        $PYPEIT_DEV/sensfunc_archive/create_deimos_sensfuncs.py all data_products sensfunc_files
+
+   The above creates spec1d files and sensfunc files from the reduced slitless source images. It then
+   stitches those sensfunc files together to form sensfuncs that cover a wider wavelength range.  This results
+   in one stitched archival sensfunc for each grating (600ZD, 830G, 900ZD, 1200B, and 1200G).
+
+   To stitch together already created sensfunc files use the ``--reuse`` option. The pre-stitch sensfunc files used to 
+   create the archival files are in the Google Drive under ``DEIMOS_Dev/sensfunc_stitch/pre_stitch_sensfuncs/``
 
 4. Copy the results to the PypeIt repository.
 
@@ -31,4 +68,66 @@ How to build archived sensitivity functions
         cd sensfunc_files
         cp keck_deimos_*.fits PypeIt/pypeit/data/sensfuncs/
 
+How each DEIMOS sensitivity functions was built
+***********************************************
+
+The archived sensitivity functions for DEIMOS were created from raw data
+gathered from scripted standard star observations. These are slitless
+observations that PypeIt cannot reduce on its own, so they are reduced using
+IDL scripts from Greg Wirth. 
+
+Examples of this data can be found in the `PypeIt dev-suite Google Drive`_ Google Drive.
+
++----------------------------------------------------------------+-------------------------------------------------------------------------------+
+| ``DEIMOS_Dev/Throughput/YYYY``                                 | Contains raw data files from that year.                                       |
++----------------------------------------------------------------+-------------------------------------------------------------------------------+
+| ``DEIMOS_Dev/Throughput/throughput_gdw/data_products/extract`` | Contains the reduced data from IDL used to generate the sensitivity functions.|
++----------------------------------------------------------------+-------------------------------------------------------------------------------+
+
+
+Converting the IDL reduced files to spec1d
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In the PypeIt-development-suite repo,  the ``batch_convert_throughput_to_spec1d.py``
+script can create PypeIt spec1d files from the IDL output.
+
+By default, this script splits the IDL output into two detectors (DET03 and DET07). Because the IDL output isn't always
+even in length, it is neccessary to pass "pad" to the script. Alternatively the script can generate spec1d files by treating the IDL output as a 
+mosaic ('MSC03') by passing "nosplit":
+
+For example, if the files from Google Drive was downloaded to a
+``data_products`` directory. The below would create a ``spec1d_files`` directory with
+spec1d files created from the IDL output:
+
+   .. code-block:: bash
+
+      # Create two detector spec1d files
+      python $PYPEIT_DEV/dev_algorithms/fluxing/batch_convert_throughput_to_spec1d.py data_products/extract spec1d_files pad
+
+      # Create a mosaic spec1d file
+      python $PYPEIT_DEV/dev_algorithms/fluxing/batch_convert_throughput_to_spec1d.py data_products/extract spec1d_files nosplit
+
+The current DEIMOS archival sensfuncs were created by splitting the output into two detectors using the "pad" parameter.
+
+Creating the individual sensitivity functions
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+The individual DEIMOS sensitivity functions were created with the following ``pypeit_sensfunc`` input file (See :ref:`sensitivity_file`).
+
+   .. code-block:: ini
+
+      [sensfunc]
+         multi_spec_det = 3,7
+         algorithm = "IR"
+         extrap_blu = 0.
+         extrap_red = 0.
+
+
+The sensitivity functions can be created in bulk using the ``run_sensfunc_on_all_spec1d.py`` script.
+Assuming the sensfunc input file above is named ``deimos_arxiv_sensfunc.sens``, the below
+example will create individual sensfunc files for each matching 600ZD spec1d file and place
+them in the ``sens_files`` directory:
+
+   .. code-block:: bash
+
+      python $PYPEIT_DEV/dev_algorithms/fluxing/run_sensfunc_on_all_spec1d.py spec1d_files/600ZD/ *2023jan17*.fits sens_files/600ZD deimos_arxiv_sensfunc.sens
 

doc/fluxing.rst

@@ -510,8 +510,7 @@ view them.
 Archival Sensitivity Functions
 ------------------------------
 
-PypeIt supports using archived sensitivity functions for flux calibration. Currently only
-experimental ``keck_deimos`` sensitivity files are available.  They can be applied by adding
+PypeIt supports using archived sensitivity functions for flux calibration. They can be applied by adding
 ``use_archived_sens = True`` to the flux file passed to ``pypeit_flux_calib``. For example:
 
 .. code-block:: ini
@@ -523,6 +522,11 @@ experimental ``keck_deimos`` sensitivity files are available.  They can be appli
         spec1d_d1010_0056-HIT2015-mask03_DEIMOS_20151010T045816.550.fits
     flux end
 
+*Disclaimer*: Currently only ``keck_deimos`` sensitivity files are available. These DEIMOS archival 
+sensitivity functions do not provide an absolute flux calibration.  Instead, they are only intended to 
+remove the instrumental response, providing a relative flux calibration up to some unknown normalization.
+
+
 Troubleshooting
 ===============
 

doc/releases/1.14.1dev.rst

@@ -45,6 +45,7 @@ Functionality/Performance Improvements and Additions
 Instrument-specific Updates
 ---------------------------
 
+- Updated archival sensivity functions for DEIMOS 1200G, 800G, and 600ZD gratings.
 - Keck/KCWI and Keck/KCRM: Turned on polynomial correction for sky subtraction.
 - We now support the reduction of VLT/FORS2 data taken in MOS mode.
 - Fixed fluxing file format in the ``Keck-DEIMOS HOWTO`` documentation.

pypeit/scripts/collate_1d.py

@@ -776,16 +776,17 @@ def main(args):
         # Flux the spec1ds based on a archived sensfunc
         spec1d_failure_msgs = []
         copied_spec1d = False
-        if par['collate1d']['flux']:
+        if par['collate1d']['flux'] and not args.dry_run:
             if par['collate1d']['spec1d_outdir'] is not None:
                 # Fluxing modifies the spec1d files, copy them to a new output directory
                 # if requested
                 spec1d_files = copy_spec1d_to_outdir(spec1d_files, par['collate1d']['spec1d_outdir'])
                 copied_spec1d = True                
             spec1d_files = flux(par, spectrograph, spec1d_files, spec1d_failure_msgs)
+
 
         # Perform reference frame correction
-        if par['collate1d']['refframe'] in  ['heliocentric', 'barycentric']:
+        if par['collate1d']['refframe'] in  ['heliocentric', 'barycentric'] and not args.dry_run:
             if not copied_spec1d and par['collate1d']['spec1d_outdir'] is not None:
                 # Refframe correction modifies the spec1d files, copy them to a new output directory
                 # if requested and fluxing hasn't already done so
@@ -810,12 +811,18 @@ def main(args):
         successful_source_list = []
         failed_source_msgs = []
         for source in source_list:
+
             coaddfile = os.path.join(par['collate1d']['outdir'], build_coadd_file_name(source))
             msgs.info(f'Creating {coaddfile} from the following sources:')
             for i in range(len(source.spec_obj_list)):
                 msgs.info(f'    {source.spec1d_file_list[i]}: {source.spec_obj_list[i].NAME} '
                           f'({source.spec_obj_list[i].MASKDEF_OBJNAME})')
 
+            # Exclude sources with a single object to coadd
+            if len(source.spec_obj_list) == 1:
+                excluded_obj_msgs.append(f"Excluding {source.spec_obj_list[0].NAME} in {source.spec1d_file_list[0]} because there's no other SpecObj to coadd with.")
+                continue
+
             if not args.dry_run:
                 try:
                     coadd(par, coaddfile, source)

pypeit/spectrographs/keck_deimos.py

@@ -1900,7 +1900,7 @@ def deimos_read_1chip(hdu,chipno):
     return data, oscan
 
 
-def load_wmko_std_spectrum(fits_file:str, outfile=None, pad = False):
+def load_wmko_std_spectrum(fits_file:str, outfile=None, pad = False, split=True):
     """Load up a Standard spectrum generated by WMKO IDL scripts
     of the great Greg Wirth
 
@@ -1909,6 +1909,9 @@ def load_wmko_std_spectrum(fits_file:str, outfile=None, pad = False):
     Args:
         fits_file (str): filename
         outfile ([type], optional): Write the SpecObjs object to a FITS file. Defaults to None.
+        pad (bool,optional): True if the resulting SpecObjs should be padded to an even length. Defaults to False
+        split (bool,optional): True if the resulting SpecObjs should be split into two detectors, False if it
+                               should be treated as a mosaic. Defaults to True.
 
     Returns:
         specobjs.SpecObjs: object holding the spectra
@@ -1920,7 +1923,10 @@ def load_wmko_std_spectrum(fits_file:str, outfile=None, pad = False):
     idl_spec = Table(hdul[2].data)
 
     # Hope this always works..
-    npix = int(len(idl_spec)/2)
+    if split:
+        npix = int(len(idl_spec)/2)
+    else:
+        npix = len(idl_spec)
 
     if pad:
         if int(len(idl_spec)) % 2 != 0:
@@ -1931,27 +1937,37 @@ def load_wmko_std_spectrum(fits_file:str, outfile=None, pad = False):
     idl_vac = wave.airtovac(idl_spec['WAVELENGTH']*units.AA)
 
     # Generate SpecObj
-    sobj1 = specobj.SpecObj.from_arrays('MultiSlit', idl_vac.value[0:npix],
-                                  idl_spec['COUNTS'].data[0:npix], 
-                                   1./(idl_spec['COUNTS'].data[0:npix]),
-                                   DET='DET03')
-    sobj2 = specobj.SpecObj.from_arrays('MultiSlit', idl_vac.value[npix:],
-                                  idl_spec['COUNTS'].data[npix:], 
-                                   1./(idl_spec['COUNTS'].data[npix:]), 
-                                   DET='DET07')
+    if not split:
+        sobj1 = specobj.SpecObj.from_arrays('MultiSlit', idl_vac.value[0:npix],
+                                    idl_spec['COUNTS'].data[0:npix], 
+                                    1./(idl_spec['COUNTS'].data[0:npix]),
+                                    DET='MSC03')
+    else:
+        sobj1 = specobj.SpecObj.from_arrays('MultiSlit', idl_vac.value[0:npix],
+                                    idl_spec['COUNTS'].data[0:npix], 
+                                    1./(idl_spec['COUNTS'].data[0:npix]),
+                                    DET='DET03')
+
+        sobj2 = specobj.SpecObj.from_arrays('MultiSlit', idl_vac.value[npix:],
+                                    idl_spec['COUNTS'].data[npix:], 
+                                    1./(idl_spec['COUNTS'].data[npix:]), 
+                                    DET='DET07')
 
     # SpecObjs
     sobjs = specobjs.SpecObjs()
     sobjs.add_sobj(sobj1)
-    sobjs.add_sobj(sobj2)
+    if split:
+        sobjs.add_sobj(sobj2)
 
     # Fill in header
     coord = linetools.utils.radec_to_coord((meta['RA'][0], meta['DEC'][0]))
     sobjs.header = dict(EXPTIME=1., 
-                        AIRMASS=float(meta['AIRMASS']), 
+                        AIRMASS=float(meta['AIRMASS'][0]), 
                         DISPNAME=str(meta['GRATING'][0]), 
+                        SLMSKNAM='', # No slitmask is used for the throughput data
                         PYP_SPEC='keck_deimos', 
                         PYPELINE='MultiSlit',
+                        BINNING=f'{meta["SPEC_BIN"][0]},{meta["SPEC_BIN"][0]}',
                         RA=coord.ra.deg, 
                         DEC=coord.dec.deg
                    )