Home

Running the Pipeline

These steps will walk through an example run-through of the photometry pipeline preprocessing and reduction

This includes details on:

• How to run the pipeline
• Important methods
• Example input/output using LMI format

See Readme.md for an abbreviated version of these steps using Ratir format

The pipeline is separated into two distinct sections preprocessing and reduction

Preprocessing

1. Initialization

• Before running the pipeline, make sure that all FITS image files are placed in a single directory. The pipeline will automatically select image types via header keyword 'OBSTYPE'.

• Create two sub-directories called 'selected' and 'reduced' for future use. The 'selected' directory will store the selected object files and the 'reduced' directory will store files saved during the reduction

** Lmi Example**

imdata  
│
│ lmi.0001.fits
│ lmi.0002.fits
| ...
│   
└───selected
│
└───reduced

2. Bias/Dark/Flat frame Selection and Calibration

Next the pipeline will standardize the image file names and headers and begin calibrating bias, dark, and flat frames using choose.choose_calib(). See Adding New Instruments for the standard header and filename information.

Run the following code in a python terminal, replacing 'instrument' with the correct instrument keyname 'ratir', 'lmi', 'rimas', etc. Also allow 'cams' to simply be a list of all cameras for a particular instrument. (ex: In 'lmi' cams = [0], in 'ratir' cams = [0,1,2,3]). Finally, let 'workdir' be the directory in which the FITS images are stored.

 from photopipe.reduction.preprocess import choose
 
 # Bias frames calibration 
 
 bias_calib = choose.choose_calib( 'instrument', 
                                    'bias', 
                                    workdir='~/imdata/', 
                                    cams=[0,1,...], 
                                    auto=True, 
                                    amin=0.0, amax=1.0, 
                                    reject_sat=False, 
                                    save_select=True, 
                                    noplot=False )
 # Dark frames calibration 
 
 dark_calib = choose.choose_calib( 'instrument', 
                                    'dark', 
                                    workdir='~/imdata/', 
                                    cams=[0,1,...], 
                                    auto=True, 
                                    amin=0.0, amax=1.0, 
                                    reject_sat=False, 
                                    save_select=True, 
                                    noplot=False )
 # Flat frames calibration 
 
 flat_calib = choose.choose_calib( 'instrument', 
                                    'flat', 
                                    workdir='~/imdata/', 
                                    cams=[0,1,...], 
                                    auto=True, 
                                    amin=0.0, amax=1.0, 
                                    reject_sat=False, 
                                    save_select=True, 
                                    noplot=False )  

Each image type calibration will save a corresponding p file in the image directory. Additionally, every image filename will be changed to display the time, image type (b -> 'bias', d -> 'dark', f -> 'flat', o -> 'object'), and camera number (C0, C1, ...).

** Lmi Example**

imdata  
│
│ 20200129T010235C0f.fits
│ 20200129T021605C0o.fits
| 20200129T010115C0b.fits
| ...
| bias_20200908T172930.p
| dark_20200908T175521.p
| flat_20200908T175539.p
│   
└───selected
│
└───reduced

2. Object frame Selection and Calibration

This step preforms the same operations from the previous step to the object frames and saves copies of the object frames to the 'selected' folder.

Run the following using the same notation as before

  # Select science frames
  
  science_dict = choose.choose_science( 'instrument', 
                                         workdir='~/imdata/, 
                                         targetdir='~/imdata/selected/', 
                                         cams=[0,1,...], 
                                         auto=True, 
                                         save_select=True, 
                                         calibrate=False, 
                                         noplot=False ) 

Now the selected folder stores copies of the object image frames and a p file for the object frames is created in the main directory.

** Lmi Example**

imdata  
│
│ 20200129T010235C0f.fits
│ 20200129T021605C0o.fits
| 20200129T010115C0b.fits
| ...
| bias_20200908T172930.p
| dark_20200908T175521.p
| flat_20200908T175539.p
| object_20200908T176349.p
│   
└───selected
│   | 20200129T021605C0o.fits
|   | ...
│
└───reduced

3. Creating Master Bias/Dark/Flat

Next the pipeline will combine frames by type to create each a master_bias, master_dark, and master_flat using master.mkmaster()

Run:

  from photopipe.reduction.preprocess import master

  # Make master frames
  
  master.mkmaster('instrument', bias_calib, 'bias')

  master.mkmaster('instrument', dark_calib, 'dark')
 
  master.mkmaster('instrument', flat_calib, 'flat')  

The master frames will each be saved in your current directory. Make sure to move them into the 'reduced' directory before continuing onto the automatic preocessing. After moving the master frames, the directory should look similar to the following example. Note: for this example there are no 'dark' frames so the 'master dark' is missing.

** Lmi Example**

imdata  
│
│ 20200129T010235C0f.fits
│ 20200129T021605C0o.fits
| 20200129T010115C0b.fits
| ...
| bias_20200908T172930.p
| dark_20200908T175521.p
| flat_20200908T175539.p
| object_20200908T176349.p
│   
└───selected
│   | 20200129T021605C0o.fits
|   | ...
│
└───reduced
│   | bias_C0.fits
|   | flat_SDSS-R.fits

Automatic Processing

This steps involves only one action, running the autoproc_2.py method autoproc_2() using the following:

from photopipe.reduction.auto.autoproc_2 import autoproc_2

autoproc_2(datadir=~/imdata/selected/,
         imdir=~/imdata/reduced/,
         redo=1, nomastersky=True)

Once this is done, the final Coadded frames are produced and can be put in a single directory for future photometry

The rest of these instructions will walk through the substeps of this final procedure.

autoproc_2.py has eight steps:

• Prepare
• Flatten
• Fringe Correction
• Sky Subtraction
• Cosmic Ray Cleaning
• Astrometry
• Zero point and Flux Scale Calculations
• Stacking

The methods for these steps can be found in ~\reduction\auto\steps Note: Each step only selects files with the prefix corresponding to the step before it (excluding the first step)

1. Prepare

• Subtracts the master bias and master dark from each object frame
• Deletes all headers besides those in 'mandatorykey' in autoproc_depend.py
• Adds prefix 'p' to the image filename
• Process occurs using prepare.py

2. Flatten

• Divides each object frame by the master flat corresponding to the filter used
• Adds prefix 'f' to the image filename
• Process occurs using prepare.py

3. Fringe Correction

• Removes fringe pattern for susceptible bands (ie. Z-band)

4. Sky Subtraction

• Subtracts the median of the sky for each object frame
• Adds prefix 's' to the image filename
• Process occurs using sky.py

5. Cosmic Ray Cleaning

• Removes cosmic rays using Laplacian cosmic ray identification (see cosmics.py
• Adds prefix 'z' to the image filename
• Process occurs using crclean.py

6. Astrometry

• Preforms quick astrometry calulation to fix WSC coordinates using autoastrometry3.py
• Refines astrometry twice using sextractor and scamp
• Adds prefix 'a' to the image filename
• Process occurs using astrometry.py

7. Zero point and Flux Scale

• Calculates zero point of each object frame
• Uses the zero point to find the relative flux scale
• Removes frames with bad zero point and flux scale values
• Adds prefix 't' to the image filename
• Process occurs using zpoint.py

8. Stack

• Uses flux scale to stack the images into Coadd frames
• Saves Coadd frames along with several relative files used in stacking
• Process occurs using stack.py