Home

Running the Pipeline

These steps will walk through an example run-through of the photometry pipeline preprocessing and reduction

This includes details on:

• How to run the pipeline
• Important methods
• Example input/output using LMI format

See Readme.md for an abbreviated version of these steps using Ratir format

The pipeline is separated into two distinct sections preprocessing and reduction

Preprocessing

1. Initialization

• Before running the pipeline, make sure that all FITS image files are placed in a single directory. The pipeline will automatically select image types via header keyword 'OBSTYPE'.

• Create two sub-directories called 'selected' and 'reduced' for future use. The 'selected' directory will store the selected object files and the 'reduced' directory will store files saved during the reduction

** Lmi Example**

imdata  
│
│ lmi.0001.fits
│ lmi.0002.fits
| ...
│   
└───selected
│
└───reduced

2. Bias/Dark/Flat frame Selection and Calibration

Next the pipeline will standardize the image file names and headers and begin calibrating bias, dark, and flat frames using choose.choose_calib(). See Adding New Instruments for the standard header and filename information.

Run the following code in a python terminal, replacing 'instrument' with the correct instrument keyname 'ratir', 'lmi', 'rimas', etc. Also allow 'cams' to simply be a list of all cameras for a particular instrument. (ex: In 'lmi' cams = [0], in 'ratir' cams = [0,1,2,3]). Finally, let 'workdir' be the directory in which the FITS images are stored.

 from photopipe.reduction.preprocess import choose
 
 # Bias frames calibration 
 
 bias_calib = choose.choose_calib( 'instrument', 
                                    'bias', 
                                    workdir='~/imdata/', 
                                    cams=[0,1,...], 
                                    auto=True, 
                                    amin=0.0, amax=1.0, 
                                    reject_sat=False, 
                                    save_select=True, 
                                    noplot=False )
 # Dark frames calibration 
 
 dark_calib = choose.choose_calib( 'instrument', 
                                    'dark', 
                                    workdir='~/imdata/', 
                                    cams=[0,1,...], 
                                    auto=True, 
                                    amin=0.0, amax=1.0, 
                                    reject_sat=False, 
                                    save_select=True, 
                                    noplot=False )
 # Flat frames calibration 
 
 flat_calib = choose.choose_calib( 'instrument', 
                                    'flat', 
                                    workdir='~/imdata/', 
                                    cams=[0,1,...], 
                                    auto=True, 
                                    amin=0.0, amax=1.0, 
                                    reject_sat=False, 
                                    save_select=True, 
                                    noplot=False )  

Each image type calibration will save a corresponding p file in the image directory. Additionally, every image filename will be changed to display the time, image type (b -> 'bias', d -> 'dark', f -> 'flat', o -> 'object'), and camera number (C0, C1, ...).

** Lmi Example**

imdata  
│
│ 20200129T010235C0f.fits
│ 20200129T021605C0o.fits
| 20200129T010115C0b.fits
| ...
| bias_20200908T172930.p
| dark_20200908T175521.p
| flat_20200908T175539.p
│   
└───selected
│
└───reduced

2. Object frame Selection and Calibration

This step preforms the same operations from the previous step to the object frames and saves copies of the object frames to the 'selected' folder.

Run the following using the same notation as before

  # Select science frames
  
  science_dict = choose.choose_science( 'instrument', 
                                         workdir='~/imdata/, 
                                         targetdir='~/imdata/selected/', 
                                         cams=[0,1,...], 
                                         auto=True, 
                                         save_select=True, 
                                         calibrate=False, 
                                         noplot=False ) 

Now the selected folder stores copies of the object image frames and a p file for the object frames is created in the main directory.

** Lmi Example**

imdata  
│
│ 20200129T010235C0f.fits
│ 20200129T021605C0o.fits
| 20200129T010115C0b.fits
| ...
| bias_20200908T172930.p
| dark_20200908T175521.p
| flat_20200908T175539.p
| object_20200908T176349.p
│   
└───selected
│   | 20200129T021605C0o.fits
|   | ...
│
└───reduced

3. Creating Master Bias/Dark/Flat

Next the pipeline will combine frames by type to create each a master_bias, master_dark, and master_flat using master.mkmaster()

Run:

  from photopipe.reduction.preprocess import master

  # Make master frames
  
  master.mkmaster('instrument', bias_calib, 'bias')

  master.mkmaster('instrument', dark_calib, 'dark')
 
  master.mkmaster('instrument', flat_calib, 'flat')  

The master frames will each be saved in your current directory. Make sure to move them into the 'reduced' directory before continuing onto the automatic preocessing. After moving the master frames, the directory should look similar to the following example. Note: for this example there are no 'dark' frames so the 'master dark' is missing.

** Lmi Example**

imdata  
│
│ 20200129T010235C0f.fits
│ 20200129T021605C0o.fits
| 20200129T010115C0b.fits
| ...
| bias_20200908T172930.p
| dark_20200908T175521.p
| flat_20200908T175539.p
| object_20200908T176349.p
│   
└───selected
│   | 20200129T021605C0o.fits
|   | ...
│
└───reduced
│   | bias_C0.fits
|   | flat_SDSS-R.fits

Automatic Processing

This steps involves only one action, running the autoproc.py method autoproc() using the following:

from photopipe.reduction.auto.autoproc import autoproc

autoproc(datadir=~/imdata/selected/,
         imdir=~/imdata/reduced/,
         redo=1, nomastersky=True)

Once this is done, the final Coadded frames are produced and can be put in a single directory for future photometry

The rest of these instructions will walk through the substeps of this final procedure.

autoproc.py has eight steps:

• Prepare
• Flatten
• Make Master Sky
• Sky Subtraction
• Cosmic Ray Cleaning
• Astrometry
• Zero point and Flux Scale Calculations
• Stacking

The methods for these steps can be found in ~\reduction\auto\steps Note: Each step only selects files with the prefix corresponding to the step before it (excluding the first step)