-
Notifications
You must be signed in to change notification settings - Fork 1
Python Interface
The pipeline is separated into two distinct sections preprocessing and reduction
See [examples](*Instrument Examples) of the following instruments:
Before running the pipeline, make sure that all raw FITS image files are placed in a single directory. The pipeline will automatically select image types via header keyword OBSTYPE. See the following example LMI raw data structure. (Note: raw file formatting is found in specific_instruments.py or see Instrument Examples
LMI example:
~/imdata
|
| lmi.0001.fits
| lmi.0002.fits
| lmi.0003.fits
| lmi.0004.fits
| ....
Pipeline assumes that the data directory contains all of the raw files. During the preprocess step it will create the following directory structure:
~/imdata
|
|----calibration # all calibration files from the preprocess step will appear here
|----selected # all science files selected and formatted by the preprocess step will appear here
|----reduced # all files created by the autoproc step will appear here
|----coadd # coadded frames from the autoproc step will be moved here to be ready for the photometry step
Run the following code in a python terminal, replacing 'instrument' with the correct instrument keyname 'ratir', 'lmi', 'rimas', etc. Also allow 'cams' to simply be a list of all cameras for a particular instrument. (ex: In 'lmi' cams = [0], in 'ratir' cams = [0,1,2,3], in 'rimas' cams = [0,1]). Let 'workdir' be the directory in which the raw FITS files are stored. See choose_calib for a comprehensive list of parameters.
from photopipe.reduction.preprocess import choose
# Bias frames calibration
bias_calib = choose.choose_calib( 'instrument',
'bias',
workdir='~/imdata',
cams=[0,1,...],
auto=True,
amin=0.0, amax=1.0,
reject_sat=False,
save_select=True,
noplot=False )
# Dark frames calibration
dark_calib = choose.choose_calib( 'instrument',
'dark',
workdir='~/imdata',
cams=[0,1,...],
auto=True,
amin=0.0, amax=1.0,
reject_sat=False,
save_select=True,
noplot=False )
# Flat frames calibration
flat_calib = choose.choose_calib( 'instrument',
'flat',
workdir='~/imdata',
cams=[0,1,...],
auto=True,
amin=0.0, amax=1.0,
reject_sat=False,
save_select=True,
noplot=False ) Every image filename will be changed to display the time, image type (b -> 'bias', d -> 'dark', f -> 'flat', o -> 'object'), and camera number (C0, C1, ...). Also a dictionary of the selected frames by image type will be saved for future use.
Example
~/imdata
|
| 20210505T005040C0b.fits
| 20210505T016389C0d.fits
| 20210505T047384C0f.fits
| 20210505T064357C0o.fits
| ....
| bias_*.p
| dark_*.p
| flat_*.p
|
|----calibration # all calibration files from the preprocess step will appear here
|----selected # all science files selected and formatted by the preprocess step will appear here
|----reduced # all files created by the autoproc step will appear here
|----coadd # coadded frames from the autoproc step will be moved here to be ready for the photometry step
This step preforms the same operations from the previous step to the object frames and saves copies of the object frames to the 'targetdir' which is the 'selected' folder by default. See choose_science for a comprehensive list of parameters.
Run the following using the same notation as before
# Select science frames
science_dict = choose.choose_science( 'instrument',
workdir='~/imdata/,
targetdir='~/imdata/selected/',
cams=[0,1,...],
auto=True,
save_select=True,
calibrate=False,
noplot=False ) Now the selected folder stores copies of the object FITS files and will save a dictionary of the selected object frames to ~/imdata.
Example
~/imdata
|
|----calibration # all calibration files from the preprocess step will appear here
|----selected # all science files selected and formatted by the preprocess step will appear here
| | 20210505T064357C0o.fits
| | 20210505T069374C0o.fits
| | ....
|
|----reduced # all files created by the autoproc step will appear here
|----coadd # coadded frames from the autoproc step will be moved here to be ready for the photometry step
Next the pipeline will combine frames by type to create each a master bias, master dark, and master flat using mkmaster. The pipeline will also move master files to the calibration folder.
from photopipe.reduction.preprocess import master
# Make master frames
master.mkmaster('instrument', bias_calib, 'bias')
master.mkmaster('instrument', dark_calib, 'dark')
master.mkmaster('instrument', flat_calib, 'flat') The master frames will be saved in your current directory, or else you can pass the argument master_dir (TODO, add argument and link to preproc).
Example
~/imdata
|
|----calibration # all calibration files from the preprocess step will appear here
| |
| | bias_[camera].fits
| | ....
| | flat_[filter].fits
| | ....
| | dark_[camera].fits
| | ....
|
|----selected # all science files selected and formatted by the preprocess step will appear here
|----reduced # all files created by the autoproc step will appear here
|----coadd # coadded frames from the autoproc step will be moved here to be ready for the photometry step
This step involves the autoproc.py method. See autoproc for parameter details:
from photopipe.reduction.auto.autoproc import autoproc
autoproc(datadir=~/imdata/selected/,
imdir=~/imdata/reduced/,
redo=1, nomastersky=True)Once this is done, the final Coadded frames are produced and put in a single directory for photometry
Example
~/imdata
|
|----calibration # all calibration files from the preprocess step will appear here
|----selected # all science files selected and formatted by the preprocess step will appear here
|----reduced # all files created by the autoproc step will appear here
| | p20210505T064357C0o.fits
| | fp20210505T064357C0o.fits
| | sfp20210505T064357C0o.fits
| | zsfp20210505T064357C0o.fits
| | azsfp20210505T064357C0o.fits
| | tazsfp20210505T064357C0o.fits
| | ....
|
|----coadd # coadded frames from the autoproc step will be moved here to be ready for the photometry step
| | coaddGRB180618A_20210505T035244220_SDSS-Z
| | ....
Does not require 'dark' reduction or preprocessing.
Key Parameters:
- instrument = 'lmi'
- cams = [0]
from photopipe.reduction.preprocess import choose
from photopipe.reduction.preprocess import master
from photopipe.reduction.auto.autoproc import autoproc
# Bias frames calibration
bias_calib = choose.choose_calib( 'lmi',
'bias',
workdir='~/imdata',
cams=[0],
auto=True,
amin=0.0, amax=1.0,
reject_sat=False,
save_select=True,
noplot=False )
# Flat frames calibration
flat_calib = choose.choose_calib( 'lmi',
'flat',
workdir='~/imdata',
cams=[0],
auto=True,
amin=0.0, amax=1.0,
reject_sat=False,
save_select=True,
noplot=False )
# Object Frame Calibration
science_dict = choose.choose_science( 'lmi',
workdir='~/imdata/,
targetdir='~/imdata/selected/',
cams=[0],
auto=True,
save_select=True,
calibrate=False,
noplot=False )
# Make master frames
master.mkmaster('lmi', bias_calib, 'bias')
master.mkmaster('lmi', flat_calib, 'flat')
# Automatic Processing
autoproc(datadir=~/imdata/selected/,
imdir=~/imdata/reduced/,
redo=1, nomastersky=True)Key Parameters:
- instrument = 'ratir'
- cams = [0,1] For bias and dark calibration only
- cams = [0,1,2,3] For flat and object calibration only
- amin = 0.2, amax = 0.8 For flat and object calibration only
from photopipe.reduction.preprocess import choose
from photopipe.reduction.preprocess import master
from photopipe.reduction.auto.autoproc import autoproc
# Bias frames calibration
bias_calib = choose.choose_calib( 'ratir',
'bias',
workdir='~/imdata',
cams=[0,1],
auto=True,
amin=0.0, amax=1.0,
reject_sat=False,
save_select=True,
noplot=False )
# Dark frames calibration
dark_calib = choose.choose_calib( 'ratir',
'dark',
workdir='~/imdata',
cams=[0,1],
auto=True,
amin=0.0, amax=1.0,
reject_sat=False,
save_select=True,
noplot=False )
# Flat frames calibration
flat_calib = choose.choose_calib( 'ratir',
'flat',
workdir='~/imdata',
cams=[0,1,2,3],
auto=True,
amin=0.2, amax=0.8,
reject_sat=False,
save_select=True,
noplot=False )
# Object Frame Calibration
science_dict = choose.choose_science( 'ratir',
workdir='~/imdata/,
targetdir='~/imdata/selected/',
cams=[0,1,2,3],
auto=True,
save_select=True,
calibrate=False,
noplot=False )
# Make master frames
master.mkmaster('ratir', bias_calib, 'bias')
master.mkmaster('ratir', dark_calib, 'dark')
master.mkmaster('ratir', flat_calib, 'flat')
# Automatic Processing
autoproc(datadir=~/imdata/selected/,
imdir=~/imdata/reduced/,
redo=1, nomastersky=True)Key Parameters:
- instrument = 'rimas'
- cams = [0,1]
from photopipe.reduction.preprocess import choose
from photopipe.reduction.preprocess import master
from photopipe.reduction.auto.autoproc import autoproc
# Bias frames calibration
bias_calib = choose.choose_calib( 'rimas',
'bias',
workdir='~/imdata',
cams=[0,1],
auto=True,
amin=0.0, amax=1.0,
reject_sat=False,
save_select=True,
noplot=False )
# Flat frames calibration
flat_calib = choose.choose_calib( 'rimas',
'flat',
workdir='~/imdata',
cams=[0,1],
auto=True,
amin=0.0, amax=1.0,
reject_sat=False,
save_select=True,
noplot=False )
# Object Frame Calibration
science_dict = choose.choose_science( 'rimas',
workdir='~/imdata/,
targetdir='~/imdata/selected/',
cams=[0,1],
auto=True,
save_select=True,
calibrate=False,
noplot=False )
# Make master frames
master.mkmaster('rimas', bias_calib, 'bias')
master.mkmaster('rimas', flat_calib, 'flat')
# Automatic Processing
autoproc(datadir=~/imdata/selected/,
imdir=~/imdata/reduced/,
redo=1, nomastersky=True)