Skip to content

CLI

Jump to bottom
durba edited this page Aug 11, 2021 · 1 revision

Command Line Interface

Usage

photopipe_cli.py [-h] [-d DATADIR] [--nopreprocess] [--noautoproc]
                        [--nophotometry] [--instrument INSTRUMENT]
                        [--cams CAMS] [--noautoselect] [--norejectsat]
                        [--amin AMIN] [--amax AMAX] [--nosaveselect]
                        [--fmin FMIN] [--noplot] [--nobias] [--noflat]
                        [--nodark] [--noscienceselect] [--start START]
                        [--stop STOP] [--step STEP] [--nocrclean]
                        [--nomastersky] [--noskyflattarg] [--redo] [--quiet]
                        [--rmifiles] [--customcat CUSTOMCAT]
                        [--customcatfilt CUSTOMCATFILT] [--debug]
                        cmd

Pipeline assumes that the data directory contains all of the raw files. During the preprocess step it will create the following directory structure:

data
|
|----calibration  # all calibration files from the preprocess step will appear here
|    |
|    | bias_[camera].fits
|    | ....
|    | flat_[filter].fits
|    | ....
|    | dark_[camera].fits
|    | ....
|
|----selected  # all science files selected and formatted by the preprocess step will appear here
|----reduced  # all files created by the autoproc step will appear here
|----coadd  # coadded frames from the autoproc step will be moved here to be ready for the photometry step

LMI example:

Data directory will look something like this:

~/path/to/files
|
| lmi.0001.fits
| lmi.0002.fits
| lmi.0003.fits
| lmi.0004.fits
| ....

From the bash shell we an reduce the data using:

$ photopipe_cli.py pipeline_all --datadir ~/path/to/files --instrument lmi --nodark

cmd

$ phopipe_cli.py [cmd] [optional arguments]

Command to run, options are:

  • pipeline_all - runs preprocess, autoproc and photometry steps
  • preprocess - formats raw data in the datadir to match the pipeline format, sorts science data into selected directory, and generates master bias, flat and dark calibration files that will appear in the calibration directory. Described in more detail below.
  • autoproc - Processes the science data from selected form to final reduced coadded frames. All files from this reduction appear in the reduced folder. Described in more detail below.
  • photometry - Creates a stacked image with all filters (saved to multicolor.fits and multicolor.weight.fits). Crops all of the filter images and combined image to the same size and coordinates (file).ref.multi.fits. Then finds sources using the master combined image, and calculates the magnitude based on just the filtered images (with weight file using sextractor. Saves new sextractor values to "fluxes_(FILTER).txt'. Calculates absolute magnitude errors based on fluxes_(FILTER).txt and keyword in file for absolute zeropoint RMS. Saves final magnitudes to 'finalphot(FILTER).am'. Described in more detail below.
  • test - runs the pipeline_all command on a small zipped data set
  • ipython - starts an ipython shell. Convenient for the docker container.
  • bash - starts a bash shell. Convenient for the docker container.

Optional Arguments

  • -h, --help

    • show this help message and exit

  • -d DATADIR, --datadir DATADIR

    • raw data directory (default: /data)/ pipeline will create nested directories for the following steps if they don't exist already. For example:

        data
   │
   |
   │ 20160628T032914C0b.fits 
   │ 20160628T032914C1b.fits 
   │ ... 
   | └───calibration
   | └───selected
   │ └───reduced
   │ └───coadd

      NOTE If you are using docker, you need to have the data directory mounted as a volume, e.g.

      -v /path/to/data:/data

  • --nopreprocess

    • skips all preprocessing steps:
      • master calibration frame creation - flats, darks, biases
      • science data formatting

  • --noautoproc

    • skips all of the reduction steps:
      • prepare
      • flatten
      • makesky
      • skysub
      • crclean
      • astrometry
      • zpoint
      • stack

  • --nophotometry

    • skips the photometry step

  • --instrument INSTRUMENT

    • instrument data being processed (default: lmi). Options are:
      • lmi
      • ratir
      • rimas

  • --cams CAMS

    • camera numbers (default: all). example:

      photopipe_cli.py pipeline_all --cams '0,1,2,3'

  • --noautoselect

    • no automated frame selection. Requires interactive selection of calibration and science frames

  • --norejectsat

    • don't reject frames based on saturated pixel criteria, e.g --amin and --amax

  • --amin AMIN

    • minimum fraction of saturation value for median during automated flat selection (default: 0.2)

  • --amax AMAX

    • maximum fraction of saturation value for median during automated flat selection, (default: 0.8)

  • --nosaveselect

    • don't save dictionary of selected frames to python pickle file during preprocessing

  • --fmin FMIN

    • Minimum number of files needed to make a master frame (default: 5)

  • --noplot

    • don't show selected frames

  • --nobias

    • skip master bias creation, useful if you already have a master bias in the datadir or have no bias data

  • --noflat

    • skip master flat creation, useful if you already have a master flat in the datadir or have no flat data

  • --nodark

    • skip master dark creation, useful if you already have a master dark in the datadir or have no dark data

  • --noscienceselect

    • skip science data selection, useful if you already did this

  • --start START

    • autoproc starting step, options are: 'prepare', 'flatten', 'makesky', 'skysub', 'crclean', 'astrometry', 'zpoint', 'stack'

  • --stop STOP

    • autoproc stopping step, options are: 'prepare', 'flatten', 'makesky', 'skysub', 'crclean', 'astrometry', 'zpoint', 'stack'

  • --step STEP

    • autoproc only do this step, options are: 'prepare', 'flatten', 'makesky', 'skysub', 'crclean', 'astrometry', 'zpoint', 'stack'

  • --nocrclean

    • skip cosmic ray removal

  • --nomastersky

    • skip master sky creation and use median value instead

  • --noskyflattarg

    • skip creating master sky by target, make master sky using entire observation run instead

  • --redo

    • redo steps that have already been completed

  • --quiet

    • reduces the number of print statements

  • --rmifiles

    • removes intermediate files

  • --customcat CUSTOMCAT

    • Custom catalog (txt file) to determine instrumental zeropoint corrections Must be in same format as what get_SEDs.py produces

      ra(deg) dec(deg) u g r i z y B V R I J H K u_err g_err r_err i_err y_err B_err V_err R_err I_err J_err H_err K_err Mode

      First line will be skipped (use for headers) everything but JHK are expected to be in AB magnitudes, JHK should be in same units as 2MASS (Vega mags) File is expected to be in datadir location

  • --customcatfilt CUSTOMCATFILT

    • Filters relevant to custom catalog file (all other filters will use get_SEDs.py to calculate catalog from 2MASS + (SDSS or APASS or USNOB1) in that order Usage: --customcatfilt "u,g,z"

  • --debug

    • turn on debug outputs

Photopipe command details

preprocess

Formats raw data in the datadir to match the pipeline format, sorts science data into selected directory, and generates master bias, flat and dark calibration files that will appear in the calibration directory.

Does this by running the functions in the choose and master files within photopipe.reduction.preproc

Relevant Optional Arguments

  • --datadir DATADIR
    • expects the raw data to be in this directory
  • --instrument INSTRUMENT
    • Tells the pipeline which file name pattern to look for, and which fits headers to look at. lmi chosen by default
  • --cams
    • Tells the pipeline which cameras you want to work with. Useful if you only want to process certain cameras. All cameras for an instrument by default
  • --noautoselect
    • Turns off automatic selection of frames. Requires user input to confirm which frames are flats, biases, darks and science.
  • --norejectsat
    • Turns off rejection of calibration frames based on median pixel saturation criteria determined by --amin and --amax
  • --amin AMIN
    • minimum percentage of saturation allowed for a calibration frame to be selected
  • --amax AMAX
    • maximum percentage of saturation allowed for a calibration frame to be selected
  • --nosaveselect
    • stops the pipeline from saving the selection dictionary pickle file from auto-selection of frames
  • --fmin FMIN
    • Minimum number of files needed to make a master calibration frame (default: 5)
  • --noplot
    • don't display images and plots during processing
  • --nobias
    • skip bias frame selection and master bias frame creation
  • --noflat
    • skip flat frame selection and master flat frame creation
  • --nodark
    • skip dark frame selection and master dark frame creation
  • --noscienceselect
    • skip science frame selection

autoproc

Processes the raw science data in the selected directory to final reduced coadded frames. All files from this reduction appear in the reduced folder. It does this using the autproc function

Relevant Optional Arguments

  • --datadir DATADIR

    • expects the selected science data from preprocessing to be in the "selected" subdirectory of the DATADIR. Also expects the master calibration data from preprocessing to be in the "calibration" subdirectory of the DATADIR. It puts reduction files in the "reduced" subdirectory of DATADIR.

  • --start START

    • Determines which step you want to start on. Useful if you've already run some steps

  • --stop STOP

    • Determines which step you want to stop on. Useful if you don't want to run all of the steps

  • --step STEP

    • Determines which step you want to run. Useful if you only want to run one step.

  • --nocrclean

    • Skips cosmic ray removal. Cosmic ray removal is time intensive so, if you want fast results, you may want to skip this step.

  • --nomastersky

    • Subtract median value of frame instead of generating and subtracting master sky file.

  • --noskyflattarg

    • Generate master sky using all science frames with the same filter instead of only using science frames from the same target and filter.

  • --redo

    • Redo steps that have already been performed if the step's output files already exist. Useful if you make changes, and don't want to delete the old outputs.

  • --quiet

    • reduce the number of print statements

  • --debug

    • increase the number of print statements, and output files

  • --rmifiles

    • remove intermediate reduction files when autoproc completes

  • --customcat CUSTOMCAT

    • Custom catalog (txt file) to determine instrumental zeropoint corrections Must be in same format as what get_SEDs.py produces

      ra(deg) dec(deg) u g r i z y B V R I J H K u_err g_err r_err i_err y_err B_err V_err R_err I_err J_err H_err K_err Mode

    First line will be skipped (use for headers) everything but JHK are expected to be in AB magnitudes, JHK should be in same units as 2MASS (Vega mags) File is expected to be in datadir location

  • --customcatfilt CUSTOMCATFILT

    • Filters relevant to custom catalog file (all other filters will use get_SEDs.py to calculate catalog from 2MASS + (SDSS or APASS or USNOB1) in that order Usage: --customcatfilt "u,g,z"

photometry

Creates a stacked image with all filters (saved to multicolor.fits and multicolor.weight.fits). Crops all of the filter images and combined image to the same size and coordinates (file).ref.multi.fits. Then finds sources using the master combined image, and calculates the magnitude based on just the filtered images (with weight file using sextractor. Saves new sextractor values to "fluxes_(FILTER).txt'. Calculates absolute magnitude errors based on fluxes_(FILTER).txt and keyword in file for absolute zeropoint RMS. Saves final magnitudes to 'finalphot(FILTER).am'

Does this by running the autoredux function on the data in the coadd directory.

Relevant Optional Arguments

  • --datadir DATADIR
    • expects the coadded data from autoproc to be in the "coadd" subdirectory of the DATADIR.

Installation

Installation Overview

Docker

From Source

Usage

Reference Documentation

Overview

Development

Overview

Clone this wiki locally