Skip to content

Common recipes

Jump to bottom
J Tseng edited this page Jul 13, 2023 · 4 revisions

This page will collect some common graph recipes.

Useful CSV features

A CSV configuration file specifies an analysis graph by listing each module, one line at a time. In each line, the fields are as follows:

  1. Unique name of the module, which will be used by subsequent modules to refer to this one
  2. Plugin name which implements this module. These are found in the snewpdag/plugins directory and its subdirectories. To refer to a plugin a subdirectory (such as the plugin in snewpdag/plugins/gen/TrueTimes), the name is gen.TrueTimes.
  3. List of modules to observe, identified by their unique names. Separate the module names by commas. When the upstream module has an update to send further down the graph, it'll call the appropriate method (such as alert()) of the present module.
  4. Arguments. This takes the form of a python dictionary, but without the enclosing curly brackets.

Any further fields are ignored.

Comments

In a CSV file, you can insert comment lines by starting them with 5 commas, and then enclosing the comment in double quotes.

,,,,,"This is a comment"

Environment variables

You can substitute values from environment variables into your argument lists by prefixing the variable name with a $. For instance, you might specify an average event yield for a generator with the argument

'sig_mean': $YIELD_SNOPLUS

If you need to substitute a string, remember to enclose the substitution with single quotes, i.e., the substitution will take place between the single quotes:

'sig_filename':'$MODEL_PATH'

It is recommended that you list all the environment variables in comments at the top of your CSV file, e.g.,

,,,,,"Environment variables:"
,,,,,"  YIELD_SNOPLUS = mean number of events at SNO+"
,,,,,"  MODEL_PATH = neutrino lightcurve"

Writing fields into the payload

It is often necessary to write something directly into the payload. For instance, in a part of the graph which simulates the SNO+ detector, you will probably need to signal to downstream modules that it's SNO+ data coming down the line. We use the Write plugin to do this.

"SNOP-data","Write","SNOP-signal","'on':['alert'],'write':(('detector','SNOP'),)"

(SNOP-signal here refers to a previously defined module which simulates the SNO+ lightcurve.)

The argument list here tells Write that it will operate only on alert messages, and when it sees one, it will write the string 'SNOP' into the payload field detector. The write argument takes a python tuple as an argument, in which each element is a 2-element tuple containing the field name and the value. Since only one field is written here, the single element is followed by a comma to make sure python knows it's a tuple.

Another common use of Write is to specify filename patterns to be used by downstream modules to write output files. For instance,

"SN","Write","SN-times","'on':['alert','report'],'write':(('png_pattern','output/xwls-q$NBINS-{}-{}-{}.png'),('py_pattern','output/xwls-q$NBINS-{}-{}-{}.png'),)"

(In this case, the last comma is unnecessary, but it's often useful to keep it there just to make sure (if you delete one of the elements) that there's always a last comma to indicate that it's a tuple.)

These fields are passed downstream in the payload, and can be used by renderers to construct the filename of output files. For instance,

"dt1-render","renderers.Hist1D","dt1-acc","'on':['report'],'in_field':'hist1','title':'dt1','xlabel':'sec','ylabel':'entries','filename':'[png_pattern]','scriptname':'[py_pattern]'"

In this case, the renderers.Hist1D plugin will read png_pattern from the payload and use it to specify the name of the output plot file. It will also read the py_pattern field from the payload to write the corresponding script file.

It is a common convention to include -{}-{}-{} in a filename. In most cases, the three fields will be filled with the module name of the renderer, a count index (starting from 0, to ensure that the files don't overwrite eachother), and a burst identifier string read from the payload field burst_id.

It should be mentioned that using square brackets to indicate that an argument should be read from the payload must be implemented by the plugin by invoking the snewpdag library routine fetch_field(); it's not an automatic feature of plugins. If you need to the feature, but the plugin doesn't use fetch_field() yet, feel free to update the code and check it in.

Simulating a lightcurve

There are several methods for simulating a supernova lightcurve.

Time series

This method separates generation into the following steps:

"Control","Pass",,"'line':100"
"SN-times","gen.TrueTimes","Control","'detector_location':'snewpdag/data/detector_location.csv', 'detectors':['SNOP','SK','IC','JUNO'], 'ra':$RA, 'dec':$DEC, 'time':'2021-11-01 05:22:36.328'"
"SNOP-new","ops.NewTimeSeries","SN-times","'out_field':'timeseries','start':'2021-11-01 05:22:34'"
"SNOP-signal","gen.GenTimeDist","SNOP-new","'field':'timeseries','sig_mean':$YIELD_SNOPLUS,'sig_t0':('truth','dets','SNOP','true_t'),'sig_filetype':'tng','sig_filename':'$MODEL_PATH'"
"SNOP-bg","gen.Uniform","SNOP-signal","'field':'timeseries','rate':1,'tmin':'2021-11-01 05:22:35','tmax':'2021-11-01 05:22:55'"
"SNOP-data","Write","SNOP-bg","'on':['alert'],'write':(('detector','SNOP'),)"

Line by line,

  1. This is the start of the graph; you tell the snewpdag application to start a new trial by sending messages to the module named Control.
  2. Generate true burst arrival times at all the detectors, based on a supernova direction. The arguments point to a database of detector parameters, including locations in Earth coordinates. The detectors argument lists the detectors for which the module will generate times. ra is the longitude, and dec the latitude, in the usual convention that the Equator is at 0 degrees. Both ra and dec are in degrees. time gives the time the burst crosses the center of the Earth.
  3. Create an empty time series object and put it in the payload with the key "timeseries". The start time is optional, and isn't really used in this example.
  4. Fill the time series under the "timeseries" key with timestamps from a signal simulation. sig_mean specifies the average number of events to be generated in each trial. sig_t0 indicates how the module will find its burst start time: here, the value is a tuple, which means the module will read the burst start time from the field data['truth']['dets']['SNOP']['true_t'] (which is where gen.TrueTimes will put it without asking you).
  5. Add a uniform background using a flat background generator. This will just append the events to the time series (n.b., the timestamps won't be sorted), with a rate of 1 per second. tmin and tmax give the start and stop times of the uniform distribution. Note that tmin specifies a time before the true burst time specified for gen.TrueTimes.
  6. Write the detector label into the payload so the data is identified properly to subsequent analysis code.

If you don't need the background, omit line 5 and edit line 6 to point back to SN-signal.

Filter by a value in the payload

Sometimes you want to run a part of your analysis (or make a plot) only for one event, to make sure it looks okay. One way to do this is to use the FilterValue plugin:

"First","FilterValue","dt1","'in_field':'trial_id', 'value':0"

This will only pass events with the field trial_id equal to 0. trial_id is a snewpdag counter, starting from 0. Any modules which point back to this module will only be executed if trial_id is 0.

(In this case, dt1 is an analysis module which was specified prior to this one.)

Histograms

Fill from the payload

Let's say your analysis code writes a result such as a time lag in the payload field data['lags'][('SNOP','SK')][0]. You can fill the histogram with the following

"dt1-acc","ops.FillHist1D","dt1","'nbins':160,'xlow':-0.04,'xhigh':0.04,'in_field':('lags',('SNOP','SK'),0), 'out_field':'hist1'"

The module ops.FillHist1D creates the histogram object with the given parameters, and then fills it from the specified payload field, using the tuple method of specifying the payload field. The histogram object is written into the payload as data['hist1'].

Render the histogram

When snewpdag has run through all its trials (i.e., the alerts), it issues a report action. This is the signal for the graph to produce summary plots. To render the histogram specified in the dt1-acc module,

"dt1-render","renderers.Hist1D","dt1-acc","'on':['report'],'in_field':'hist1','title':'dt1','xlabel':'sec','ylabel':'entries','filename':'[png_pattern]','scriptname':'[py_pattern]'"

This module reads data['hist1'] and uses matplotlib.pyplt to render a simple histogram, storing it in the filename specified. scriptname is optional, but if given, it generates a python script which produces the same plot. The script can then be customized for incorporation into talks or papers. The script also contains certain histogram metadata such as the mean and RMS, in comments.

Other renderers include

  • Mollview: a healpix skymap
  • ScatterPlot: and x-y scatter plot
  • MultiPlot: like renderers.Hist1D, but somewhat extended to put multiple graphs in the same plot.
  • FitsSkymap: outputs a healpix skymap in a FITS-format file.
  • JsonOutput: dumps a specified set of payload fields into a JSON-format file. The file can be read back in with JsonAlertInput.
  • PickleOutput: dumps the payload into the given filename. The pickle can be read back in with PickleInput.
Clone this wiki locally