Code Validation

Introduction

BioNetGen is currently tested by running a suite of validation models and comparing the output to a database of curated results. The process is semi-automated by the script validate_examples.pl found in the Validate subfolder of bng2. The "database" of curated results is contained in the folder Validate\DAT_validate.

At present, there are no unit tests for BNG code.

Model Validation

Several options are available for model validation:

• Comparison of .net, .bngl and .m output files by diff comparison (ignores differences in whitespace).
• Quantitative comparison of ODE trajectories.
• Statistical comparison of equilibrium distributions generated by SSA or NFsim (Chi-square test).

Recommendations

• When developers check in code that implements a new feature, they are encouraged to add a relevant validation model concurrently.
• Model output should be validated either manually or by comparison to published results before adding reference output to the database.

validate_examples.pl documentation

The following material is copied from the comments in validate_examples.pl. Please see the script for up-to-date documentation.

SYNOPSIS:
   validate_examples.pl [OPTIONS]              : validate all models
   validate_examples.pl [OPTIONS] MODEL...     : validate MODEL
   validate_Examples.pl --help                 : display help menu

Exits with value 0 if all validation tests are passed, otherwise exits with a positive value equal to the number of failed tests, or -1 (=255) if there was some problem executing the validation.

A log file named MODEL.log is created for each validation model. If a test fails, check the log file for more details.

What is New?

• 8sep2011: validation of equilibrium distribution for SSA models
• 19apr2012: better support under Windows OS
• 19apr2012: improved logfiles and new console "look"

More Details: This script will automatically recognize and validate the following output files for each MODEL in the models directory:

output file	reference file	description
MODEL.net	MODEL.net	reaction network
MODEL.xml	MODEL.xml	XML specification
MODEL.gdat	MODEL.gdat	ODE observables trajectory
MODEL.cdat	MODEL.cdat	ODE species trajectory
MODEL_ssa_equil.gdat	MODEL_ssa_equil.stats	SSA equilibrium samples
MODEL_nf_equil.gdat	MODEL_nf_equil.stats	NFsim equilibrium samples

To add new validation MODEL:

1. Create MODEL.bngl, including actions that generate some or all of the recognized output files.
2. Put the validation model in $modeldir directory.
3. Generate reference output files and place in the $datdir subdirectory (see notes).

Notes:

• For most validations, the reference file shares the same format and name as the output file. However, special reference files are required for validating stochastic equilibrium samples. These reference files have a ".stat" extension and describe a binned equilibrium distribution of a model observable. See $datdir/gene_expr_ssa_equil.stats for an example. Equilibrium samples are compared to the reference distribution by a Chi-square goodness of fit test. The stats file defines the observable of interest, bin widths and probabilities, and the Chi-square values corresponding to various significance levels.
• It is the Modeler's responsibility to validate the reference trajectory. It's advisable to compare the reference to analytic results, simulations reported in the literature, or simulations generated from independent platforms (e.g. MATLAB).