Adding new output variables

Introduction

Evosuite provides a method for data to be recorded during execution via the -Doutput_variables setting. This page will provide a guide on how to use the setting, as well as how to extend its functionality.

Background

The following classes are relevant to the functionality at hand:

• org.evosuite.Properties.StatisticsBackend: The type of statistics backend to use when writing output variables.
• org.evosuite.statistics.RuntimeVariable: A list of runtime variables to choose from.
• org.evosuite.statistics.SearchStatistics: Class that collects data values reported by a single client node and writes it to backend.
• org.evosuite.rmi.service.MasterNodeImpl: Master node that takes in output variables to save after Evosuite execution.
• org.evosuite.rmi.service.ClientNodeImpl: Client node that runs search and tracks output variables.

Listeners and Senders are also tangentially relevant but are not part of the core functionality; hence their discussion is omitted here.

Certain useful choices for output variables include:

• TARGET_CLASS: The class that the MUT belongs to.
• target_method: The name of the MUT.
• criterion: Coverage criterion.
• CoverageTimeline: The coverage of the generated test suite, recorded at set time intervals as determined by -Dtimeline_interval.

How to use -Doutput_variables

During Evosuite invocation, add the following flag(s) to your settings:

-Doutput_variables <variables> [-Dtimeline_interval <timeline_interval>]

where

• <variables> := The output variables you wish to track, separated by a comma (,).
• <timeline_interval> := The interval for timeline variables, in milliseconds. Only use this if you are tracking timeline variables.

The default setting for the statistics backend (which decides where the output variables are recorded) is StatisticsBackend.CSV, so if you have changed it from the default (e.g. if you are running directly in feature.objectconstruction.testgeneration.testcase.SF100OverallTest and have changed the default in beforeTest()) you will have to change it to this value.

The list of possible output variables can be found in org.evosuite.statistics.RuntimeVariable and Properties.getParameters(). Some of the output variables are timeline variables - special variables that track something over intervals as set in -Dtimeline_interval.

After setting these flags, the data chosen will be recorded in /{project_folder}/evosuite-report/statistics.csv (i.e. if you are running methods from e.g. 89_jiggler, the statistics will be saved in /89_jiggler/evosuite-report/statistics.csv). In any case, the section of code responsible for saving the statistics (org.evosuite.statistics.SearchStatistics:397,428) can be augmented with print statements to explicitly mention where the statistics are being saved to.

How to extend the functionality

It is a very probable scenario that new empirical studies in the future might require recording new types of data not available in the existing selection of runtime variables. This section will outline how to do so.

1. Add a new RuntimeVariable

   Add a new entry in the org.evosuite.statistics.RuntimeVariable enum. The entry will be the name used to refer to your new output variable when selecting it in -Doutput_variables. You should also add a brief description of what your new output variable represents.

2. Add logic to generate your data

   This can be done any time before the output variable is sent out (the trackOutputVariable call in step 3).

3. Track output variable

   Add a call to org.evosuite.rmi.service.ClientNodeLocal.trackOutputVariable(<your_runtime_variable>, <variable_value>) anywhere before org.evosuite.rmi.service.MasterNodeImpl.evosuite_flushStatisticsForClassChange(String) is called. This method will trigger a writing of the output variables to the statistics backend.

An example of how this may be done can be found here.

TODO: Add a guide on how to add timeline variables.