Running the benchmarking program

Syntax

The syntax of benchmarking auxiliary program is:

> perl bin/run_benchmarking.pl  OUTPUT_FILE_PREFIX  INSTANCE_MODE COST_FUNCTION_CODE  k  n   SEARCH_MODE  [max_number_of_calls]

Where:

• OUTPUT_FILE_PREFIX: prefix of the output file name.

• COST_FUNCTION_CODE: code of the employed cost function.

• INSTANCE_MODE: must be 0 or 1; if the value is 0, then for each size of instance in [1,n], it creates k random instances of the chosen cost function. Otherwise, it reads input files already generated, which must have k files per instance size, which in turn must range from 1 to n.

• SEARCH_MODE: must be 0 or 1; if the value is 0, then it performs a complete search, which may be optimal or not depending on the algorithm. Otherwise, it runs a search constrained by 'max_number_of_calls' calls of the cost function.

In the following sections, we show two examples with different usage of the syntax.

---

Example 1:

Running optimal search with subset sum instances created on-the-fly by the benchmarking program.

Try to type:

> perl bin/run_benchmarking.pl test 0 subset_sum 10 7 0

With this command, it should generate an HTML table which contains, for each subset sum instance of size i in [1,7] and for each algorithm j, the average results of 10 runs for:

• Required total time (in seconds);

• Required time to compute all cost function calls (in seconds);

• Number of calls of the cost function (i.e., number of computed nodes);

• Number of times j had the best solution among all algorithms.

It also generates, for each algorithm, an eps graph which depicts the average required computational time (both total and cost function ones) as a function of the instance size. For instance, if the algorithms are "ES", "SFS", "SFFS", and "UBB", then for the example above the /output directory should contain the following files:

  test_ES.eps
  test_SFFS.eps
  test_SFS.eps
  test_table.html
  test_UBB.eps

---

Example 2:

Running an optimal search with Mean Conditional Entropy instances read directly from the /input/tmp directory.

Try to type:

> perl bin/run_benchmarking.pl test 1 mce 3 4 0

In this example, we assume that there are, for each instance size between 1 and 4, at least 3 Mean Conditional Entropy (mce) instances in .dat format insde the input/tmp directory.

---

Example 3:

Running a suboptimal search with Explicit instances created on-the-fly by the benchmarking program, with a limit of 100 calls of the cost function; these instances range from size 1 to 5, with 20 instances per size.

Try to type:

> perl bin/run_benchmarking.pl test 1 explicit 20 5 1 100

In this example, for each instance (regardless of its size), each algorithm will compute the cost function up to 100 times and then end its execution.

---

Back to home