Using performance test harness

Introduction

Basic usage:

#include <vt/tests/common/test_harness.h>

using namespace vt::tests::perf::common;

struct MyTest : PerfTestHarness {};

VT_PERF_TEST(MyTest, test_case_name){
  // Test code
}

VT_PERF_TEST_MAIN()

Note: With the current implementation, it's limited to one test per file, meaning you can't have multiple VT_PERF_TEST in a single test file.

• By default, the test function (created with VT_PERF_TEST) will be run 50 times (this value can be changed, using the runtime flag --vt_perf_num_runs)

Example

#include "common/test_harness.h"

using namespace vt::tests::perf::common;

std::vector<int32_t*> ptrs;

void allocateAndTouch(std::size_t num_bytes) {
  auto ptr = new int32_t[num_bytes / 4];
  for (std::size_t i = 0; i < num_bytes / 4; i++) {
    ptr[i] = 20 * i + 23;
  }
  ptrs.push_back(ptr);
}

void deallocate() {
  if (ptrs.size() > 0) {
    auto ptr = ptrs.back();
    ptrs.pop_back();
    delete[] ptr;
  }
}

struct MyTest : PerfTestHarness { };

VT_PERF_TEST(MyTest, test_name) {
  constexpr auto num_iters = 32;

  StartTimer("Allocate");
  for (int i = 0; i < num_iters; i++) {
    allocateAndTouch(1024 * 1024 * 64);
    GetMemoryUsage();
  }
  StopTimer("Allocate");

  StartTimer("Deallocate");
  for (int i = 0; i < num_iters; i++) {
    deallocate();
    GetMemoryUsage();
  }
  StopTimer("Deallocate");
}

VT_PERF_TEST_MAIN()

Command:

mpirun -n 2 /build/vt/tests/example or ctest -L perf_test

Output:

Generating graphs locally

Graphs can be generated locally using Build Stats repository.

1. Run performance test with --vt_perf_gen_file runtime flag, which will generate two CSV files (one for memory usage and one containing timers data)
2. (Optionally) Go to Build Stats directory and create some temporary folder, where the graphs will reside.
3. Run Build Stats' python script generate_graph.py like so: python generate_graph.py -time /path/to/file/test_name_time.csv -mem /path/to/file/test_name_mem.csv

This will generate:

• time execution graph for 'test_name' (entire function created with VT_PERF_TEST)
• separate graph for each unique timer (iterative timers, i.e. 'xxx Bytes' or 'xxx Phase' will be aggregated into single graph)
• memory usage for 'test_name'

4. The above was for the new tests that weren't run on CI before, or if you don't want to compare the execution time for given test with previous CI runs. Otherwise you should run Build Stats' python script generate_graph.py with additional flag -wiki that specifies the path to vt.wiki repository (all resutls of CI runs are stored there, so in order to graph historic data, we need path to it)

Example: python generate_graph.py -time /path/test_name_time.csv -mem /path/test_name_mem.csv -wiki /some/path/vt.wiki

This will generate:

• time execution graph for 'test_name' (including historic data)
• separate graph for each unique timer (same as before)
• memory usage (same as before)