-
Notifications
You must be signed in to change notification settings - Fork 8
Using performance test harness
Jacob Domagala edited this page Jul 2, 2021
·
1 revision
Basic usage:
#include <vt/tests/common/test_harness.h>
using namespace vt::tests::perf::common;
struct MyTest : PerfTestHarness {};
VT_PERF_TEST(MyTest, test_case_name){
// Test code
}
VT_PERF_TEST_MAIN()
Note: With the current implementation, it's limited to one test per file, meaning you can't have multiple VT_PERF_TEST
in a single test file.
- By default, the test function (created with
VT_PERF_TEST
) will be run 50 times (this value can be changed, using the runtime flag--vt_perf_num_runs
)
#include "common/test_harness.h"
using namespace vt::tests::perf::common;
std::vector<int32_t*> ptrs;
void allocateAndTouch(std::size_t num_bytes) {
auto ptr = new int32_t[num_bytes / 4];
for (std::size_t i = 0; i < num_bytes / 4; i++) {
ptr[i] = 20 * i + 23;
}
ptrs.push_back(ptr);
}
void deallocate() {
if (ptrs.size() > 0) {
auto ptr = ptrs.back();
ptrs.pop_back();
delete[] ptr;
}
}
struct MyTest : PerfTestHarness { };
VT_PERF_TEST(MyTest, test_name) {
constexpr auto num_iters = 32;
StartTimer("Allocate");
for (int i = 0; i < num_iters; i++) {
allocateAndTouch(1024 * 1024 * 64);
GetMemoryUsage();
}
StopTimer("Allocate");
StartTimer("Deallocate");
for (int i = 0; i < num_iters; i++) {
deallocate();
GetMemoryUsage();
}
StopTimer("Deallocate");
}
VT_PERF_TEST_MAIN()
mpirun -n 2 /build/vt/tests/example
or ctest -L perf_test
Graphs can be generated locally using Build Stats repository.
- Run performance test with
--vt_perf_gen_file
runtime flag, which will generate two CSV files (one for memory usage and one containing timers data) - (Optionally) Go to Build Stats directory and create some temporary folder, where the graphs will reside.
- Run Build Stats' python script
generate_graph.py
like so:python generate_graph.py -time /path/to/file/test_name_time.csv -mem /path/to/file/test_name_mem.csv
This will generate:
- time execution graph for 'test_name' (entire function created with VT_PERF_TEST)
- separate graph for each unique timer (iterative timers, i.e. 'xxx Bytes' or 'xxx Phase' will be aggregated into single graph)
- memory usage for 'test_name'
- The above was for the new tests that weren't run on CI before, or if you don't want to compare the execution time for given test with previous CI runs. Otherwise you should run Build Stats' python script
generate_graph.py
with additional flag-wiki
that specifies the path to vt.wiki repository (all resutls of CI runs are stored there, so in order to graph historic data, we need path to it)
Example:
python generate_graph.py -time /path/test_name_time.csv -mem /path/test_name_mem.csv -wiki /some/path/vt.wiki
This will generate:
- time execution graph for 'test_name' (including historic data)
- separate graph for each unique timer (same as before)
- memory usage (same as before)