This benchmark suite provides a systemic way to measure the performance of operators for a wide range of inputs. The generated benchmark data fully characterized the performance of an operator in terms of execution time and the efficiency of the PyTorch/Caffe2 frameworks used.
Key Features:
1. Language used: Python
2. Supported Frameworks: PyTorch and Caffe2
3. Supported PyTorch mode: eager and JIT
4. Input shapes: user-defined shapes, randomly generated shapes
The instruction below installs a cpp_extension for PyTorch and it is required to run the benchmark suite.
$ cd pt_extension
$ python setup.py install
Run torch.add benchmark:
$ cd pytorch/benchmarks/operator_benchmark
$ python -m pt.add_test --omp-num-threads 1 --mkl-num-threads 1
Note: we set the number of OpenMP and MKL threads both to 1. If you want to benchmark operators with multithreading (intra-op parallelism), use the --omp-num-threads and --mkl-num-threads flags.
List all the supported tests:
$ python -m pt.add_test --list-tests
Filter and run a test (use add_M8_N16_K32 as an example):
$ python -m pt.add_test --test-name add_K32_M8_N1
--omp-num-threads 1 --mkl-num-threads 1
Run all the supported benchmarks:
$ python -m benchmark_all_test
The following example shows the code to support torch.add with 27 different tests. In the subpages of this wiki, we'll step through the complete flow of adding PyTorch and Caffe2 operators to the benchmark suite. Existing benchmarks for operators are in pt and c2 directories and we highly recommend putting your new operators in those locations.
add_short_configs = op_bench.cross_product_configs(
M=[8, 64, 128],
N=range(2, 10, 3),
K=[2 ** x for x in range(0, 3)],
tags=["short"]
)
class AddBenchmark(op_bench.TorchBenchmarkBase):
def init(self, M, N, K, device):
self.inputs = {
"input_one": torch.rand(M, N, K, device=device, requires_grad=self.auto_set()),
"input_two": torch.rand(M, N, K, device=device, requires_grad=self.auto_set())
}
self.set_module_name("add")
def forward(self, input_one, input_two):
return torch.add(input_one, input_two)
op_bench.generate_pt_test(add_short_configs, AddBenchmark)
The output is intended to be a human readable format. Here is an example output for torch.add:
# ----------------------------------------
# PyTorch/Caffe2 Operator Micro-benchmarks
# ----------------------------------------
# Tag : short
# Benchmarking PyTorch: add
# Mode: Eager
# Name: add_M8_N16_K32
# Input: M: 8, N: 16, K: 32
Forward Execution Time (us) : 6.651
# Benchmarking PyTorch: add
# Mode: Eager
# Name: add_M16_N16_K64
# Input: M: 16, N: 16, K: 64
Forward Execution Time (us) : 11.976
# Benchmarking PyTorch: add
# Mode: Eager
# Name: add_M64_N64_K128
# Input: M: 64, N: 64, K: 128
Forward Execution Time (us) : 222.370
At a high level, the output includes the execution time of torch.add with three different inputs. Let's look at each line in detail:
1. Tag: short tags a group of inputs. For each operator, you could be interested in a large number of inputs, but you may not always want to run all the inputs. Tag allows you to only run some of the inputs. Most of the inputs to operators being supported in the benchmark are grouped using two tags. One group is tagged with short which stores some commonly used shapes. The other group is tagged with long which stores many random inputs to have better coverage compared with short.
2. Benchmarking PyTorch: Add shows name of the operator being benchmarked.
3. Mode: Eager shows that PyTorch eager mode is here.
4. Name: add_M8_N16_K32 is the name of the test and it can be used to filter tests.
5. Input: M: 8, N: 16, K: 32 shows inputs to the operator.
6. Forward Execution Time (us) : 6.651 reports the execution time of an operator in microseconds.
You can control all the aspects of the benchmark suite through the command-line. Please find details of those arguments by running the following command or look into benchmark_runner.py.
$ python benchmark_runner.py --help
Run all the supported benchmarks:
$ python -m benchmark_all_test --omp-num-threads 1 --mkl-num-threads 1
List all the supported operators:
$ python -m benchmark_all_test --list-ops
List all the supported tests:
$ python -m benchmark_all_test --list-tests
Filter and run an operator (use add as an example):
$ python -m benchmark_all_test --operators add --omp-num-threads 1 --mkl-num-threads 1
Note: this filter is based on the operator name rather than the file name.
Run torch.add benchmark with tag 'long':
$ python -m pt.add_test --tag-filter long
In the previous sections, we gave several examples to show how to run the already available operators in the benchmark suite. In the following sections, we'll step through the complete flow of adding PyTorch and Caffe2 operators to the benchmark suite. Existing benchmarks for operators are in pt and c2 directories and we highly recommend putting your new operators in those directories as well.
Let's say you want to measure the execution time of the following operator:
C = torch.add(A, B) # Shape of A and B is [M, N, K]
The code below shows how to add it to the benchmark suite. Let's go over the example line by line.
import operator_benchmark as op_bench
import torch
add_long_configs = op_bench.cross_product_configs(
M=[8, 64, 128],
N=range(2, 10, 3),
K=[2 ** x for x in range(0, 3)],
tags=["long"]
)
add_short_configs = op_bench.config_list(
attr_names=["M", "N", "K"],
attrs=[
[8, 16, 32],
[16, 16, 64],
[64, 64, 128],
],
tags=["short"],
)
class AddBenchmark(op_bench.TorchBenchmarkBase):
def init(self, M, N, K, device):
self.inputs = {
"input_one": torch.rand(M, N, K, device=device, requires_grad=self.auto_set()),
"input_two": torch.rand(M, N, K, device=device, requires_grad=self.auto_set())
}
self.set_module_name("add")
def forward(self, input_one, input_two):
return torch.add(input_one, input_two)
op_bench.generate_pt_test(add_long_configs + add_short_configs, AddBenchmark)
if __name__ == "__main__":
op_bench.benchmark_runner.main()
For the torch.add operator, we would like to make sure it delivers good performance with input tensors which are of small, medium and large sizes. We have introduced two helper functions for users to easily generate a combination of inputs.
# Generate list configurations that will be used for benchmark experiments
add_long_configs = op_bench.cross_product_configs(
M=[8, 64, 128],
N=range(2, 10, 3),
K=[2 ** x for x in range(0, 3)],
tags=["long"]
)
add_short_configs = op_bench.config_list(
attr_names=["M", "N", "K"],
attrs=[
[8, 16, 32],
[16, 16, 64],
[64, 64, 128],
],
tags=["short"],
)
Let's look at it in detail:
1. op_bench.config_list is a helper function which specifies a list of inputs to operators. It takes three parameters which are attrs_names, attrs, and tags, all of them are python lists. attr_names stores the names of the inputs. attrs stores the real value of each input. In this example, three different inputs will be returned which are: M=8, N=16, K=32; M=16, N=16, K=64; M=64, N=64, K=128.
2. op_bench.cross_product_configs is another helper function to generate a cartesian product of the inputs. Each input is specified in a python list. In this example, the helper method will return a combination of 27 (len(M) * len(N) * len(K)) inputs.
After inputs are provided, we now look at adding the computation of an operator. Adding a new operator requires implementing a new TorchBenchmarkBase subclass. Every new class is required to implement 2 methods:
initis used to create tensors based on the inputs we provided before. In this example, the parameters toinitareM, N, and Kwhich have been specified in the input configuration.initalso packed all the needed inputs together into a dictionaryself.inputswhich will be provided toforwardas arguments for running the benchmark.forwardincludes the operator to be tested and the computation based on the created tensors ininit. Apart fromself, the order of the arguments must match the entries specified inself.inputs.
The example below shows the code for torch.add:
# Given one set of M, N, K, the init method creates input tensors based on
# that. The forward method does torch.add calculation on those input tensors.
class AddBenchmark(op_bench.TorchBenchmarkBase):
def init(self, M, N, K, device):
# this is the method where you need to create tensors
# M, N, and K can be in different order, but they must match with
# names in the configs.
self.inputs = {
"input_one": torch.rand(M, N, K, device=device, requires_grad=self.auto_set()),
"input_two": torch.rand(M, N, K, device=device, requires_grad=self.auto_set())
}
self.set_module_name("add")
def forward(self, input_one, input_two):
# this is the method to have operator and do computation
return torch.add(input_one, input_two)
After we have inputs and the benchmark class, it's time to register them with our benchmark suite. Here is how it looks like:
op_bench.generate_pt_test(add_long_configs + add_short_configs, AddBenchmark)
generate_pt_test takes two parameters which are inputs configs and the benchmark class.
To run the benchmark, we use the main method in benchmark_runner module.
if __name__ == "__main__":
op_bench.benchmark_runner.main()
That's it. You just added a new operator to the benchmark suite!
The steps to add a new Caffe2 operator is the same as that for a PyTorch operator. The code below shows how to add Caffe2 Add operator:
import operator_benchmark as op_bench
from caffe2.python import core
add_long_configs = op_bench.cross_product_configs(
M=[8, 64, 128],
N=range(2, 10, 3),
K=[2 ** x for x in range(0, 3)],
tags=["long"]
)
add_short_configs = op_bench.config_list(
attrs=[
[8, 16, 32],
[16, 16, 64],
[64, 64, 128],
],
attr_names=["M", "N", "K"],
tags=["short"],
)
class AddBenchmark(op_bench.Caffe2BenchmarkBase):
def init(self, M, N, K):
self.input_one = self.tensor(M, N, K)
self.input_two = self.tensor(M, N, K)
self.output = self.tensor(M, N, K)
self.set_module_name("add")
def forward(self):
op = core.CreateOperator(
"Add", [self.input_one, self.input_two], self.output, **self.args
)
return op
op_bench.generate_c2_test(add_long_configs + add_short_configs, AddBenchmark)
if __name__ == "__main__":
op_bench.benchmark_runner.main()
There are two things worth mentioning in this code:
self.tensoris a helper function which takes shapes and returns a Caffe2 blob. It is designed to make the tensor creation step easier compared to the standard Caffe2 way.generate_c2_testis used to register Caffe2 tests with the benchmark.
In the previous sections, we introduced the steps required to add a single operator to the benchmark suite. There are scenarios where you want to extend the benchmark suite with a list of operators which can share the same inputs. For example, to benchmark abs and acos operators, you can use the same set of inputs for both.
Let's say we want to benchmark the following operators separately:
C = torch.abs(A) # Shape of A [M, N]
C = torch.acos(A) # Shape of A [M, N]
The following code shows how to do that:
import operator_benchmark as op_bench
import torch
unary_ops_configs = op_bench.config_list(
attrs=[
[128, 128],
[256, 256],
[1024, 1024],
],
attr_names=["M", "N"],
tags=["short"]
)
unary_ops_list = op_bench.op_list(
attr_names=["op_name", "op_func"],
attrs=[
["abs", torch.abs],
["acos", torch.acos],
],
)
class UnaryOpBenchmark(op_bench.TorchBenchmarkBase):
def init(self, M, N, device, op_func):
self.inputs = {
"input": torch.rand(M, N, device=device)
}
self.op_func = op_func
def forward(self, input):
return self.op_func(input)
op_bench.generate_pt_tests_from_op_list(unary_ops_list, unary_ops_configs, UnaryOpBenchmark)
if __name__ == "__main__":
op_bench.benchmark_runner.main()
The inputs to those operators are specified using the same method we went over before. So we just skip it here.
To add a list of operators to the benchmark suite, we introduce the op_bench.op_list method which takes two parameters:
attrsstores the name of the operator and the method to do the real calculation.attr_namesstores the names of values in attrs.
The example below shows the code to add torch.abs and torch.acos :
unary_ops_list = op_bench.op_list(
attr_names=["op_name", "op_func"],
attrs=[
["abs", torch.abs],
["acos", torch.acos],
],
)
In this example, both operators share the same input so we only need to implement one TorchBenchmarkBase subclass. Every new subclass is required to implement 3 methods:
initis used to create tensors and set the operator name and function. In this example, the parameters toinitareM,N, andop_funcwhich have been specified in the configurations.forwardincludes the operator to be tested and the computation based on the created tensors ininit. Apart fromself, the order of the arguments must match the entries specified inself.inputs. Here is the code forabsandacos:
class UnaryOpBenchmark(op_bench.TorchBenchmarkBase):
def init(self, M, N, device, op_func):
# The M and N match with the attr_names in the input configuration
# The op_func matches with the attr_name in the ops configuration
self.inputs = {
"input": torch.rand(M, N, device=device)
}
self.op_func = op_func
def forward(self, input):
return self.op_func(input)
To register multiple operators, we introduced the generate_pt_tests_from_op_list function which takes three parameters. First, the list of operators. Second,the configs. Third, the benchmark class.
Here is an example:
op_bench.generate_pt_tests_from_op_list(unary_ops_list, unary_ops_configs, UnaryOpBenchmark)
In this section, we go over the steps to benchmark the backward path of operators.
To measure the performance of an operator in its backward path, there are only two changes needed in addition to the steps we covered for the forward path:
1. Specify requires_grad=True when creating the tensor. This is a standard PyTorch way of enabling backward path.
2. Use generate_pt_gradient_test to register the tests.
The example below shows the relevant code for that:
self.input_one = torch.rand(M, N, K, requires_grad=True)
generate_pt_gradient_test(long_configs + short_configs, TorchAddBenchmark)
To add Caffe2 gradient ops, we need to implement a new backward method in the benchmark class:
class AddBenchmark(op_bench.Caffe2BenchmarkBase):
def init(self, M, N, K):
self.input_one = self.tensor(M, N, K)
self.input_two = self.tensor(M, N, K)
self.input_one_grad = self.tensor(M, N, K)
self.input_two_grad = self.tensor(M, N, K)
self.output = self.tensor(M, N, K)
self.set_module_name("add")
def forward(self):
op = core.CreateOperator(
"Add", [self.input_one, self.input_two], self.output, **self.args
)
return op
def backward(self):
grad_op = core.CreateOperator(
"AddGradient",
[self.output, self.input_one, self.input_two],
[self.input_one_grad, self.input_two_grad], **self.args
)
return grad_op
op_bench.generate_c2_gradient_test(long_configs + short_configs,AddBenchmark)
After the class is implemented, we need to register the tests with generate_c2_gradient_test function.
This concludes the overview of the operator benchmark suite.