You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The way that CI/CD evaluates the benchit examples is simply by running
benchit example_namy.py --example-options
However, benchit captures many exceptions at runtime and then exits "successfully". None of our examples should throw an exception, so if it happens then we'll get a false positive on the test.
Example: On a system that doesn't have docker installed, benchit examples/cli/scripts/hello_world.py will output the following, which is a false positive:
Models discovered during profiling:
hello_world.py:
pytorch_model (executed 1x)
Model Type: Pytorch (torch.nn.Module)
Class: SmallModel (<class 'hello_world.SmallModel'>)
Location: /build/source/examples/script_api/scripts/hello_world.py, line 32
Parameters: 50 (<0.1 MB)
Hash: a6d4aa6b
Status: Unknown benchit error: Docker installation not found. Please install Docker>=20.10
Traceback (most recent call last):
File "/nix/store/l7db9chq17nhpbpk1mz2a9wihaani9zk-python3.10-mlagility-3.0.2/lib/python3.10/site-packages/mlagility/analysis/analysis.py", line 122, in call_benchit
perf = benchmark_model(
File "/nix/store/l7db9chq17nhpbpk1mz2a9wihaani9zk-python3.10-mlagility-3.0.2/lib/python3.10/site-packages/mlagility/api/model_api.py", line 164, in benchmark_model
perf = cpu_model.benchmark(backend=backend)
File "/nix/store/l7db9chq17nhpbpk1mz2a9wihaani9zk-python3.10-mlagility-3.0.2/lib/python3.10/site-packages/mlagility/api/ortmodel.py", line 20, in benchmark
benchmark_results = self._execute(repetitions=repetitions, backend=backend)
File "/nix/store/l7db9chq17nhpbpk1mz2a9wihaani9zk-python3.10-mlagility-3.0.2/lib/python3.10/site-packages/mlagility/api/ortmodel.py", line 75, in _execute
devices.execute_ort_locally(
File "/nix/store/l7db9chq17nhpbpk1mz2a9wihaani9zk-python3.10-mlagility-3.0.2/lib/python3.10/site-packages/mlagility/api/devices.py", line 609, in execute_ort_locally
raise ValueError("Docker installation not found. Please install Docker>=20.10")
ValueError: Docker installation not found. Please install Docker>=20.10
pytorch_outputs: tensor([ 0.0452, -0.2537, -0.1148, 0.2036, 0.1839], grad_fn=<AddBackward0>)
Woohoo! The 'benchmark' command is complete.
...
Suggested solution: disable capturing exceptions when testing examples.
Setting priority to P2 since this hasn't caused us major headaches yet.
cc @danielholanda @ramkrishna2910
The text was updated successfully, but these errors were encountered:
The way that CI/CD evaluates the benchit examples is simply by running
However, benchit captures many exceptions at runtime and then exits "successfully". None of our examples should throw an exception, so if it happens then we'll get a false positive on the test.
Example: On a system that doesn't have
docker
installed,benchit examples/cli/scripts/hello_world.py
will output the following, which is a false positive:The text was updated successfully, but these errors were encountered: