Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI/CD testing false positives on certain benchit example failures #268

Open
jeremyfowers opened this issue May 2, 2023 · 0 comments
Open
Labels
bug Something isn't working new Work that hasn't started yet p2 not a priority testing Relating to testing

Comments

@jeremyfowers
Copy link
Contributor

The way that CI/CD evaluates the benchit examples is simply by running

benchit example_namy.py --example-options

However, benchit captures many exceptions at runtime and then exits "successfully". None of our examples should throw an exception, so if it happens then we'll get a false positive on the test.

Example: On a system that doesn't have docker installed, benchit examples/cli/scripts/hello_world.py will output the following, which is a false positive:

Models discovered during profiling:
hello_world.py:
        pytorch_model (executed 1x)
                Model Type:     Pytorch (torch.nn.Module)
                Class:          SmallModel (<class 'hello_world.SmallModel'>)
                Location:       /build/source/examples/script_api/scripts/hello_world.py, line 32
                Parameters:     50 (<0.1 MB)
                Hash:           a6d4aa6b
                Status:         Unknown benchit error: Docker installation not found. Please install Docker>=20.10
                Traceback (most recent call last):
                  File "/nix/store/l7db9chq17nhpbpk1mz2a9wihaani9zk-python3.10-mlagility-3.0.2/lib/python3.10/site-packages/mlagility/analysis/analysis.py", line 122, in call_benchit
                    perf = benchmark_model(
                  File "/nix/store/l7db9chq17nhpbpk1mz2a9wihaani9zk-python3.10-mlagility-3.0.2/lib/python3.10/site-packages/mlagility/api/model_api.py", line 164, in benchmark_model
                    perf = cpu_model.benchmark(backend=backend)
                  File "/nix/store/l7db9chq17nhpbpk1mz2a9wihaani9zk-python3.10-mlagility-3.0.2/lib/python3.10/site-packages/mlagility/api/ortmodel.py", line 20, in benchmark
                    benchmark_results = self._execute(repetitions=repetitions, backend=backend)
                  File "/nix/store/l7db9chq17nhpbpk1mz2a9wihaani9zk-python3.10-mlagility-3.0.2/lib/python3.10/site-packages/mlagility/api/ortmodel.py", line 75, in _execute
                    devices.execute_ort_locally(
                  File "/nix/store/l7db9chq17nhpbpk1mz2a9wihaani9zk-python3.10-mlagility-3.0.2/lib/python3.10/site-packages/mlagility/api/devices.py", line 609, in execute_ort_locally
                    raise ValueError("Docker installation not found. Please install Docker>=20.10")
                ValueError: Docker installation not found. Please install Docker>=20.10
pytorch_outputs: tensor([ 0.0452, -0.2537, -0.1148,  0.2036,  0.1839], grad_fn=<AddBackward0>)
Woohoo! The 'benchmark' command is complete.
...

Suggested solution: disable capturing exceptions when testing examples.

Setting priority to P2 since this hasn't caused us major headaches yet.

cc @danielholanda @ramkrishna2910 
@jeremyfowers jeremyfowers added bug Something isn't working new Work that hasn't started yet p2 not a priority testing Relating to testing labels May 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working new Work that hasn't started yet p2 not a priority testing Relating to testing
Projects
None yet
Development

No branches or pull requests

1 participant