Skip to content

Commit

Permalink
Deflake some pkgci jobs. (iree-org#19472)
Browse files Browse the repository at this point in the history
* Increase real weight test timeouts from 4 minutes to 10 minutes to
work around
https://github.com/iree-org/iree/actions/runs/12281522213/job/34271200734#step:9:1461
    
    ```
============================== slowest durations
===============================
240.00s call
SHARK-TestSuite/iree_tests/sharktank/punet/int8/test_cases.json::sdxl_unet_int8_export.mlir::gpu_rocm::real_weights
31.44s call
SHARK-TestSuite/iree_tests/sharktank/punet/fp16/test_cases.json::sdxl_unet_fp16_export.mlir::gpu_rocm::real_weights
11.22s call
SHARK-TestSuite/iree_tests/sharktank/llama/open-llama-3b-v2-f16/test_cases.json::open-llama-3b-v2-f16.mlirbc::gpu_rocm::real_weights_prefill
0.08s call
SHARK-TestSuite/iree_tests/pytorch/models/resnet50/test_cases.json::resnet50.mlirbc::gpu_rocm::real_weights
0.07s call
SHARK-TestSuite/iree_tests/pytorch/models/opt-125M/test_cases.json::opt-125M.mlirbc::gpu_rocm::real_weights
    
    (10 durations < 0.005s hidden.  Use -vv to show these durations.)
=========================== short test summary info
============================
PASSED
SHARK-TestSuite/iree_tests/sharktank/llama/open-llama-3b-v2-f16/test_cases.json::open-llama-3b-v2-f16.mlirbc::gpu_rocm::real_weights_prefill
PASSED
SHARK-TestSuite/iree_tests/sharktank/punet/fp16/test_cases.json::sdxl_unet_fp16_export.mlir::gpu_rocm::real_weights
XFAIL
SHARK-TestSuite/iree_tests/pytorch/models/opt-125M/test_cases.json::opt-125M.mlirbc::gpu_rocm::real_weights
- Expected compilation to fail (included in 'expected_compile_failures')
XFAIL
SHARK-TestSuite/iree_tests/pytorch/models/resnet50/test_cases.json::resnet50.mlirbc::gpu_rocm::real_weights
- Expected compilation to fail (included in 'expected_compile_failures')
FAILED
SHARK-TestSuite/iree_tests/sharktank/punet/int8/test_cases.json::sdxl_unet_int8_export.mlir::gpu_rocm::real_weights
- Failed: Timeout >240.0s
======= 1 failed, 2 passed, 2 deselected, 2 xfailed in 282.99s (0:04:42)
=======
    ```

* Skip flaky test_gridsample_zeros_padding op test to work around
https://github.com/iree-org/iree/actions/runs/12286576807/job/34287344921#step:8:59
    
    ```
_ IREE compile and run:
test_gridsample_zeros_padding::model.mlir::model.mlir::cpu_llvm_sync _
[gw3] linux -- Python 3.11.10
/home/runner/work/iree/iree/venv/bin/python
    Error invoking iree-run-module
    Error code: 1
    Stderr diagnostics:
    
    Stdout diagnostics:
    EXEC @test_gridsample_zeros_padding
[FAILED] result[0]: element at index 3 (2.80544E+13) does not match the
expected (0); expected that the view is equal to contents of a view of
1x1x2x4xf32
      expected:
    1x1x2x4xf32=[[[0 0 1.7 0][0 1.7 0 0]]]
      actual:
    1x1x2x4xf32=[[[0 0 1.7 2.80544E+13][2.80544E+13 1.7 0 2.80544E+13]]]
    ```

and
https://github.com/iree-org/iree/actions/runs/12285879922/job/34285283119#step:8:51
    
    ```
_ IREE compile and run:
test_gridsample_zeros_padding::model.mlir::model.mlir::cpu_llvm_sync _
[gw3] linux -- Python 3.11.11
/home/runner/work/iree/iree/venv/bin/python
    Error invoking iree-run-module
    Error code: 1
    Stderr diagnostics:
    
    Stdout diagnostics:
    EXEC @test_gridsample_zeros_padding
[FAILED] result[0]: element at index 3 (39529.7) does not match the
expected (0); expected that the view is equal to contents of a view of
1x1x2x4xf32
      expected:
    1x1x2x4xf32=[[[0 0 1.7 0][0 1.7 0 0]]]
      actual:
    1x1x2x4xf32=[[[0 0 1.7 39529.7][39529.7 1.7 0 39529.7]]]
    ```

(This test seems to be failing consistently as of
iree-org@ea9176a,
but with differing outputs, we could mark it as failing or skip)
  • Loading branch information
ScottTodd authored Dec 12, 2024
1 parent d68cd28 commit 27742f6
Show file tree
Hide file tree
Showing 2 changed files with 4 additions and 2 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/pkgci_regression_test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ jobs:
--no-skip-tests-missing-files \
--capture=no \
--log-cli-level=info \
--timeout=240 \
--timeout=600 \
--durations=0 \
--config-files=${MODELS_CONFIG_FILE_PATH}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,9 @@
"onnx/node/generated/test_group_normalization_epsilon_expanded",
"onnx/node/generated/test_group_normalization_example_expanded"
],
"skip_run_tests": [],
"skip_run_tests": [
"onnx/node/generated/test_gridsample_zeros_padding"
],
"expected_compile_failures": [
"onnx/node/generated/test_affine_grid_2d",
"onnx/node/generated/test_affine_grid_2d_align_corners",
Expand Down

0 comments on commit 27742f6

Please sign in to comment.