Pilz integration tests have started failing sporadically on Rolling due to exit code -11

[sea-bass]

Description

There have lately been pretty frequent (~50% or so) flakes with some exit codes in Pilz integration tests:
https://github.com/moveit/moveit2/actions/runs/12589290424/job/35088819135?pr=3187#step:11:8621

    ======================================================================
    FAIL: test_exit_codes (pilz_industrial_motion_planner.TestOutcome.test_exit_codes)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/home/runner/work/moveit2/moveit2/.work/target_ws/src/moveit2/moveit_planners/pilz_industrial_motion_planner/test/unit_tests/launch/unittest_planning_context_loaders.test.py", line 53, in test_exit_codes
        launch_testing.asserts.assertExitCodes(
      File "/opt/ros/rolling/lib/python3.12/site-packages/launch_testing/asserts/assert_exit_codes.py", line 62, in assertExitCodes
        assert info.returncode in allowable_exit_codes, 'Proc {} exited with code {}'.format(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    AssertionError: Proc unittest_planning_context_loaders-1 exited with code -11

This seems to be specifically happening in the rolling CI jobs, but it's usually cleared away after 1 or 2 retries.

Creating this issue to keep track.

ROS Distro

Rolling

OS and version

Ubuntu 24.04

Source or binary build?

Source

If binary, which release version?

No response

If source, which branch?

No response

Which RMW are you using?

None

Steps to Reproduce

Run CI

Expected behavior

CI is green

Actual behavior

CI is not always green

Backtrace or Console output

No response

[GayarUsmanov]

Is there a way to reproduce the issue locally on my machine?

[sea-bass]

Is there a way to reproduce the issue locally on my machine?

In these integration tests I often rely on the stress command to reproduce locally: https://www.geeksforgeeks.org/linux-stress-command-with-examples/

However, issues pertaining to exit codes can be tricky and sometimes can't be reproduced. Worth a try, though.

[GayarUsmanov]

Sorry, I should've been clearer. I meant to ask how do you run the integration test on local machine?
Once I know how to do that I'll run it under a memory checker, e.g. Valgrind.

[sea-bass]

Aha. You can do something like this to the invocation of colcon test

https://docs.ros.org/en/humble/How-To-Guides/Getting-Backtraces-in-ROS-2.html#debugging-tests-with-gdb

Edited for better doc link. There's a section in that doc too for how to prefix a node in a launch file with gdb (or other tools)

[MarcoMagriDev]

At the current stage I am not able to reproduce this issue locally.

---

For those of you are interested in reproducing this locally i suggest to use this command:

colcon build --cmake-clean-cache --mixin debug --packages-select pilz_industrial_motion_planner pilz_industrial_motion_planner_testutils && \
source install/setup.bash && \
colcon test --packages-select pilz_industrial_motion_planner --ctest-args --output-on-failure --interactive-debugger gdb

then using colcon test-result --verbose you can see the test outcome with gdb prints.