catkin_test_results tool chain not providing useful info that leads to the cause of test failure (e.g. RLException)

[130s]

I'm not sure where to report this. Since catkin_tools is the most donwstream in the toolchain I'm interacting with, I'm filing here.

System Info

• Operating System: uname -a
• Python Version: python --version
• Version of catkin_tools: catkin --version OR Git revision
• ROS Distro: echo $ROS_DISTRO

  # uname -a
  Linux 6c5f8e6a351b 5.4.0-60-generic #67-Ubuntu SMP Tue Jan 5 18:31:36 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
  # python --version
  Python 2.7.12
  # catkin --version
  catkin_tools 0.4.5 (C) 2014-2021 Open Source Robotics Foundation
  catkin_tools is released under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0)
  ---
  Using Python 2.7.12 (default, Oct  5 2020, 13:56:01) [GCC 5.4.0 20160609]
  # echo $ROS_DISTRO
  kinetic
  

Build / Run Issue

• Works with catkin_make
• Works with catkin_make_isolated --merge
• Works with catkin build
• Works with catkin build -p1

root@6c5f8e6a351b:/cws# catkin_test_results 
build/b_intest/test_results/b_intest/MISSING-rostest-ah_10_9_pick__tcp_port_11010__suppress_logging_true.xml: 1 tests, 0 errors, 1 failures, 0 skipped
build/b_intest/test_results/b_intest/MISSING-rostest-ah_10_9_place__tcp_port_11011__suppress_logging_true.xml: 1 tests, 0 errors, 1 failures, 0 skipped
build/b_intest/test_results/b_intest/MISSING-rostest-ah_12_15_pick_mixed_smartpak__tcp_port_11013__suppress_logging_true.xml: 1 tests, 0 errors, 1 failures, 0 skipped
build/b_intest/test_results/b_intest/MISSING-rostest-ah_2_26_pick_fat_envelopes__suppress_logging_true.xml: 1 tests, 0 errors, 1 failures, 0 skipped
Summary: 4 tests, 0 errors, 4 failures, 0 skipped
root@6c5f8e6a351b:/cws# more build/b_intest/test_results/b_intest/MISSING-rostest-ah_10_9_pick__tcp_port_11010__suppress_logging_true.xml
<?xml version="1.0" encoding="UTF-8"?>
<testsuite tests="1" failures="1" time="1" errors="0" name="rostest-ah_10_9_pick__tcp_port_11010__suppress_logging_true.xml">
  <testcase name="test_ran" status="run" time="1" classname="Results">
    <failure message="Unable to find test results for rostest-ah_10_9_pick__tcp_port_11010__suppress_logging_true.xml, test did not run.
Expected results in /cws/build/b_intest/test_results/b_intest/rostest-ah_10_9_pick__tcp_port_11010__suppress_logging_true.xml" type=""/>
  </testcase>
</testsuite>

# more build/b_intest/test_results/b_intest/MISSING-rostest-ah_10_9_pick__tcp_port_11010__suppress_logging_true.xml
<?xml version="1.0" encoding="UTF-8"?>
<testsuite tests="1" failures="1" time="1" errors="0" name="rostest-ah_10_9_pick__tcp_port_11010__suppress_logging_true.xml">
  <testcase name="test_ran" status="run" time="1" classname="Results">
    <failure message="Unable to find test results for rostest-ah_10_9_pick__tcp_port_11010__suppress_logging_true.xml, test did not run.
Expected results in /cws/build/b_intest/test_results/b_intest/rostest-ah_10_9_pick__tcp_port_11010__suppress_logging_true.xml" type=""/>
  </testcase>
</testsuite>

# more /cws/build/b_intest/test_results/b_intest/rostest-ah_10_9_pick__tcp_port_11010__suppress_logging_true.xml
more: stat of /cws/build/b_intest/test_results/b_intest/rostest-ah_10_9_pick__tcp_port_11010__suppress_logging_true.xml failed: No such file or directory

Expected Behavior

catkin_test_results shows info that allows the user to reach the clue why the test failed (e.g. errors printed in logs/b_intest/build.make.log).

In the example above, I finally found a clue in ./logs/b_intest/build.make.log that shows following errors, which helped me to debug BUT I didn't see in any files in the test result chain. And it is by no means clear that the build.make.log file contains useful error.

• rostest failed with roslaunch.core.RLException: Invalid <arg> tag: b_data.
• rosparam failing to load robot_description from xacro that was failing due to invalud xml format.

Actual Behavior

catkin_test_results does show test result file (e.g. MISSING-rostest-ah_10_9_pick__tcp_port_11010__suppress_logging_true.xml in the example above), which doesn't contain meaningful info nor link to files that contain such info).