NO TESTS RAN failure on Noble for an ament_python package

[stonier]

Hi all,

I'm seeing a difference in behavior between Iron and Jammy for ROS PR jobs using ament_python.

• Package: py_trees
• PR with CI Jobs: PR 450
• ✔️ Iron PR Job - Console Log - Succeeded, despite Ran 0 tests
• ❌ Rolling PR Job - Console Log - Failed on NO TESTS RAN

Tests are covered by github actions, so I've been leaving them out on the ROS PR Job (which serves mainly to smoke test deb builds). Has there been a toggle in colcon or the underlying tools in Noble that now require them?

I'm actually surprised it's doing anything here at all since there is no callout to pytest in [setup.py].

[cottsay]

Hi there.

There are two strategies that colcon uses to test Python packages. The default is to directly invoke the unittest module. If colcon detects that a package has a test dependency on pytest, it will use pytest instead.

So your package uses unittest. It appears that there was a change in behavior in Python 3.12: python/cpython#102051

I'm open to suggestions here, but I'm guessing that from your perspective you'd like the pre-3.12 behavior of "no error" when there were no tests. Is that correct?

[stonier]

there was a change in behavior in Python 3.12: python/cpython#102051

Oh, good digging. Thanks. I was barking up the colcon/pytest tree.

So your package uses unittest.

Is it defaulting to that? I don't specify it anywhere.

I'm open to suggestions here, but I'm guessing that from your perspective you'd like the pre-3.12 behavior of "no error" when there were no tests. Is that correct?

I'm not sure yet. It would be reasonable to expect that if no test configuration is provided, no tests are run. However, this is all against my usual bias for requiring tests everywhere. Might be worth just activating the pytests that are there and see if the extra configuration is tolerable. Let me explore for a bit.

[cottsay]

Is it defaulting to that? I don't specify it anywhere.

That's correct. That was chosen as the default likely because it's built into Python's standard library and will reliably be available.

For a low-effort fix, it might be good to add a super simple unittest-compatible import check. Very high bang-for-your-buck minimal test.

import unittest


class ImportTest(unittest.TestCase):

    def test_import(self):
        import py_trees  # noqa: F401

[stonier]

Aye, was thinking along those lines too. Thanks!

[texhnolyze]

I came across this issue too and while upgrading our code to ros jazzy (and ubuntu 24.04 / python 3.12). While adding an empty test case is a quick fix, I don't think that when running tests for a whole code base of a multitude of packages and there are some, that do not have any test cases it should fail the whole test run.

Or at least I think, that there should be an option e.g. by passing --allow-empty-test-suites or something similar, which would ignore unittest exits with error code "5".
If you agree, I would be happy to create a PR.

[cottsay]

This is a little tricky from my perspective.

• Nothing changed in colcon, this was a change in the Python standard library
• The change was intentional and attempts to surface common misconfiguration

That said, much of colcon's value comes from providing a cohesive experience across various build (and test) systems, and it appears that this may be a deviation for unittest when compared to ctest and pytest which are more popular among colcon users.

Implicitly converting exit code 5 to 0 for unittest invocations if sys.version_info >= (3, 12) seems reasonable to me. If I recall correctly, the user is already presented with stderr text stating that no tests were run, so as long as that continues I think we can revert to the previous (and more popular) behavior of treating 'no tests to run' as a successful invocation of a package's tests.