Anybody had luck with python 3.11 ?

[thomasegriffith]

Has anybody had luck with python 3.11 ?

I'm hoping to use it because it is supposed to be noticeably faster than 3.10, according to this doc.

I've been unsuccessful in my project. I've been able to reproduce the same error using one of the examples in rules_ros2. If I edit examples/WORKSPACE to use 3.11 instead of 3.10, I get the following:

$ bazel run //chatter:py_talker 
INFO: Analyzed target //chatter:py_talker (25 packages loaded, 4066 targets configured).
INFO: Found 1 target...
Target //chatter:py_talker up-to-date:
  bazel-bin/chatter/py_talker
INFO: Elapsed time: 23.322s, Critical Path: 14.31s
INFO: 177 processes: 37 internal, 140 linux-sandbox.
INFO: Build completed successfully, 177 total actions
INFO: Running command line: bazel-bin/chatter/py_talker
Traceback (most recent call last):
  File "/opt/ros/humble/local/lib/python3.10/dist-packages/rosidl_generator_py/import_type_support_impl.py", line 46, in import_type_support
    return importlib.import_module(module_name, package=pkg_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.cache/bazel/_bazel_user/7384e45116af376abd1c4cdcfc681cb7/external/rules_ros2_python_x86_64-unknown-linux-gnu/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1140, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'rcl_interfaces.rcl_interfaces_s__rosidl_typesupport_c'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/.cache/bazel/_bazel_user/7384e45116af376abd1c4cdcfc681cb7/execroot/com_github_mvukov_rules_ros2_examples/bazel-out/k8-fastbuild/bin/chatter/py_talker.runfiles/com_github_mvukov_rules_ros2_examples/chatter/py_talker.py", line 54, in <module>
    main()
  File "/home/user/.cache/bazel/_bazel_user/7384e45116af376abd1c4cdcfc681cb7/execroot/com_github_mvukov_rules_ros2_examples/bazel-out/k8-fastbuild/bin/chatter/py_talker.runfiles/com_github_mvukov_rules_ros2_examples/chatter/py_talker.py", line 39, in main
    minimal_publisher = MinimalPublisher()
                        ^^^^^^^^^^^^^^^^^^
  File "/home/user/.cache/bazel/_bazel_user/7384e45116af376abd1c4cdcfc681cb7/execroot/com_github_mvukov_rules_ros2_examples/bazel-out/k8-fastbuild/bin/chatter/py_talker.runfiles/com_github_mvukov_rules_ros2_examples/chatter/py_talker.py", line 22, in __init__
    super().__init__('minimal_publisher')
  File "/home/user/.cache/bazel/_bazel_user/7384e45116af376abd1c4cdcfc681cb7/execroot/com_github_mvukov_rules_ros2_examples/bazel-out/k8-fastbuild/bin/chatter/py_talker.runfiles/ros2_rclpy/rclpy/rclpy/node.py", line 199, in __init__
    self._parameter_event_publisher = self.create_publisher(
                                      ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.cache/bazel/_bazel_user/7384e45116af376abd1c4cdcfc681cb7/execroot/com_github_mvukov_rules_ros2_examples/bazel-out/k8-fastbuild/bin/chatter/py_talker.runfiles/ros2_rclpy/rclpy/rclpy/node.py", line 1290, in create_publisher
    check_is_valid_msg_type(msg_type)
  File "/home/user/.cache/bazel/_bazel_user/7384e45116af376abd1c4cdcfc681cb7/execroot/com_github_mvukov_rules_ros2_examples/bazel-out/k8-fastbuild/bin/chatter/py_talker.runfiles/ros2_rclpy/rclpy/rclpy/type_support.py", line 35, in check_is_valid_msg_type
    check_for_type_support(msg_type)
  File "/home/user/.cache/bazel/_bazel_user/7384e45116af376abd1c4cdcfc681cb7/execroot/com_github_mvukov_rules_ros2_examples/bazel-out/k8-fastbuild/bin/chatter/py_talker.runfiles/ros2_rclpy/rclpy/rclpy/type_support.py", line 29, in check_for_type_support
    msg_or_srv_type.__class__.__import_type_support__()
  File "/opt/ros/humble/local/lib/python3.10/dist-packages/rcl_interfaces/msg/_parameter_event.py", line 29, in __import_type_support__
    module = import_type_support('rcl_interfaces')
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/ros/humble/local/lib/python3.10/dist-packages/rosidl_generator_py/import_type_support_impl.py", line 48, in import_type_support
    raise UnsupportedTypeSupport(pkg_name)
rosidl_generator_py.import_type_support_impl.UnsupportedTypeSupport: Could not import 'rosidl_typesupport_c' for package 'rcl_interfaces'

[lalten]

Funny enough I could only get 3.11 to work on the bzlmod branch, not 3.10 😅
Haven't looked into why that is yet

[thomasegriffith]

Woha, that is interesting!

Fwiw I just tried out the bzlmod branch on the lalten/rules_ros2 fork and hit the same error.

[thomasegriffith]

Ok, so it turns out that if you have ros2-humble system installed and have its setup.bash sourced (like in your .bashrc or something), then you'll hit the above error when changing the python toolchain in your WORKSPACE from 3.10 to 3.11.

My guess is the PYTHONPATH is leaking into the environment at runtime (it's not actually hermetic) and is causing compatibility issues when the system-installed ros2-humble toolchain is invoked.

A workaround for now is to just remove source /opt/ros/humble/setup.bash from the .bashrc.

[ahans]

Building is hermetic (w.r.t. Python), but running is not. I have a rule for nodes that creates a bash wrapper that does an explicit unset PYTHONPATH to avoid this issue.

[mvukov]

In principle, python3.11 should be even more safe, see https://github.com/bazelbuild/rules_python/blob/main/python/private/python_bootstrap_template.txt#L486.

Another alternative is to use e.g. https://github.com/aspect-build/rules_py. This one tries to make hermetic deployments.

A yet another alternative is to make your own bootstraper for py_binary/py_test, this is possible w/ Bazel 7. https://bazel.build/reference/be/python#py_runtime.bootstrap_template -- for this to work you'd need to set up your own python toolchain -- not a lot of work IMO. With your own bootstraper you could e.g. unset the pythonpath like Alex is doing.

[thomasegriffith]

Nice, thanks for the resources!