Revert "Run manual jobs with sequential executor by default"

[cottsay]

This reverts commit 4c2c576.

No CI to run - we already run the nightlies with the parallel executor.

Diff:

$ ./create_jenkins_job.py
Connecting to Jenkins 'https://ci.ros2.org'
Connected to Jenkins version '2.319.2'
Updating job 'ci_linux' (dry run)
    <<<
    --- remote config
    +++ new config
    @@ -178 +178 @@
    -          <defaultValue>--event-handlers console_cohesion+ --retest-until-pass 2 --ctest-args -LE xfail --pytest-args -m "not xfail" --executor sequential</defaultValue>
    +          <defaultValue>--event-handlers console_cohesion+ --retest-until-pass 2 --ctest-args -LE xfail --pytest-args -m "not xfail"</defaultValue>
    >>>
Skipped 'test_ci_linux' because the config is the same (dry run)
Skipped 'ci_packaging_linux' because the config is the same (dry run)
Skipped 'test_packaging_linux' because the config is the same (dry run)
Skipped 'packaging_linux' because the config is the same (dry run)
Skipped 'nightly_linux_debug' because the config is the same (dry run)
Skipped 'nightly_linux_address_sanitizer' because the config is the same (dry run)
Skipped 'nightly_linux_clang_libcxx' because the config is the same (dry run)
Updating job 'ci_linux_clang_libcxx' (dry run)
    <<<
    --- remote config
    +++ new config
    @@ -178 +178 @@
    -          <defaultValue>--event-handlers console_cohesion+ --ctest-args -LE xfail --pytest-args -m "not xfail" --packages-select rcutils --executor sequential</defaultValue>
    +          <defaultValue>--event-handlers console_cohesion+ --ctest-args -LE xfail --pytest-args -m "not xfail" --packages-select rcutils</defaultValue>
    >>>
Skipped 'nightly_linux_thread_sanitizer' because the config is the same (dry run)
Updating job 'ci_linux_coverage' (dry run)
    <<<
    --- remote config
    +++ new config
    @@ -178 +178 @@
    -          <defaultValue>--event-handlers console_cohesion+ --retest-until-pass 2 --ctest-args -LE xfail --pytest-args -m "not xfail" --packages-skip qt_gui_cpp --packages-skip-by-dep qt_gui_cpp --packages-up-to action_msgs ament_index_cpp builtin_interfaces class_loader composition_interfaces console_bridge_vendor diagnostic_msgs fastcdr fastrtps foonathan_memory_vendor geometry_msgs libstatistics_collector libyaml_vendor lifecycle_msgs nav_msgs rcl rcl_action rcl_interfaces rcl_lifecycle rcl_logging_spdlog rcl_yaml_param_parser rclcpp rclcpp_action rclcpp_components rclcpp_lifecycle rcpputils rcutils rmw rmw_dds_common rmw_fastrtps_cpp rmw_fastrtps_shared_cpp rmw_implementation rosgraph_msgs rosidl_default_runtime rosidl_runtime_c rosidl_runtime_cpp rosidl_typesupport_c rosidl_typesupport_cpp rosidl_typesupport_fastrtps_c rosidl_typesupport_fastrtps_cpp rosidl_typesupport_interface rosidl_typesupport_introspection_c rosidl_typesupport_introspection_cpp spdlog_vendor statistics_msgs std_msgs std_srvs tracetools trajectory_msgs unique_identifier_msgs visualization_msgs interactive_markers launch_testing_ros message_filters ros2action ros2component ros2doctor ros2interface ros2lifecycle ros2lifecycle_test_fixtures ros2param ros2topic rosbag2_compression rosbag2_cpp rosbag2_storage rosbag2_storage_default_plugins rosbag2_test_common rosbag2_tests rosbag2_transport rosidl_generator_c rosidl_generator_cpp rosidl_generator_py rosidl_runtime_py rosidl_typesupport_introspection_tests test_cli test_cli_remapping test_communication test_launch_ros test_msgs test_quality_of_service test_rclcpp test_security test_tf2 test_tracetools tf2 tf2_bullet tf2_eigen tf2_geometry_msgs tf2_kdl tf2_msgs tf2_py tf2_ros tf2_sensor_msgs --executor sequential</defaultValue>
    +          <defaultValue>--event-handlers console_cohesion+ --retest-until-pass 2 --ctest-args -LE xfail --pytest-args -m "not xfail" --packages-skip qt_gui_cpp --packages-skip-by-dep qt_gui_cpp --packages-up-to action_msgs ament_index_cpp builtin_interfaces class_loader composition_interfaces console_bridge_vendor diagnostic_msgs fastcdr fastrtps foonathan_memory_vendor geometry_msgs libstatistics_collector libyaml_vendor lifecycle_msgs nav_msgs rcl rcl_action rcl_interfaces rcl_lifecycle rcl_logging_spdlog rcl_yaml_param_parser rclcpp rclcpp_action rclcpp_components rclcpp_lifecycle rcpputils rcutils rmw rmw_dds_common rmw_fastrtps_cpp rmw_fastrtps_shared_cpp rmw_implementation rosgraph_msgs rosidl_default_runtime rosidl_runtime_c rosidl_runtime_cpp rosidl_typesupport_c rosidl_typesupport_cpp rosidl_typesupport_fastrtps_c rosidl_typesupport_fastrtps_cpp rosidl_typesupport_interface rosidl_typesupport_introspection_c rosidl_typesupport_introspection_cpp spdlog_vendor statistics_msgs std_msgs std_srvs tracetools trajectory_msgs unique_identifier_msgs visualization_msgs interactive_markers launch_testing_ros message_filters ros2action ros2component ros2doctor ros2interface ros2lifecycle ros2lifecycle_test_fixtures ros2param ros2topic rosbag2_compression rosbag2_cpp rosbag2_storage rosbag2_storage_default_plugins rosbag2_test_common rosbag2_tests rosbag2_transport rosidl_generator_c rosidl_generator_cpp rosidl_generator_py rosidl_runtime_py rosidl_typesupport_introspection_tests test_cli test_cli_remapping test_communication test_launch_ros test_msgs test_quality_of_service test_rclcpp test_security test_tf2 test_tracetools tf2 tf2_bullet tf2_eigen tf2_geometry_msgs tf2_kdl tf2_msgs tf2_py tf2_ros tf2_sensor_msgs</defaultValue>
    >>>
Skipped 'test_linux_coverage' because the config is the same (dry run)
Skipped 'nightly_linux_coverage' because the config is the same (dry run)
Skipped 'nightly_linux_humble_coverage' because the config is the same (dry run)
Skipped 'nightly_linux_iron_coverage' because the config is the same (dry run)
Skipped 'nightly_linux_jazzy_coverage' because the config is the same (dry run)
Skipped 'nightly_linux_release' because the config is the same (dry run)
Skipped 'nightly_linux_repeated' because the config is the same (dry run)
Skipped 'nightly_linux_xfail' because the config is the same (dry run)
Updating job 'ci_linux-aarch64' (dry run)
    <<<
    --- remote config
    +++ new config
    @@ -178 +178 @@
    -          <defaultValue>--event-handlers console_cohesion+ --retest-until-pass 2 --ctest-args -LE "(mimick|xfail)" --pytest-args -m "not xfail" --executor sequential</defaultValue>
    +          <defaultValue>--event-handlers console_cohesion+ --retest-until-pass 2 --ctest-args -LE "(mimick|xfail)" --pytest-args -m "not xfail"</defaultValue>
    >>>
Skipped 'test_ci_linux-aarch64' because the config is the same (dry run)
Skipped 'ci_packaging_linux-aarch64' because the config is the same (dry run)
Skipped 'test_packaging_linux-aarch64' because the config is the same (dry run)
Skipped 'packaging_linux-aarch64' because the config is the same (dry run)
Skipped 'nightly_linux-aarch64_debug' because the config is the same (dry run)
Skipped 'nightly_linux-aarch64_release' because the config is the same (dry run)
Skipped 'nightly_linux-aarch64_repeated' because the config is the same (dry run)
Skipped 'nightly_linux-aarch64_xfail' because the config is the same (dry run)
Updating job 'ci_linux-rhel' (dry run)
    <<<
    --- remote config
    +++ new config
    @@ -178 +178 @@
    -          <defaultValue>--event-handlers console_cohesion+ --retest-until-pass 2 --ctest-args -LE xfail --pytest-args -m "not xfail" --executor sequential</defaultValue>
    +          <defaultValue>--event-handlers console_cohesion+ --retest-until-pass 2 --ctest-args -LE xfail --pytest-args -m "not xfail"</defaultValue>
    >>>
Skipped 'test_ci_linux-rhel' because the config is the same (dry run)
Skipped 'ci_packaging_linux-rhel' because the config is the same (dry run)
Skipped 'test_packaging_linux-rhel' because the config is the same (dry run)
Skipped 'packaging_linux-rhel' because the config is the same (dry run)
Skipped 'nightly_linux-rhel_debug' because the config is the same (dry run)
Skipped 'nightly_linux-rhel_release' because the config is the same (dry run)
Skipped 'nightly_linux-rhel_repeated' because the config is the same (dry run)
Skipped 'nightly_linux-rhel_xfail' because the config is the same (dry run)
Updating job 'ci_windows' (dry run)
    <<<
    --- remote config
    +++ new config
    @@ -188 +188 @@
    -          <defaultValue>--event-handlers console_cohesion+ --retest-until-pass 2 --ctest-args -LE xfail --pytest-args -m "not xfail" --executor sequential</defaultValue>
    +          <defaultValue>--event-handlers console_cohesion+ --retest-until-pass 2 --ctest-args -LE xfail --pytest-args -m "not xfail"</defaultValue>
    >>>
Skipped 'test_ci_windows' because the config is the same (dry run)
Skipped 'ci_packaging_windows' because the config is the same (dry run)
Skipped 'test_packaging_windows' because the config is the same (dry run)
Skipped 'packaging_windows' because the config is the same (dry run)
Skipped 'packaging_windows_debug' because the config is the same (dry run)
Skipped 'nightly_win_deb' because the config is the same (dry run)
Skipped 'nightly_win_rel' because the config is the same (dry run)
Skipped 'nightly_win_rep' because the config is the same (dry run)
Skipped 'nightly_win_xfail' because the config is the same (dry run)
Updating job 'ci_launcher' (dry run)
    <<<
    --- remote config
    +++ new config
    @@ -174 +174 @@
    -          <defaultValue>--event-handlers console_cohesion+ --retest-until-pass 2 --ctest-args -LE xfail --pytest-args -m "not xfail" --executor sequential</defaultValue>
    +          <defaultValue>--event-handlers console_cohesion+ --retest-until-pass 2 --ctest-args -LE xfail --pytest-args -m "not xfail"</defaultValue>
    >>>
Updating job 'test_ci_launcher' (dry run)
    <<<
    --- remote config
    +++ new config
    @@ -174 +174 @@
    -          <defaultValue>--event-handlers console_cohesion+ --retest-until-pass 2 --ctest-args -LE xfail --pytest-args -m "not xfail" --executor sequential</defaultValue>
    +          <defaultValue>--event-handlers console_cohesion+ --retest-until-pass 2 --ctest-args -LE xfail --pytest-args -m "not xfail"</defaultValue>
    >>>

[clalancette]

So I realized one other thing about making this change. And that is that while we only run the nightlies against Rolling (for the most part), for all of the other jobs they are also used for Humble and Iron. And for those releases, in particular, we definitely do not have the changes in place to make things run in parallel.

I'm not quite sure where that leaves us here, but I thought I'd mention it because it is something that we'll have to contend with.

[cottsay]

Bummer.

@clalancette, I added a commit which creates a colcon defaults file that should make colcon use the sequential executor for tests by default. I need to make a similar change for Windows for this approach to be viable, but before I spend time doing that, do you think this approach might be a viable path forward?

[clalancette]

@clalancette, I added a commit which creates a colcon defaults file that should make colcon use the sequential executor for tests by default. I need to make a similar change for Windows for this approach to be viable, but before I spend time doing that, do you think this approach might be a viable path forward?

Yep, that seems like a good way forward to me. Thanks for thinking about it.

[cottsay]

Turns out to be a lot less invasive to just use the environment variable.

Rolling:

• Linux
• Linux-aarch64
• Windows

Iron:

• Linux
• Linux-aarch64
• Windows

In these jobs, you should be able to see that all of the Iron builds test sequentially, so each package's "start" marker should be followed by a "finish", "stdout", and/or "stderr" section. For rolling, the package sections will be interleaved.

[clalancette]

Turns out to be a lot less invasive to just use the environment variable.

I like that it is less invasive, but my biggest gripe with environment variables in CI (in general) is that it makes it harder to reproduce. That is, if I want to look at why something in particular failed, and want to try to reproduce locally, I need to look at:

1. The colcon command-line arguments
2. Any colcon .meta files that were installed.
3. Any COLCON environment variables.

It's not impossible, but particularly for people who aren't familiar with all of the ins-and-outs of colcon, it can be tough to follow.

So let me ask a couple of questions:

1. Can we achieve a similar thing with a .meta file instead? It's not totally ideal, but at least it is one "less" mechanism that people need to think about.
2. Can we possibly print out an "equivalent" command-line sometime during the installation? That is, assume we kept the environment variable, could we print out "you should run this command to reproduce: colcon build ... --executor sequential"?