Road to green CI (spurious CI failures)

[sureshjoshi]

https://pantsbuild.slack.com/archives/C0D7TNJHL/p1697563460898739

A discussion related to the untenability of our flaky CI.

Sample of timeouts:

## 2/10
✕ src/python/pants/backend/adhoc/run_system_binary_integration_test.py:tests failed in 60.07s.
## 3/10
✕ src/python/pants/backend/javascript/goals/lockfile_test.py:tests failed in 60.07s.
✕ src/python/pants/backend/javascript/subsystems/nodejs_test.py:tests failed in 240.10s.
## 4/10
✕ src/python/pants/backend/openapi/lint/spectral/rules_integration_test.py:tests failed in 120.07s.
✕ src/python/pants/backend/python/goals/coverage_py_integration_test.py:tests failed in 540.07s.
✕ src/python/pants/init/load_backends_integration_test.py:tests failed in 800.07s.
## 5/10
✕ src/python/pants/engine/streaming_workunit_handler_integration_test.py:streaming_workunit_handler_integration_test failed in 167.38s.
E       AssertionError: After 15 attempts in 30 seconds: The waiter process should have written its pid.
## 7/10
✕ src/python/pants/backend/javascript/dependency_inference/rules_test.py:tests failed in 60.10s.
## 8/10
✕ src/python/pants/option/global_options_test.py:tests failed in 60.10s.

[sureshjoshi]

From Slack, from me:

Some suggestions I've previously proposed, which may not fix the issue - but maybe mitigate it.

• Run --changed-since and --transitive tests
  • Though, since so many of our tests are integration tests, this may not help much, but if the system works well, this shouldn't hurt either
• Run/shard tests by backend/plugin - maybe we'd notice some trends if certain backends are kept together in tests/resources
  • Alternatively, run regular unit tests together, and then do "something" special for the long-running integration tests (again, this might help isolate backends, or timeouts, or resource contention, etc)
• Longer term: Figure out some way to pull the integration tests that use "Type 4" of Pants testing (e2e basically), to not use that.
  • Super slow, used as too much of a crutch

As a follow-up to Josh's point about thing slipping through the cracks due to not being modelled in Pants, we could still test-the-world nightly if we think it's appropes.

[sureshjoshi]

Flagging some potentially problematic timeouts on my Podman container, which is NOT matching our CI environment (intentionally). Trying to create a minimal, passing environment.

A common thread I've seen is the tests that require downloading external code seem to be (unsurprisingly) problematic on machines with low resources. pyenv downloading, node_js resolves, etc

• helm/goals/deploy_test.py timed out on my machine, but passed in a recent CI run by only about 10-20 seconds off of 180s

14:52:56.40 [INFO] Long running tasks:
  69.08s	Run Pytest for src/python/pants/backend/helm/goals/deploy_test.py:tests
14:53:26.50 [INFO] Long running tasks:
  99.17s	Run Pytest for src/python/pants/backend/helm/goals/deploy_test.py:tests
14:53:56.50 [INFO] Long running tasks:
  129.18s	Run Pytest for src/python/pants/backend/helm/goals/deploy_test.py:tests
14:54:26.60 [INFO] Long running tasks:
  159.27s	Run Pytest for src/python/pants/backend/helm/goals/deploy_test.py:tests
14:54:48.20 [WARN] Failed to generate JUnit XML data for src/python/pants/backend/helm/goals/deploy_test.py:tests.
14:54:48.20 [ERROR] Completed: Run Pytest - src/python/pants/backend/helm/goals/deploy_test.py:tests - failed (exit code -15).
============================= test session starts ==============================
collecting ... collected 3 items

src/python/pants/backend/helm/goals/deploy_test.py::test_run_helm_deploy PASSED [ 33%]
src/python/pants/backend/helm/goals/deploy_test.py::test_raises_error_when_using_invalid_passthrough_args PASSED [ 66%]
src/python/pants/backend/helm/goals/deploy_test.py::test_can_deploy_3rd_party_chart 


Exceeded timeout of 180.0 seconds when executing local process: Run Pytest for src/python/pants/backend/helm/goals/deploy_test.py:tests

[sureshjoshi]

javascript/package/rules_test.py

This one fails a bit in CI as well.

15:41:31.49 [INFO] Long running tasks:
  82.18s	Run Pytest for src/python/pants/backend/python/util_rules/pex_test.py:tests
15:41:43.79 [INFO] Completed: Run Pytest - src/python/pants/backend/python/util_rules/pex_test.py:tests - succeeded.
15:43:01.63 [INFO] Long running tasks:
  77.84s	Run Pytest for src/python/pants/backend/javascript/goals/test_integration_test.py:tests
15:43:31.73 [INFO] Long running tasks:
  107.94s	Run Pytest for src/python/pants/backend/javascript/goals/test_integration_test.py:tests
15:43:51.97 [INFO] Completed: Run Pytest - src/python/pants/backend/javascript/goals/test_integration_test.py:tests - succeeded.
15:44:52.10 [WARN] Failed to generate JUnit XML data for src/python/pants/backend/javascript/package/rules_test.py:test.
15:44:52.10 [ERROR] Completed: Run Pytest - src/python/pants/backend/javascript/package/rules_test.py:test - failed (exit code -15).
============================= test session starts ==============================
collecting ... collected 12 items

src/python/pants/backend/javascript/package/rules_test.py::test_creates_tar_for_package_json[pnpm] PASSED [  8%]
src/python/pants/backend/javascript/package/rules_test.py::test_creates_tar_for_package_json[npm] PASSED [ 16%]
src/python/pants/backend/javascript/package/rules_test.py::test_creates_tar_for_package_json[yarn] PASSED [ 25%]
src/python/pants/backend/javascript/package/rules_test.py::test_packages_files_as_resource[pnpm] PASSED [ 33%]
src/python/pants/backend/javascript/package/rules_test.py::test_packages_files_as_resource[npm] PASSED [ 41%]
src/python/pants/backend/javascript/package/rules_test.py::test_packages_files_as_resource[yarn] PASSED [ 50%]
src/python/pants/backend/javascript/package/rules_test.py::test_packages_files_as_resource_in_workspace[pnpm] PASSED [ 58%]
src/python/pants/backend/javascript/package/rules_test.py::test_packages_files_as_resource_in_workspace[npm] PASSED [ 66%]
src/python/pants/backend/javascript/package/rules_test.py::test_packages_files_as_resource_in_workspace[yarn] PASSED [ 75%]
src/python/pants/backend/javascript/package/rules_test.py::test_extra_envs[pnpm] 


Exceeded timeout of 60.0 seconds when executing local process: Run Pytest for src/python/pants/backend/javascript/package/rules_test.py:test

[sureshjoshi]

nodejs_tool_test

15:44:54.13 [INFO] Completed: Run Pytest - src/python/pants/backend/javascript/nodejs_project_test.py:tests - succeeded.
15:44:55.26 [INFO] Completed: Run Pytest - src/python/pants/backend/javascript/goals/test_test.py:tests - succeeded.
15:45:51.08 [INFO] Completed: Run Pytest - src/python/pants/backend/openapi/lint/spectral/rules_integration_test.py:tests - succeeded.
15:45:55.45 [INFO] Completed: Run Pytest - src/python/pants/backend/javascript/nodejs_project_environment_integration_test.py:tests - succeeded.
15:47:02.18 [INFO] Long running tasks:
  66.72s	Run Pytest for src/python/pants/backend/javascript/subsystems/nodejs_tool_test.py:tests
15:47:32.21 [INFO] Long running tasks:
  96.76s	Run Pytest for src/python/pants/backend/javascript/subsystems/nodejs_tool_test.py:tests
15:47:55.60 [WARN] Failed to generate JUnit XML data for src/python/pants/backend/javascript/subsystems/nodejs_tool_test.py:tests.
15:47:55.60 [ERROR] Completed: Run Pytest - src/python/pants/backend/javascript/subsystems/nodejs_tool_test.py:tests - failed (exit code -15).
============================= test session starts ==============================
collecting ... collected 7 items

src/python/pants/backend/javascript/subsystems/nodejs_tool_test.py::test_version_option_overrides_default PASSED [ 14%]
src/python/pants/backend/javascript/subsystems/nodejs_tool_test.py::test_execute_process_with_package_manager[yarn] PASSED [ 28%]
src/python/pants/backend/javascript/subsystems/nodejs_tool_test.py::test_execute_process_with_package_manager[npm] PASSED [ 42%]
src/python/pants/backend/javascript/subsystems/nodejs_tool_test.py::test_execute_process_with_package_manager[pnpm] PASSED [ 57%]
src/python/pants/backend/javascript/subsystems/nodejs_tool_test.py::test_resolve_dictates_version[yarn_resolve] PASSED [ 71%]
src/python/pants/backend/javascript/subsystems/nodejs_tool_test.py::test_resolve_dictates_version[pnpm_resolve] 


Exceeded timeout of 120.0 seconds when executing local process: Run Pytest for src/python/pants/backend/javascript/subsystems/nodejs_tool_test.py:tests

[sureshjoshi]

pyright? How is this a transient bug

edit... Ah maybe because we need Node to do stuff?

=================================== FAILURES ===================================
__________________ test_additional_source_roots[from_resolve] __________________

files = {'src/js/lib3/BUILD': 'package_json()', 'src/js/lib3/package-lock.json': '{"name": "@the-company/project", "lockfileVe...-company/project", "dependencies": {"pyright": "1.1.316"}}', 'src/py/project/lib1/core/BUILD': 'python_sources()', ...}
extra_args = ("--source-root-patterns=['src/py/project/lib1', 'src/py/project/lib2', 'src/js']", '--pyright-install-from-resolve=lib3')
rule_runner = RuleRunner(build_root=/tmp/_BUILD_ROOTu3itxban)

    @pytest.mark.parametrize(
        "files, extra_args",
        [
            pytest.param(
                {
                    f"{LIB_1_PACKAGE}/core/a.py": GOOD_FILE,
                    f"{LIB_1_PACKAGE}/core/BUILD": "python_sources()",
                    f"{LIB_2_PACKAGE}/core/b.py": "from core.a import add",
                    f"{LIB_2_PACKAGE}/core/BUILD": "python_sources()",
                },
                (f"--source-root-patterns=['{LIB_1_PACKAGE}', '{LIB_2_PACKAGE}']",),
                id="from_version",
            ),
            pytest.param(
                {
                    f"{LIB_1_PACKAGE}/core/a.py": GOOD_FILE,
                    f"{LIB_1_PACKAGE}/core/BUILD": "python_sources()",
                    f"{LIB_2_PACKAGE}/core/b.py": "from core.a import add",
                    f"{LIB_2_PACKAGE}/core/BUILD": "python_sources()",
                    "src/js/lib3/BUILD": "package_json()",
                    "src/js/lib3/package.json": json.dumps(
                        {"name": "@the-company/project", "dependencies": {"pyright": "1.1.316"}}
                    ),
                    "src/js/lib3/package-lock.json": PYRIGHT_LOCKFILE,
                },
                (
                    f"--source-root-patterns=['{LIB_1_PACKAGE}', '{LIB_2_PACKAGE}', 'src/js']",
                    "--pyright-install-from-resolve=lib3",
                ),
                id="from_resolve",
            ),
            pytest.param(
                {
                    f"{LIB_1_PACKAGE}/core/a.py": GOOD_FILE,
                    f"{LIB_1_PACKAGE}/core/BUILD": "python_sources()",
                    f"{LIB_2_PACKAGE}/core/b.py": "from core.a import add",
                    f"{LIB_2_PACKAGE}/core/BUILD": "python_sources()",
                    "BUILD": "package_json(name='root_package')",
                    "package.json": json.dumps(
                        {"name": "@the-company/project", "dependencies": {"pyright": "1.1.316"}}
                    ),
                    "package-lock.json": PYRIGHT_LOCKFILE,
                },
                (
                    f"--source-root-patterns=['{LIB_1_PACKAGE}', '{LIB_2_PACKAGE}', '/']",
                    "--pyright-install-from-resolve=nodejs-default",
                ),
                id="from_resolve_at_root",
            ),
        ],
    )
    def test_additional_source_roots(
        files: dict[str, str], extra_args: tuple[str, ...], rule_runner: PythonRuleRunner
    ) -> None:
        rule_runner.write_files(files)
        tgts = [
            rule_runner.get_target(Address(f"{LIB_1_PACKAGE}/core", relative_file_path="a.py")),
            rule_runner.get_target(Address(f"{LIB_2_PACKAGE}/core", relative_file_path="b.py")),
        ]
        result = run_pyright(rule_runner, tgts)
        assert len(result) == 1
        assert result[0].exit_code == 1
        assert "reportMissingImports" in result[0].stdout
    
>       result = run_pyright(
            rule_runner,
            tgts,
            extra_args=extra_args,
        )

src/python/pants/backend/python/typecheck/pyright/rules_integration_test.py:267: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/python/pants/backend/python/typecheck/pyright/rules_integration_test.py:127: in run_pyright
    result = rule_runner.request(
src/python/pants/testutil/rule_runner.py:401: in request
    result = assert_single_element(self.scheduler.product_request(output_type, [params]))
src/python/pants/engine/internals/scheduler.py:582: in product_request
    return self.execute(request)
src/python/pants/engine/internals/scheduler.py:523: in execute
    self._raise_on_error([t for _, t in throws])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <pants.engine.internals.scheduler.SchedulerSession object at 0x7ff91c4e80a0>
throws = [Throw(exc=ProcessExecutionFailure("Process 'Preparing configured [email protected].' failed with exit code 1.\nstdout:\nPrepa...ts.backend.python.typecheck.pyright.rules.pyright_typecheck', 'Typecheck using Pyright - pyright'), ('select', None)])]

    def _raise_on_error(self, throws: list[Throw]) -> NoReturn:
        exception_noun = pluralize(len(throws), "Exception")
        others_msg = f"\n(and {len(throws) - 1} more)\n" if len(throws) > 1 else ""
>       raise ExecutionError(
            f"{exception_noun} encountered:\n\n"
            f"{throws[0].render(self._scheduler.include_trace_on_error)}\n"
            f"{others_msg}",
            wrapped_exceptions=tuple(t.exc for t in throws),
        )
E       pants.engine.internals.scheduler.ExecutionError: 1 Exception encountered:
E       
E       Engine traceback:
E         in select
E           ..
E         in pants.backend.python.typecheck.pyright.rules.pyright_typecheck
E           Typecheck using Pyright - pyright
E       
E       Traceback (most recent call last):
E         File "src/python/pants/backend/python/typecheck/pyright/rules.py", line 306, in pyright_typecheck
E           partitioned_results = await MultiGet(
E         File "src/python/pants/engine/internals/selectors.py", line 374, in MultiGet
E           return await _MultiGet(tuple(__arg0))
E         File "src/python/pants/engine/internals/selectors.py", line 172, in __await__
E           result = yield self.gets
E       pants.engine.process.ProcessExecutionFailure: Process 'Preparing configured [email protected].' failed with exit code 1.
E       stdout:
E       Preparing [email protected] for immediate activation...
E       Internal Error: Error when performing the request
E           at ClientRequest.<anonymous> (/tmp/immutable_inputs0TY3rX/.tmpur9hWm/106533f8da8fc65cb5c71358ebf756ed99a30b24bc23ccf16555a633a30dbd74/node-v16.15.0-linux-x64/lib/node_modules/corepack/dist/corepack.js:3937:20)
E           at ClientRequest.emit (node:events:527:28)
E           at TLSSocket.socketErrorListener (node:_http_client:454:9)
E           at TLSSocket.emit (node:events:527:28)
E           at emitErrorNT (node:internal/streams/destroy:157:8)
E           at emitErrorCloseNT (node:internal/streams/destroy:122:3)
E           at processTicksAndRejections (node:internal/process/task_queues:83:21)
E       
E       stderr:
E       
E       
E       
E       Use `--keep-sandboxes=on_failure` to preserve the process chroot for inspection.

src/python/pants/engine/internals/scheduler.py:507: ExecutionError
------------------------------ Captured log call -------------------------------
WARNING  pants.backend.python.dependency_inference.rules:rules.py:353 Pants cannot infer owners for the following imports in the target src/py/project/lib2/core/b.py:

  * core.a.add (line: 1)

If you do not expect an import to be inferrable, add `# pants: no-infer-dep` to the import line. Otherwise, see https://www.pantsbuild.org/v2.19/docs/troubleshooting#import-errors-and-missing-dependencies for common problems.
- generated xml file: src.python.pants.backend.python.typecheck.pyright.rules_integration_test.py.tests.xml -
=========================== short test summary info ============================
FAILED src/python/pants/backend/python/typecheck/pyright/rules_integration_test.py::test_additional_source_roots[from_resolve]
============= 1 failed, 12 passed, 1 skipped in 191.76s (0:03:11) ==============

[sureshjoshi]

coverage_py

src/python/pants/backend/python/goals/coverage_py_integration_test.py::test_default_coverage_issues_12390 FAILED [100%]

=================================== FAILURES ===================================
______________________ test_default_coverage_issues_12390 ______________________

    def test_default_coverage_issues_12390() -> None:
        # N.B.: This ~replicates the repo used to reproduce this issue at
        # https://github.com/alexey-tereshenkov-oxb/monorepo-coverage-pants.
        if platform.system() == "Darwin" and platform.machine() == "arm64":
            pytest.skip(reason="No PySide2 wheels available for macOS M1")
        if platform.system() == "Linux" and platform.machine() == "aarch64":
            pytest.skip(reason="No PySide2 wheels available for Linux ARM")
        files = {
            "requirements.txt": "PySide2==5.15.2",
            "BUILD": dedent(
                """\
                python_requirements(
                    module_mapping={{
                        "PySide2": ["PySide2"],
                    }},
                )
                """
            ),
            "minimalcov/minimalcov/src/foo.py": 'print("In the foo module!")',
            "minimalcov/minimalcov/src/BUILD": "python_sources()",
            "minimalcov/minimalcov/tests/test_foo.py": dedent(
                """\
                import minimalcov.src.foo
    
                from PySide2.QtWidgets import QApplication
    
                def test_1():
                    assert True
                """
            ),
            "minimalcov/minimalcov/tests/BUILD": "python_tests()",
        }
        with setup_tmpdir(files) as tmpdir:
            command = [
                "--backend-packages=pants.backend.python",
                "test",
                "--use-coverage",
                "::",
                f"--source-root-patterns=['/{tmpdir}/minimalcov']",
                "--coverage-py-report=raw",
            ]
            result = run_pants(command)
>           result.assert_success()

src/python/pants/backend/python/goals/coverage_py_integration_test.py:409: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = PantsResult(command=['.pants.d/tmp/tmpbapvqdg_.pants.d', pid=113747)
msg = None

    def assert_success(self, msg: str | None = None) -> None:
>       assert self.exit_code == 0, self._format_unexpected_error_code_msg(msg)
E       AssertionError: .pants.d/tmp/tmpbapvqdg_.pants.d --pants-ignore=+['.coverage.*', '.python-build-standalone'] --no-pantsd --pants-config-files=[] --pants-ignore=+['/conftest.py'] --python-interpreter-constraints=['>=3.7,<4'] --backend-packages=pants.backend.python test --use-coverage :: --source-root-patterns=['/tmpw3vgscpc/minimalcov'] --coverage-py-report=raw
E         exit_code: 2
E         stdout:
E         	
E         stderr:
E         	16:26:51.31 [INFO] Starting: Building requirements.pex with 1 requirement: PySide2==5.15.2
E         	16:27:22.44 [INFO] Completed: Building requirements.pex with 1 requirement: PySide2==5.15.2
E         	16:27:22.52 [INFO] Starting: Building pytest_runner.pex
E         	16:27:24.40 [INFO] Completed: Building pytest_runner.pex
E         	16:27:24.84 [ERROR] Completed: Run Pytest - tmpw3vgscpc/minimalcov/minimalcov/tests/test_foo.py - failed (exit code 2).
E         	============================= test session starts ==============================
E         	platform linux -- Python 3.9.18, pytest-7.0.1, pluggy-1.0.0
E         	rootdir: /tmp/pants-sandbox-dtiz5N, configfile: pyproject.toml
E         	plugins: cov-3.0.0, xdist-2.5.0, forked-1.6.0
E         	collected 0 items / 1 error
E         	
E         	==================================== ERRORS ====================================
E         	_____ ERROR collecting tmpw3vgscpc/minimalcov/minimalcov/tests/test_foo.py _____
E         	ImportError while importing test module 'tmpw3vgscpc/minimalcov/minimalcov/tests/test_foo.py'.
E         	Hint: make sure your test modules/packages have valid Python names.
E         	Traceback:
E         	/usr/local/lib/python3.9/importlib/__init__.py:127: in import_module
E         	    return _bootstrap._gcd_import(name[level:], package, level)
E         	tmpw3vgscpc/minimalcov/minimalcov/tests/test_foo.py:3: in <module>
E         	    from PySide2.QtWidgets import QApplication
E         	E   ImportError: libGL.so.1: cannot open shared object file: No such file or directory
E         	------------------------------- Captured stdout --------------------------------
E         	In the foo module!
E         	=============================== warnings summary ===============================
E         	../../root/.cache/pants/named_caches/pex_root/venvs/s/8cefd7a8/venv/lib/python3.9/site-packages/_pytest/config/__init__.py:1249
E         	  /root/.cache/pants/named_caches/pex_root/venvs/s/8cefd7a8/venv/lib/python3.9/site-packages/_pytest/config/__init__.py:1249: PytestConfigWarning: Unknown config option: asyncio_mode
E         	  
E         	    self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")
E         	
E         	-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
E         	- generated xml file: tmpw3vgscpc.minimalcov.minimalcov.tests.test_foo.py.xml -
E         	
E         	
E         	=========================== short test summary info ============================
E         	ERROR tmpw3vgscpc/minimalcov/minimalcov/tests/test_foo.py
E         	!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
E         	========================= 1 warning, 1 error in 0.11s ==========================
E         	
E         	./pytest_runner.pex_pex_shim.sh: line 34: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8): No such file or directory
E         	
E         	
E         	
E         	✕ tmpw3vgscpc/minimalcov/minimalcov/tests/test_foo.py failed in 0.43s.
E         	
E         	Wrote raw coverage report to `dist/coverage/python`
E         	
E         	Name                                                  Stmts   Miss  Cover
E         	-------------------------------------------------------------------------
E         	tmpw3vgscpc/minimalcov/minimalcov/src/foo.py              1      0   100%
E         	tmpw3vgscpc/minimalcov/minimalcov/tests/test_foo.py       4      2    50%
E         	-------------------------------------------------------------------------
E         	TOTAL                                                     5      2    60%
E         	
E       assert 2 == 0
E         +2
E         -0

src/python/pants/testutil/pants_integration_test.py:58: AssertionError
----------------------------- Captured stdout call -----------------------------
pants.log +++ 
pants.log >>> 16:26:49.91 [WARN] File watcher exiting with: The watcher was shut down.
pants.log >>> 16:27:24.95 [INFO] notify invalidation: cleared 1 and dirtied 129 nodes for: {"", ".tmpV7AVnO.hardlink_canary"}
pants.log >>> 16:27:24.95 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {".tmpV7AVnO.hardlink_canary", ""}
pants.log --- 
- generated xml file: src.python.pants.backend.python.goals.coverage_py_integration_test.py.tests.xml -
=========================== short test summary info ============================
FAILED src/python/pants/backend/python/goals/coverage_py_integration_test.py::test_default_coverage_issues_12390
============= 1 failed, 15 passed, 4 skipped in 246.64s (0:04:06) ==============

[sureshjoshi]

Final failures, after a couple hours - using 1 core, 24GB of memory, and batch-size of 1 in a Podman container.

Docker failures can be ignored, since I don't have docker access inside that container.

Several of the failures were related to my environment not being setup (e.g. missing C/C++ or something).

Javascript and PyEnv might be covered with some local pre-cached downloads of NodeJS and Python (I'm guessing). I believe all of the timeouts would be removed if we split up the file so that each test got its own file + timeout (or ideally, we could run at that granularity).

✕ .github/workflows/tests/auto_cherry_picker_smoke_test.py failed in 2.50s.
✕ src/python/pants/backend/adhoc/run_system_binary_integration_test.py:tests failed in 34.82s.
✕ src/python/pants/backend/cc/util_rules/toolchain_test.py:tests failed in 1.46s.
✕ src/python/pants/backend/docker/goals/package_image_integration_test.py:tests failed in 1.80s.
✕ src/python/pants/backend/docker/goals/publish_test.py:tests failed in 2.16s.
✕ src/python/pants/backend/docker/goals/run_image_integration_test.py:tests failed in 6.68s.
✕ src/python/pants/backend/docker/util_rules/docker_build_context_test.py:tests failed in 53.17s.
✕ src/python/pants/backend/python/goals/coverage_py_integration_test.py:tests failed in 247.66s.
✕ src/python/pants/backend/python/lint/bandit/rules_integration_test.py:tests failed in 15.21s.
✕ src/python/pants/backend/python/providers/pyenv/custom_install/rules_integration_test.py:tests failed in 299.79s.
✕ src/python/pants/backend/python/typecheck/pyright/rules_integration_test.py:tests failed in 192.31s.
✕ src/python/pants/backend/python/util_rules/pex_from_targets_test.py:tests failed in 80.07s.
✕ src/python/pants/backend/shell/util_rules/shell_command_test.py:tests failed in 7.55s.
✕ src/python/pants/base/exception_sink_test.py:exception_sink_test failed in 1.30s.
✕ src/python/pants/core/goals/test_integration_test.py:integration failed in 9.84s.
✕ src/python/pants/core/util_rules/archive_test.py:tests failed in 5.30s.
✕ src/python/pants/engine/internals/docker_test.py:tests failed in 1.05s.
✕ src/python/pants/engine/internals/platform_rules_test.py:tests failed in 1.09s.
✕ src/python/pants/backend/helm/goals/deploy_test.py:tests failed in 180.70s.
✕ src/python/pants/backend/javascript/package/rules_test.py:test failed in 60.09s.
✕ src/python/pants/backend/javascript/subsystems/nodejs_tool_test.py:tests failed in 120.10s.
✕ src/python/pants/backend/python/providers/pyenv/rules_integration_test.py:tests failed in 600.36s.