Event based smoke tests -- manged jobs

[zpoint]

This PR adds support for event based testing for managed jobs. And modify some test cases to be event based.

This is the following PR of #4284, refer to the link for more detail

Tested (run the relevant ones):

Before

pytest -n 6 tests/test_smoke.py::test_launch_fast_with_autostop tests/test_smoke.py::test_aws_with_ssh_proxy_command tests/test_smoke.py::test_managed_jobs tests/test_smoke.py::test_managed_jobs_failed_setup tests/test_smoke.py::test_managed_jobs_recovery_aws tests/test_smoke.py::test_managed_jobs_cancellation_aws tests/test_smoke.py::test_multi_echo --aws

bringing up nodes...
[test_launch_fast_with_autostop] Test started. Log: less /tmp/test_launch_fast_with_autostop-b77k_ln_.log
[managed_jobs_failed_setup] Test started. Log: less /tmp/managed_jobs_failed_setup-v7zg9kze.log
[managed_jobs_recovery_aws] Test started. Log: less /tmp/managed_jobs_recovery_aws-lf_bbhur.log
[managed-jobs] Test started. Log: less /tmp/managed-jobs-x6mhjlov.log
[aws_with_ssh_proxy_command] Test started. Log: less /tmp/aws_with_ssh_proxy_command-23zowuvc.log
[managed_jobs_cancellation_aws] Test started. Log: less /tmp/managed_jobs_cancellation_aws-e3pamgz1.log
[managed_jobs_failed_setup] Passed.
[managed_jobs_failed_setup] Log: less /tmp/managed_jobs_failed_setup-v7zg9kze.log
[managed_jobs_failed_setup]
.[managed-jobs] Passed.
[managed-jobs] Log: less /tmp/managed-jobs-x6mhjlov.log
[managed-jobs]
.[test_launch_fast_with_autostop] Failed.
[test_launch_fast_with_autostop] Reason: unset SKYPILOT_DEBUG; s=$(sky launch -y -c t-launch-fast-with-au-19-42 --fast -i 1 tests/test_yamls/minimal.yaml) && echo "$s" && echo "==Validating launching==" && echo "$s" | grep -A 1 "Launching on" | grep "is up." && echo "$s" && echo "==Validating setup output==" && echo "$s" | grep -A 1 "Running setup on" | grep "running setup" && echo "==Validating running output hints==" && echo "$s" | grep -A 1 "Job submitted, ID:" | grep "Waiting for task resources on " && echo "==Validating task output starting==" && echo "$s" | grep -A 1 "Job started. Streaming logs..." | grep "(min, pid=" && echo "==Validating task output ending==" && echo "$s" | grep -A 1 "task run finish" | grep "Job finished (status: SUCCEEDED)" && echo "==Validating task output ending 2==" && echo "$s" | grep -A 5 "Job finished (status: SUCCEEDED)" | grep "Job ID:" && echo "$s" | grep -A 1 "Job ID:" | grep "Useful Commands"
[test_launch_fast_with_autostop] Log: less /tmp/test_launch_fast_with_autostop-b77k_ln_.log
[test_launch_fast_with_autostop]
F[multi_echo] Test started. Log: less /tmp/multi_echo-mvn1ilsb.log
[aws_with_ssh_proxy_command] Passed.
[aws_with_ssh_proxy_command] Log: less /tmp/aws_with_ssh_proxy_command-23zowuvc.log
[aws_with_ssh_proxy_command]
.[multi_echo] Passed.
[multi_echo] Log: less /tmp/multi_echo-mvn1ilsb.log
[multi_echo]
.[managed_jobs_recovery_aws] Passed.
[managed_jobs_recovery_aws] Log: less /tmp/managed_jobs_recovery_aws-lf_bbhur.log
[managed_jobs_recovery_aws]
.[managed_jobs_cancellation_aws] Passed.
[managed_jobs_cancellation_aws] Log: less /tmp/managed_jobs_cancellation_aws-e3pamgz1.log
[managed_jobs_cancellation_aws]
.
=================================================================================================== FAILURES ===================================================================================================
________________________________________________________________________________________ test_launch_fast_with_autostop ________________________________________________________________________________________
[gw1] linux -- Python 3.10.15 /home/ubuntu/miniconda3/envs/sky/bin/python
tests/test_smoke.py:505: in test_launch_fast_with_autostop
    run_one_test(test)
tests/test_smoke.py:289: in run_one_test
    raise Exception(f'test failed: less {log_file.name}')
E   Exception: test failed: less /tmp/test_launch_fast_with_autostop-b77k_ln_.log
=========================================================================================== short test summary info ============================================================================================
FAILED tests/test_smoke.py::test_launch_fast_with_autostop - Exception: test failed: less /tmp/test_launch_fast_with_autostop-b77k_ln_.log
1 failed, 6 passed, 1128 warnings in 1245.81s (0:20:45)

After

pytest -n 6 tests/test_smoke.py::test_launch_fast_with_autostop tests/test_smoke.py::test_aws_with_ssh_proxy_command tests/test_smoke.py::test_managed_jobs tests/test_smoke.py::test_managed_jobs_failed_setup tests/test_smoke.py::test_managed_jobs_recovery_aws tests/test_smoke.py::test_managed_jobs_cancellation_aws tests/test_smoke.py::test_multi_echo --aws
bringing up nodes...
[managed_jobs_cancellation_aws] Test started. Log: less /tmp/managed_jobs_cancellation_aws-02av2a6v.log
[aws_with_ssh_proxy_command] Test started. Log: less /tmp/aws_with_ssh_proxy_command-629fm9tm.log
[managed_jobs_failed_setup] Test started. Log: less /tmp/managed_jobs_failed_setup-ardr1p2k.log
[test_launch_fast_with_autostop] Test started. Log: less /tmp/test_launch_fast_with_autostop-370zfiuu.log
[managed-jobs] Test started. Log: less /tmp/managed-jobs-zvcyyiba.log
[managed_jobs_recovery_aws] Test started. Log: less /tmp/managed_jobs_recovery_aws-xiircbmf.log
[managed_jobs_failed_setup] Passed.
[managed_jobs_failed_setup] Log: less /tmp/managed_jobs_failed_setup-ardr1p2k.log
[managed_jobs_failed_setup]
.[managed-jobs] Passed.
[managed-jobs] Log: less /tmp/managed-jobs-zvcyyiba.log
[managed-jobs]
.[managed_jobs_recovery_aws] Passed.
[managed_jobs_recovery_aws] Log: less /tmp/managed_jobs_recovery_aws-xiircbmf.log
[managed_jobs_recovery_aws]
.[test_launch_fast_with_autostop] Passed.
[test_launch_fast_with_autostop] Log: less /tmp/test_launch_fast_with_autostop-370zfiuu.log
[test_launch_fast_with_autostop]
.[multi_echo] Test started. Log: less /tmp/multi_echo-if3cv4jd.log
[aws_with_ssh_proxy_command] Passed.
[aws_with_ssh_proxy_command] Log: less /tmp/aws_with_ssh_proxy_command-629fm9tm.log
[aws_with_ssh_proxy_command]
.[managed_jobs_cancellation_aws] Passed.
[managed_jobs_cancellation_aws] Log: less /tmp/managed_jobs_cancellation_aws-02av2a6v.log
[managed_jobs_cancellation_aws]
.[multi_echo] Passed.
[multi_echo] Log: less /tmp/multi_echo-if3cv4jd.log
[multi_echo]
.
7 passed, 1128 warnings in 671.10s (0:11:11)

50% Faster, and less flaky.

• Code formatting: bash format.sh
• Any manual or new tests for this PR (please specify below)
• All smoke tests: pytest tests/test_smoke.py
• Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
• Backward compatibility tests: conda deactivate; bash -i tests/backward_compatibility_tests.sh

[zpoint]

Hi @romilbhardwaj , Since the original PR was merged, I've submitted this new one based on the current master. Could you help review it? It's the same PR as the closed one you already reviewed, just with a different base branch.

[romilbhardwaj]

Thanks @zpoint!