[Feature] Add an e2e test for K8s Job submitter failures

[simotw]

Why are these changes needed?

Add an test for #2579.

1. Create a RayJob
2. Wait for the Pod created by the K8s Job submitter to be ready
3. Let the Pod to run for a while (e.g. 15s)
4. Delete the Pod
5. The K8s Job should create another Pod
6. The RayJob should eventually succeed.
7. The RayJobDeployment eventually complete.
   (Add this according to the issue [Bug] RayJob falsely marked as "Running" when driver fails #2154)

Related issue number

Close #2655

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

[kevin85421]

cc @rueian would you mind reviewing this PR? Thanks!

[rueian]

LGTM.

I just also come up with a new idea of not relying on the sleep operation but using the named actor for this e2e test: the job script can exit with an manual error after registering a named actor to Ray and later being restart by k8s it can exit normally as the named actor is already registered.

This way we don’t need to wait for 15 seconds to delete the pod and 1 more minute for job completion.

What do you think @simotw, @kevin85421? Maybe this can be another PR.

[rueian]

Sorry, please ignore the above idea. It doesn’t make submitter fail.