Fixes random job failures in kubernetes

[mapk-amazon]

This fix addresses the random crashes of k8s jobs in Galaxy: galaxyproject/galaxy-helm#490 and mentioned therein. The issue is that k8s job status may not be ready, while providing already some information:
https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#jobstatus-v1-batch

In the previous code if len(job.obj["status"]) == 0: checked whether some information is there; if yes, treat it as "final" state of the job and continue processing. In the case that the job status has the field uncountedTerminatedPods, k8s is not done with analyzing whether the job failed or succeeded. The code then used this information (for me 0 succeeded, 0 active and 0 failed) and went through the decision tree to decide what to do.

My suggestion is to instead wait for k8s to determine the status of the terminated pods and only then decide what to do. This reduced the failure rate from 2-5% to 0% :)

How to test the changes?

• Instructions for manual testing are as follows:
  1. The change is difficult to test. It covers an edge where a/ the job is completed, b/ k8s has yet to determine the status of the job. For me 2-5% of jobs fail without the fix.

License

• I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.

[afgane]

This is a great find, and ultimately also a simple fix!

[mvdbeek]

Makes sense, thanks for figuring this out!

[nuwang]

Nice work! It would have taken some doing to figure this out!

[github-actions]

This PR was merged without a "kind/" label, please correct.

[mapk-amazon]

Thank you everyone for your support. I tested now quay.io/galaxyproject/galaxy-min:dev and it works as expected :)