fix: ensure that nodes complete when workflow fails with parallelism and failFast. Fixes #13806

[jswxstw]

Fixes #13806

Motivation

The FailFast feature has two serious flaws:

• If parallelism is greater than 1 and one node fails while there are still other incomplete nodes, it cannot fail fast.
• It only mark the Steps node as Failed, the last StepGroup node is still Running.

Modifications

• Do not fail fast when there are incomplete nodes.
• Mark the last StepGroup node as Failed.

Verification

local test and e2e tests

Test workflow:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: demo
spec:
  entrypoint: main
  templates:
    - name: main
      parallelism: 2
      failFast: true
      steps:
        - - name: step0
            template: sleep
        - - name: step1
            template: fail
          - name: step2
            template: sleep
    - name: fail
      container:
        image: alpine:latest
        command: [sh, -c]
        args: ["exit 1"]
    - name: sleep
      container:
        image: alpine:latest
        command: [ sh, -c ]
        args: [ "sleep 5" ]

[agilgur5]

Have you seen #10312 and #11992?

[jswxstw]

Have you seen #10312 and #11992?

@agilgur5 Not yet, but #10312 is not related to this issue, failFast in these two issues is not the same, and the problems caused are also different:

• failFast field doesn't work #10312: failFast in DAGTemplate doesn't work in some scenes.
• node stuck Running when Parallelism and FailFast is enabled during parallel execution #13806: Using failFast in Template with parallelism leads to nodes getting stuck.

[jswxstw]

We usually set global workflow-level parallelism to limit resource usage, so once the workflow enables failFast during looping, issue #13806 becomes inevitable, making feature #5315 almost unusable.
Therefore, I believe this issue is quite serious and requires urgent resolution.

[MasonM]

Looks good. I ran both examples from #13806 and verified both correctly failed with this fix:

$ ./dist/argo list -o wide
NAME                STATUS      AGE   DURATION   PRIORITY   MESSAGE                                                        P/R/C   PARAMETERS
dag-4rrxm           Failed      55s   37s        0          template has failed or errored children and failFast enabled   0/0/2   
steps-gnh5p         Failed      1m    41s        0          template has failed or errored children and failFast enabled   0/0/2   

[jswxstw]

@terrytangyuan @Joibel @isubasinghe Could you please take some time to review this PR? Thank you very much!

[terrytangyuan]

LGTM. Thanks!