fix: move taskresult reconcile earlier to avoid workflow stuck. Fixes: #13085

[shuangkun]

Fixes #13850

Motivation

Modifications

The root cause is that taskresult has not been reconceiled. So move taskresult reconcile earlier to avoid workflow stuck.

time="2024-11-03T17:44:15.553Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1753330 namespace=argo workflow=workflow-template-hello-world-gfbjh
time="2024-11-03T17:44:15.554Z" level=debug msg="Task results completion status: map[workflow-template-hello-world-gfbjh-2798872602:false]" namespace=argo workflow=workflow-template-hello-world-gfbjh
time="2024-11-03T17:44:15.554Z" level=debug msg="taskresults of workflow are incomplete or still have daemon nodes, so can't mark workflow completed" fromPhase=Running namespace=argo toPhase=Error workflow=workflow-template-hello-world-gfbjh
time="2024-11-03T17:44:15.554Z" level=error msg="Unable to set ExecWorkflow" error="WorkflowSpec may not change during execution when the controller is set `templateReferencing: Secure`" namespace=argo workflow=workflow-template-hello-world-gfbjh

Verification

The scenario in the issue is verified and the workflow is set to error.

[shuangkun]

/retest

[isubasinghe]

I am a bit worried we are just moving this around, this was an earlier change that moved the calling of the taskresult reconcile function to where it is now.

[shuangkun]

I am a bit worried we are just moving this around, this was an earlier change that moved the calling of the taskresult reconcile function to where it is now.

Thank you for your reply. After testing, if there is a problem between processing logs and taskResultReconciliation execution (for example, the template in the templateReferencing: Secure scenario is modified), the workflow will be stuck because the taskresult of the created pod will never be reconciled and returned.

[isubasinghe]

Yeah that makes sense, thanks for taking a look.

[sstarcher]

@shuangkun thanks for your work. @isubasinghe anything we can do to move this along?

[isubasinghe]

@sstarcher we cannot move the taskresult reconcile to earlier, it will result in the workflow getting stuck. So we need to somehow address this before we can get this merged.

[shuangkun]

Theoretically, after entering Processing, the earlier the taskresult is processed, the less likely it is to get stuck.