fix(backend) fix run retry for argo

[ntny]

Fix Argo Workflow v3.4.16 Retry

Argo Workflow upgrade to v3.4.16 broke the Run Retry

Long story short: there is deviation between retry specs generated via Argo v3.4.16 and kfp generated retry spec.
Argo still use node id instead of node name here. As a result incompatibility of specs
I attached according files:

• failed-argo-spec.txt - failed argo workflow spec from here
• argo-retry-spec.txt- spec generated by argo v3.4.16
• kfp-retry.txt - retry spec generated by kfp from master
• kfp-retry-after-fix.txt - spec generated by kfp after fix

Details are below:
From upgrade descriotion: Argo 3.4.16 upgrade introduces a breaking change with inconsistent node.ID vs
node.Name

However as you can see by link v3.4.16 of Argo still operates nodes by ID during retry.

Also i added the missing logic(fix node.Children/node.OutboundNodes) from the Argo Workflow retry

Using the node name instead of the ID caused Argo to fail to identify the task that needed to be retried. As a result, it would restart the entire workflow from the beginning, attempt to recreate the pod for the -system-dag-driver-, fail at this step (since such a pod already existed), It makes the workflow stuck in 'Running'.

according logs from the argo workflow-controller
level=info msg="Failed pod debug-component-pipeline-d6wjn.root-driver (debug-component-pipeline-d6wjn-system-dag-driver-2907082658) creation: already exists" namespace=arpechenin workflow=debug-component-pipeline-d6wjn

This MR is based on the assumption that when using KFP 2.3.0 & Argo 3.4.16, retries do not work on any installation, not just for some users. It has fixed Run retry on my installation.

close #11329

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign chensun for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• backend/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[google-oss-prow]

Hi @ntny. Thanks for your PR.

I'm waiting for a kubeflow member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[google-oss-prow]

@ntny: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ntny]

/retest

[google-oss-prow]

@ntny: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[juliusvonkohout]

/ok-to-test

CC @rimolive @varodrig @HumairAK @hbelmiro @kubeflow/release-team

[juliusvonkohout]

Related to kubeflow/manifests#2952

[juliusvonkohout]

/retest

[juliusvonkohout]

@kubeflow/release-team tracked as Kubeflow regression and for Kubeflow 1.10 in kubeflow/manifests#2952

[juliusvonkohout]

@HumairAK @rimolive

[hbelmiro]

/lgtm