Rollout stuck - persistent update conflict

[risentveber]

Checklist:

• I've included steps to reproduce the bug.
• I've included the version of argo rollouts.

Describe the bug

Rollout is stuck, due to persistent the object has been modified error during reconciliation.

There is race condition between rollouts informer watch and writeBackToInformer
https://github.com/argoproj/argo-rollouts/blob/v1.7.2/rollout/controller.go#L454.

To Reproduce

1. controller updates rollout - get version 1
2. somebody updates rollout - get version 2
3. informer watches version 2
4. writeBackToInformer rewrites version 1

Bingo - we have stale data in the informer.

Maybe related to #3316

Expected behavior

not stuck

Screenshots

Version

1.7.2

Logs

only on screenshot

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.