[BUG] Handle Potential Indefinite Propeller Update Loops

[pvditt]

Tracking issue

#4349

Why are the changes needed?

A user pointed out that they had workflows stuck indefinitely in a update loop with updates failing to write to etcd.

This can happen when:

• updating the mutated workflow fails w/ ErrWorkflowToLarge propeller falls back to updating the phase of the pre-mutated wf to failing such that the wf can transition to failing on the next loop
• on the next loop, HandleFlyteWorkflow will then transition/update the wf to WorkflowPhaseFailed. This doesn't matter if updating the CRD w/ the mutated workflow fails again w/ ErrWorkflowToLarge as it will fall back to updating the phase of the pre-mutated wf to failing . Workflow then gets suck in a loop of getting updated in non-terminal WorkflowPhaseFailing

What changes were proposed in this pull request?

When encountering ErrWorkflowToLarge on an update to etcd, set the phase to Failed if the mutated workflow is already in a terminal state. This should prevent the indefinite loop described in #4349.

How was this patch tested?

• Added a unit test
• Manually updated TryMutate to always cause etcd writes to fail on updates. Then ensured that propeller would stop evaluating the workflow.

Setup process

Screenshots

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

Docs link

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (5300e1b) 58.17% compared to head (4859eb1) 58.19%.
Report is 5 commits behind head on master.

❗ Current head 4859eb1 differs from pull request most recent head 3912fb9. Consider uploading reports for the commit 3912fb9 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4755      +/-   ##
==========================================
+ Coverage   58.17%   58.19%   +0.02%     
==========================================
  Files         626      626              
  Lines       53833    53845      +12     
==========================================
+ Hits        31315    31334      +19     
+ Misses      20010    20003       -7     
  Partials     2508     2508              

Flag	Coverage Δ
unittests	58.19% <100.00%> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.