Option to clear past errors from workflow state

[Tom-Newton]

Tracking issue

flyteorg#4569

Why are the changes needed?

Reduce un-needed information stored in etcd when using failure_policy=WorkflowFailurePolicy.FAIL_AFTER_EXECUTABLE_NODES_COMPLETE. This allows flyte to scale to larger workflows before hitting etcd size limits.

What changes were proposed in this pull request?

• When using FAIL_AFTER_EXECUTABLE_NODES_COMPLETE delete the previous error from the workflow state every time a new error occurs.
• This feature is behind a feature flag clear-previous-error which defaults to false (maintaining the original behaviour).
• Update the TestWorkflowExecutor_HandleFlyteWorkflow_Failing test so that more than one node fails.
• Add sub test cases within TestWorkflowExecutor_HandleFlyteWorkflow_Failing for all the combinations of OnFailurePolicy and clear-previous-error.

How was this patch tested?

Updated unittests

Setup process

Screenshots

Check all the applicable boxes

• I updated the documentation accordingly. - Added description of the new config option. I hope to make a follow up PR to document clear-state-on-any-termination and clear-previous-error in https://docs.flyte.org/en/latest/deployment/configuration/performance.html#improving-etcd-performance
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

This is part 2
Part 1 was implemented by flyteorg#4596

Docs link