Fix transient secret sync error handling

[Tom-Newton]

Tracking issue

Closes #4309

Describe your changes

• Add a new grace period configuration create-container-config-error-grace-period for CreateContainerConfigErrors similar to create-container-error-grace-period for CreateContainerError.
• I was a bit torn on whether to return a retry-able error and whether to enable cleanup.
  • Without cleanup I think I observed some issues where it was trying to make an invalid state change from FAILED -> SUCCEEDED when the grace period was exceeded but the pod continued to completion.
  • Ultimately I decided to keep it as a permanent failure without cleanup. This means non-flaky failures can still be debugged by looking at the pod events. If we enabled cleanup I think it would delete the pod and make that debugging harder.
  • I can solve my problem just by setting a long enough timeout so that all the flaky failures recover before flyte decides they have failed.
• I set the default grace period to 0 so the default behaviour should be unchanged.
• Adjust the error messages so they mention the grace period if there are failures after the grace period expires.

Check all the applicable boxes

• I updated the documentation accordingly. - The new config has the same level of documentation as create-container-error-grace-period I don't think there is anything else I need to do.
• All new and existing tests passed. - I think so. Presumably the CI checks will be the definitive answer
• All commits are signed-off.

Screenshots

Note to reviewers

• I've never used golang before so I'm just pattern matching with the existing code

[welcome]

Thank you for opening this pull request! 🙌

These tips will help get your PR across the finish line:

• Most of the repos have a PR template; if not, fill it out to the best of your knowledge.
• Sign off your commits (Reference: DCO Guide).

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (c049865) 58.89% compared to head (823711a) 58.90%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4310      +/-   ##
==========================================
+ Coverage   58.89%   58.90%   +0.01%     
==========================================
  Files         620      620              
  Lines       52440    52458      +18     
==========================================
+ Hits        30883    30901      +18     
  Misses      19090    19090              
  Partials     2467     2467              

Flag	Coverage Δ
unittests	58.90% <100.00%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[hamersaw]

@Tom-Newton this looks great! Could you fix the unit test (should be a string match update) and then I think we're good to merge.

You can run these from the root of the repo using make -C flyteplugins test_unit for the flyteplugins unit tests as an example.

[Tom-Newton]

Could you fix the unit test (should be a string match update)

Done

You can run these from the root of the repo using make -C flyteplugins test_unit for the flyteplugins unit tests as an example.

Thanks. I hadn't worked out how to run all tests.