Change retry error from RuntimeError to FlyteRecoverableException

[dansola]

Tracking issue

https://unionai.atlassian.net/browse/DOCS-385

Why are the changes needed?

It turns out that all user exceptions are considered non-recoverable unless the exception is a subclass of FlyteRecoverableException (Tasks - Flyte).

This means that the RuntimeError in the retry example here will not actually retry.

We want to change the error in the example to one which will trigger a retry of the task if the decorator is used.

Slack convo: https://unionai.slack.com/archives/C06RE4ZR5QQ/p1711576283580859

What changes were proposed in this pull request?

As only user exceptions that are subclasses of FlyteRecoverableException are compatible with retries, FlyteRecoverableException is used rather than a RuntimeError.

How was this patch tested?

This workflow was ran various times with RuntimeError (version CT3eWXJm_xOmb73ko1ww1w) and with FlyteRecoverableException (version d7nmtvMO6afNxNs4WFbTqA). With RuntimeError no retires were triggered, while with FlyteRecoverableException retries were triggered.

Screenshots

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

Docs link

[welcome]

Thank you for opening this pull request! 🙌

These tips will help get your PR across the finish line:

• Most of the repos have a PR template; if not, fill it out to the best of your knowledge.
• Sign off your commits (Reference: DCO Guide).

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 59.01%. Comparing base (c81133b) to head (4221674).
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5128      +/-   ##
==========================================
- Coverage   59.03%   59.01%   -0.02%     
==========================================
  Files         645      645              
  Lines       55726    55726              
==========================================
- Hits        32896    32888       -8     
- Misses      20233    20240       +7     
- Partials     2597     2598       +1     

Flag	Coverage Δ
unittests	59.01% <ø> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[neverett]

Thanks for opening this! For your ref link to work, I think you'll need to add .. _fault-tolerance: above the "Fault tolerance" header in this doc to create that reference, similar to the first header in that doc.

(Docs are not in great shape right now, so updating them is not straightforward, but we're working on a big project to overhaul them and make them easier to contribute to.)

[welcome]

Congrats on merging your first pull request! 🎉