Skip to content

Commit

Permalink
Document simplified retry behaviour introduced in #3902
Browse files Browse the repository at this point in the history
Signed-off-by: Fabio M. Graetz, Ph.D. <[email protected]>
Signed-off-by: Fabio Grätz <[email protected]>
  • Loading branch information
fg91 authored and Fabio Grätz committed Sep 10, 2023
1 parent 528313f commit 48dc724
Showing 1 changed file with 4 additions and 0 deletions.
4 changes: 4 additions & 0 deletions rsts/concepts/tasks.rst
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,10 @@ System retry can be of two types:
Recoverable vs. Non-Recoverable failures: Recoverable failures will be retried and counted against the task's retry count. Non-recoverable failures will just fail, i.e., the task isn’t retried irrespective of user/system retry configurations. All user exceptions are considered non-recoverable unless the exception is a subclass of FlyteRecoverableException.


.. note::

`RFC 3902 <https://github.com/flyteorg/flyte/pull/3902>`_ implements an alternative, simplified retry behaviour with which both system and user retries are counted towards a single retry budget defined in the task decorator (thus, without a second retry budget defined in the platform configuration). The last retries are always performed on non-spot instances to guarantee completion. To activate this behaviour, set ``TODO`` to ``TODO`` in the helm values.

**Timeouts**

To ensure that the system is always making progress, tasks must be guaranteed to end gracefully/successfully. The system defines a default timeout period for the tasks. It is possible for task authors to define a timeout period, after which the task is marked as ``failure``. Note that a timed-out task will be retried if it has a retry strategy defined. The timeout can be handled in the `TaskMetadata <https://docs.flyte.org/projects/flytekit/en/latest/generated/flytekit.TaskMetadata.html?highlight=retries#flytekit.TaskMetadata>`__.
Expand Down

0 comments on commit 48dc724

Please sign in to comment.