[BUG] Inconsistent number of retries on failing tasks. Always 4 attempts.

[dansola]

Describe the bug

There arrear to be some inconsistencies with how retries are handled when set in the task decorator. According to the docs, recoverable exceptions should be respected by retries, but non-recoverable exceptions should not. I am able to see this behavior respected when I directly raise FlyteRecoverableException in a task as compared to directly raising something like RuntimeError or even FlyteAssertion. The inconsistency appears to arise when certain flyte exceptions arise (e.g. FlyteAssertion) and >1 attempt occur. Specifically, four attempts appear to occur regardless of what retries is set to.

Here is an example:

from flytekit import task, workflow
from flytekit.types.file import FlyteFile


@task(retries=2)
def my_task() -> FlyteFile:
    return FlyteFile("/not/a/real/file.txt")


@workflow
def wf():
    ff = my_task()
    print(ff)

The following error occurs here after four attempts:

TypeError: Failed to convert outputs of task 'workflows.retry_test.my_task' at position 0:
  USER:AssertionError: error=File /not/a/real/file.txt does not exist

which I would expect to not retry as it is not a recoverable exception. Regardless, if it is meant to be retired, I would expect it to respect retries=2 rather than making four attempts.

It is also worth noting that when looking at the task details in the UI, the specified number of retries is populated.

Expected behavior

I would expect the task to either not retry given that the exception is non-recoverable, or if the exception is meant to be recoverable, I would expect it to respect the number of retries specified in the decorator.

Additional context to reproduce

These examples were run on union cloud.

Screenshots

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[pvditt]

@dansola That error maps to a system error. This is expected behavior for system errors.

There's a check for retries for system errors here . MaxNodeRetriesOnSystemFailures defaults to 3, so the 4 total tries is expected.

Is the concern that the particular errors should not be considered system errors?

There's also an option to configure IgnoreRetryCause so that all retryable errors count the same against a node's retries/max attempts.

[kumare3]

Shall we close this issue? @dansola

[dansola]

Yes, thank you. I will close the issue.