[BUG] Etcdserver: request is too large error causing never-ending tasks and log spamming

[deadlycoconuts]

Describe the bug

We’ve observed this issue whereby the FlytePropeller isn’t failing workflows that cannot be written to the etcd store. These workflows get stuck in an enqueue/resync/fail loop which repeats itself every minute or so indefinitely, even though they are shown as ‘failed’ on the FlyteConsole. This also floods our logging services with redundant logs.

We tried toggling the useOffloadedWorkflowClosure flag but unfortunately it didn’t help resolve the etcd errors that we’ve been seeing from the FlytePropeller. We’ve restarted both the FlyteAdmin and the FlytePropeller deployments after updating the configs but that didn’t seem to stop the existing executions from throwing the etcd errors continuously. In the end we had to delete the affected executions one by one manually to stop them from being scheduled endlessly.

There are some related issues and fixes that we found from 2-3 years ago which do not seem to work anymore, so we’re wondering what might’ve changed since:

Related Issues

• [BUG] Failing to write to etcd (large object sizes) leaves workflow in non-terminal phase #363
• [BUG] Propeller should fail the workflow when it can't write to etcd #511

Related Fixes

• Large workflows will not cause workflows to stay in running state #minor flytepropeller#240

In particular, it seems like this if condition is no longer working as expected:

flyte/flytepropeller/pkg/controller/workflowstore/passthrough.go

Line 99 in c6476cc

if kubeerrors.IsRequestEntityTooLargeError(err) {

Expected behavior

Tasks should never end up in a state where they are retried endlessly and should be terminated with an error if there is an issue updating their state.

Additional context to reproduce

No response

Screenshots

Here are some images to illustrate the problem that I’ve described:

1. Affected workflow - this was first executed at 2.17 AM (SGT/HKT) on the 7th of October and failed at around 2.34 AM; it was however being restarted endlessly ever since
   
2. FlytePropeller logs when the workflow first failed
   
3. FlytePropeller logs 11 days after the initial failure
   

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[welcome]

Thank you for opening your first issue here! 🛠

[deadlycoconuts]

Here are a couple of questions we have related to this bug:

1. What happens (assuming that the aforementioned flag has been turned on) but the etcd request for a certain workflow ends up being greater than 10MB?
2. We noticed that this issue occurs not only for executions which are stuck in the ‘Failing’ state, but also those stuck in the ‘Running’ state (see screenshot below); is this expected?
   
3. Is there any in-built mechanism to stop such executions from being retried indefinitely? It’s incredibly easy for such jobs to go unnoticed and this has a huge impact on our metrics/logs, especially if these executions are all triggered as part of a regular cron job (the amount of errors raised just grows every time a new execution gets triggered only to get stuck). It’s also painstaking work having to dig through our logs to identify these executions to terminate them manually. One of these executions that we have found came from slightly over 2 months ago (see screenshot below).
   
4. Are there any metrics in the FlytePropeller (or any other components) that would allow us to identify these stuck executions? These would help us set up alerts to bring our attention to any affected executions.

[eapolinario]

@deadlycoconuts , thanks for the detailed report. A few things to kick off this investigation:

1. what version of Flyte are you running? Specifically which version of flytepropeller?
2. the useOffloadedWorkflowClosure flag only affects new workflows.

@hamersaw , can you help in answering the questions in #4349 (comment)?

[kumare3]

@eapolinario that is right, it only works for new workflows and how to enable it is documented well here - https://docs.flyte.org/en/latest/deployment/configuration/performance.html#offloading-static-workflow-information-from-crd
For already running workflows you will have to manually go and clear them.

[deadlycoconuts]

Thanks @eapolinario and @kumare3 for getting back to us. We're using v1.9.1 of the FlytePropeller. Thanks too for highlighting that the useOffloadedWorkflowClosure flag only affects new workflows (it doesn't seem like the docs specifically mention that the flag doesn't change the behaviour of existing workflows).

What do you mean by 'clearing' workflows though? Are there docs that we can refer to to perform this? Or does 'clearing' simply mean creating a new workflow version?

[leonlnj]

We have set the below and registered a new version but the error is still happening. Must it be registered under a new workflow name? Are there any steps we can follow to validate that this is configured correctly?

configmap:
 adminServer:
      flyteadmin:
        useOffloadedWorkflowClosure: true

[eapolinario]

@leonlnj , is this still happening? Can you confirm which version of Flyte you're using?