Stuck VOD tasks

[figintern]

Describe the bug
When requesting certain tasks in VOD API ie. Import, Transcode, the tasks may be stuck in a false Waiting state if the task was never actually queued to run. This could happen if the API was unable to access the tasks queues due to credentials etc.

Expected behavior
Tasks which never made it into the task-runner queues should be put into a Failed state directly.

Additional context
enqueueTask in the scheduler could fail, provide a failure callback that sets the task state to Failed

https://github.com/livepeer/livepeer-com/blob/d3bd5c6e2d133436024b6df09592370a565d8e53/packages/api/src/task/scheduler.ts#L165

https://github.com/livepeer/livepeer-com/blob/d3bd5c6e2d133436024b6df09592370a565d8e53/packages/api/src/store/queue.ts#L224

[pglowacky]

Last week we implemented alarms when this happens, so we should not be alerted by customers when this happens in the future.

This is currently blocked by a decision about how best to fix this issue, pending some changes with recording and RabbitMQ. We should discuss this as a group with Victor as the decider.

[victorges]

Oh I did not now that we had this already and created this as well: #1031

The one I created is for a specific fix that we have to make though, which is not blocked by any decision AFAIK.

This is currently blocked by a decision about how best to fix this issue, pending some changes with recording and RabbitMQ. We should discuss this as a group with Victor as the decider.

@pglowacky I'm not sure if this is blocked by anything. Were you meaning to comment on this other issue instead? livepeer/task-runner#19
That one is more related to recordings, but I have also observed it happen on non-recording asset imports, so it is not exclusive either.

Also, what changes to RabbitMQ were you referring to?

[pglowacky]

Hm! I think this was the intended issue, and my comments were based on some conversation we had. If it's not true, disregard!

[pglowacky]

We currently have an alert for this so we have a rough idea for how often this is happening.