[BUG] SHIELD has not marked a task as failed, even though the agent restarted

[thomasmitchell]

Describe the bug
The agent restarted, and its not doing that task anymore. SHIELD has not marked the task as failed in the database (it is still running). This may cause future attempts to run this task to stay pending.

To Reproduce
Steps to reproduce the behavior:

1. Schedule a long-running backup
2. Restart the agent running it
3. Check the task list

Expected behavior
The backup should have been marked as failed.

Screenshots
If applicable, add screenshots to help explain your problem.

SHIELD versions (please complete the following information):

• SHIELD Core: 8.5.0
• SHIELD CLI: 8.5.0

Browser version(s) (please complete if reporting a web UI bug):
n/a

Additional context
Add any other context about the problem here.

[jhunt]

I believe there is a (default, but configurable) 12h timeout after which time the task will be marked as cancelled.

[thomasmitchell]

This appears to be what eventually occurred.

[thomasmitchell]

Is there any way that a task can become successful if the SSH session is terminated before task completion?

[jhunt]

Hey, I never said it was a great solution 😁

If the SSH connection fails, we should fail the task immediately. I can't wait to get this into a lab and repro it to see how you pulled this one off, @thomasmitchell ...