[Recoverable] Allow certain tasks to gracefully reconnect without crashing the entire cluster during transient errors (i.e. preemption).

[Recoverable] Allow certain tasks to gracefully reconnect without crashing the entire cluster during transient errors (i.e. preemption). #15591