Test interuptabilty of job-runner synchronous prepare/finalize steps

[bloodearnest]

If we want to automate the deployment of job-runner, we need to be able to restart at any point.

Previously, interrupting a PREPARING operation could cause issues and jobs to fail. A docker cp subprocess could hang, for example.

We're now on linux VM, and also we've moved to the BindMount API, which means docker cp (or docker at all) is not involved. We've additionally made improvements to the logic around determining if a job has finished PREPARING or not.

We should test that job-runner can be interrupted during a PREPARING stage, and restart the PREPARING step when restarted.

We should test performance of redoing the copy (copying to a destination that already exists is slow that one that doesn't). We wouldn't want a big copy to be interupted and be slower the 2nd time.

[lucyb]

For now, we should put an alert on it and observe its behaviour.

[madwort]

job server is preparing - killed & restart - does it error after the restart?