Harden dynamic job destination handling

[mvdbeek]

Earlier today galaxyproject/tpv-shared-database#61 was merged. When updating usegalaxy.org this broke a number of jobs, since this broke calculating the job destination (which we also do when a job is being handled, not just when we submit a job).

• explore if we can (at least temporarily) persist the job destination, maybe in the working directory ?
• raise a well-defined exception indicating a temporary failure in calculating the job destination (Probably in TPV)
• make it so that we don’t fail the job when we encounter that exception … just retry later if possible
• produce some kind of type interface for a given variable and validate all rules / code blocks against them (TPV)
• Version the shared database and don't deploy from main (TPV + usegalaxy playbook)
• There might be a bug in TPV that led to this situation, that also needs fixing (and a test ideally) before a new attempt at Add default resource params and slurm destination tpv-shared-database#61 can be made