task_events_mgr: handle missing job config

[oliver-sanders]

• Closes polling/set: submitted task cannot be set to succeeded #6314
• There are niche situations where the job is not stored in "TaskProxy.jobs".
• This handles the situation as gracefully as we are able to.

The solution is far from ideal, the job config info has been lost for starters, however, I think that is non-functional at this stage?

@hjoliver, any better ideas?

Check List

• I have read CONTRIBUTING.md and added my name as a Code Contributor.
• Contains logically grouped changes (else tidy your branch by rebase).
• Does not contain off-topic changes (use other PRs for other changes).
• Applied any dependency changes to both setup.cfg (and conda-environment.yml if present).
• Tests are included (or explain why tests are not needed).
• Changelog entry included if this is a change that can affect users
• Cylc-Doc pull request opened if required at cylc/cylc-doc/pull/XXXX.
• If this is a bug fix, PR should be raised against the relevant ?.?.x branch.

[oliver-sanders]

This bug has hit again raising the pressure on this one.

@hjoliver, this itask.jobs list seems a bit dodgy, we just pick the last item off the list ([-1]) and hope it's the right one.

Ideally, we would probably have Job objects to match Task(Proxy) objects and a strict mapping between them which would allow us to move some job-specific config (e.g. platform, run_mode, etc) out of TaskProxy (where is doesn't belong as job submissions of the same task may have different values) into the Job config. Perfectly possible, however, this would require a reasonably large refactor which isn't something we have time for at present.

[hjoliver]

I think your fix is good.

I wonder if, as a follow-up, we should put some checks in place to ensure that all expected critical config exists in n=0 task loaded for a restart? (And remove this change once that's done.)

[hjoliver]

Ideally, we would probably have Job objects to match Task(Proxy) objects and a strict mapping between them which would allow us to move some job-specific config (e.g. platform, run_mode, etc) out of TaskProxy (where is doesn't belong as job submissions of the same task may have different values) into the Job config. Perfectly possible, however, this would require a reasonably large refactor which isn't something we have time for at present.

Sensible, but I agree it might not be a small job - post an issue?

[oliver-sanders]

Definitely a "non-small" job!

I wonder if, as a follow-up, we should put some checks in place to ensure that all expected critical config exists in n=0 task loaded for a restart?

I think what's happened here is that itask.jobs was non-critical, once, but has started to be used because the information is hard to get from elsewhere.

[oliver-sanders]

Sensible, but I agree it might not be a small job - post an issue?

#6442