Log job failure even when it does not cause a change in task state.

[wxtim]

Closes #6151

Check List

• I have read CONTRIBUTING.md and added my name as a Code Contributor.
• Contains logically grouped changes (else tidy your branch by rebase).
• Does not contain off-topic changes (use other PRs for other changes).
• Applied any dependency changes to both setup.cfg (and conda-environment.yml if present).
• Tests are included (or explain why tests are not needed).
• CHANGES.md entry included if this is a change that can affect users
• Cylc-Doc pull request opened if required at cylc/cylc-doc/pull/XXXX.
• If this is a bug fix, PR should be raised against the relevant ?.?.x branch.

[oliver-sanders]

Unfortunately, I don't think it's this simple.

• I think this diff means that polled log messages for task failure will go back to being duplicated.
• This only covers failure, but submission failure also has retries so may require similar treatment.
• I think the failures before retries are exhausted will now get logged at CRITICAL level rather than INFO.

[wxtim]

I think this diff means that polled log messages for task failure will go back to being duplicated.

I think that you have a particular closed issue in mind, but I can't find it... Can you point it out to me?

This only covers failure, but submission failure also has retries so may require similar treatment.

I think that submission failure is already handled correctly - it certainly is in the simplistic case where you feed it platform=garbage you get a submission failed message at level critical:

CRITICAL - [1/bar/01:preparing] submission failed
CRITICAL - [1/bar/02:preparing] submission failed
CRITICAL - [1/bar/03:preparing] submission failed

These are logged at critical - and I think they should be?

I think the failures before retries are exhausted will now get logged at CRITICAL level rather than INFO.

This would be consistent with submit failure...

[oliver-sanders]

I think this diff means that polled log messages for task failure will go back to being duplicated.

I think that you have a particular closed issue in mind, but I can't find it... Can you point it out to me?

No, I'm not thinking of the other log message duplication issue.

The change made here bypassed logic that was used for suppressing duplicate log messages (the f'{FAIL_MESSAGE_PREFIX}ERR' bit):

8f20ab0#diff-d6de42ef75ecc801c26a6be3a9dc4885b64db89d32bce6f07e319595257b9b2eL930

However, in your more recent "fix" commit, you have put this back the way it was before:

3cedf2f#diff-d6de42ef75ecc801c26a6be3a9dc4885b64db89d32bce6f07e319595257b9b2eR930

because I'm blind

[wxtim]

This does not apply to submit failure, because submit failure will always log a critical warning through the jobs-submit command.

WARNING - platform: mulberry - Could not connect to mymachine.
    * mymachine has been added to the list of unreachable hosts
    * remote-init will retry if another host is available.
ERROR - [jobs-submit cmd] (remote init)
    [jobs-submit ret_code] 1
CRITICAL - [1/bar/02:preparing] submission failed
INFO - [1/bar/02:preparing] => waiting
WARNING - [1/bar:waiting] retrying in PT1S (after 2024-07-22T14:51:37+01:00)

[MetRonnie]

(Test failures)

[wxtim]

@oliver-sanders & @MetRonnie : Sorry, but I made a total hash of rebasing this and may have introduced some regressions. :(

I think that I've been through all your comments, but please double check.

[oliver-sanders]

The first commit doesn't appear to be associated with your GH account?

[wxtim]

The first commit doesn't appear to be associated with your GH account?

I think that I've messed up the setup of NewSpiceVDI.

[oliver-sanders]

You'll either need to add a mailmap entry for the new setup, or use git rebase to amend the commit to change the author, otherwise the check-contributors action will fail when next run.

[wxtim]

You'll either need to add a mailmap entry for the new setup, or use git rebase to amend the commit to change the author, otherwise the check-contributors action will fail when next run.

Fixed - I'm not adding a metoffic.gov.uk email address to my history.

[oliver-sanders]

@hjoliver, swapping in you as a reviewer as you have also attempted this in #6401 and might want to compare approaches.

[hjoliver]

The task does change state in this case, but to waiting rather than failed.

[hjoliver]

@wxtim - I made a duplicate attempt at this without realizing you'd already worked on it, sorry. I thought I'd found a small bug with no associated issue. #6401

My bad, but having done it we might as well compare approaches. Both work, but mine is simpler (a one-liner) and I think the result is more consistent between submission and execution failure - see example below.

So my feeling is, we should use my branch, but cherry-pick your integration test to it. Would you agree?

[scheduling]
    [[graph]]
        R1 = """
           a & b
        """
[runtime]
    [[a]]
        script = """
            cylc broadcast -n a -s "script = true" $CYLC_WORKFLOW_ID
            cylc broadcast -n b -s "platform = " $CYLC_WORKFLOW_ID
            false
        """
        execution retry delays = PT5S
    [[b]]
        platform = fake
        submission retry delays = PT5S

Log comparison (left me, right you):

[MetRonnie]

@hjoliver The CRITICAL level is probably too much though? Surely WARNING is the right level?

[hjoliver]

@hjoliver The CRITICAL level is probably too much though? Surely WARNING is the right level?

Perhaps, but @wxtim's approach still leaves submit-fail (with a retry) at CRITICAL - hence my consistency comment above. Why treat the two differently?

The level is arguable. I think it's OK to log the actual job or job submission failure as critical, but have the workflow then handle it automatically.

[wxtim]

I think that the correct level is error.

@hjoliver - does your PR fall victim to any of Oliver's comments from #6169 (review)?

[MetRonnie]

If execution/submission retry delays are configured, then execution/submission failures (respectively) are expected to occur. Therefore it is not a CRITICAL message to log. Only if the retries are exhausted should it be a CRITICAL level message?