node is drained if job is canceled during epilog

[garlick]

Problem: t2304-sched-simple-alloc-check.t logs the following during cleanup:

flux-cancel: Canceled 1 job (0 errors)
Jul 07 18:51:31.221338 job-manager.err[0]: f3YA1iNB: epilog: stderr: f3YA1iNB: rank 0 failed epilog, draining

I had been ignoring this as an as-designed feature, but thought maybe I'd better get the opinion of others in case this is a potential gotcha on a production cluster.

[grondo]

Hm, that doesn't seem right. We should probably look into it.

Is the instance exiting at this point? Doesn't the broker kill all subprocesses at exit, so perhaps this is what we're seeing? Or has this reproduced with a non-exiting instance?

[garlick]

It reproduces with non-exiting. The test added in flux-framework/flux-sched#1044 cancels jobs in the middle and has to undrain in order to continue.

[grondo]

Oh, oops. The perilog module terminates the job prolog on a job exception, but the handler doesn't ensure it is the prolog that is running and not the epilog.

diff --git a/src/modules/job-manager/plugins/perilog.c b/src/modules/job-manager/plugins/perilog.c
index dd8cbc400..2b8373b23 100644
--- a/src/modules/job-manager/plugins/perilog.c
+++ b/src/modules/job-manager/plugins/perilog.c
@@ -460,6 +460,7 @@ static int exception_cb (flux_plugin_t *p,
     if ((proc = flux_jobtap_job_aux_get (p,
                                         FLUX_JOBTAP_CURRENT_JOB,
                                         "perilog_proc"))
+        && proc->prolog
         && flux_subprocess_state (proc->sp) == FLUX_SUBPROCESS_RUNNING) {
         if (prolog_kill (proc) < 0
             || prolog_kill_timer_start (proc,

I also notice the prolog will be terminated on any job exception, since the exception handler doesn't check the severity of the exception. I'll try to create a PR to fix both these issues. Sorry!