finish implementing task expiration

[hjoliver]

BLOCKED: BUILT ON TOP OF #5658 - merge that PR first.

---

Question 1: should force-expiring an active task kill the job?

I think it should, and this branch does that, but the proposal does not cover this.

An already-active task will not auto expire on the clock (although I think it can be argued that it should).

But if the user explicitly commands the scheduler to expire an active task, it should kill the job then expire the task.

Rationale: expire means "don't bother to run this task, or its (non-expire) children"; therefore, if the task is already active there is no point in letting it continue, so we should cancel or kill the job. (We could expire the task without killing the job, which would orphan the job, but that leaves potential for duplicate job trouble if the task gets triggered again in a new flow).

---

Question 2: should force-expire have to be flagged in the graph?

I've implemented this as per the proposal (i.e., <task>:expire? must appear in the graph) but I think that's wrong.

• the potential for automatic (clock) expire should be flagged in the graph for visibility
• but forced expiry is by definition not going to be a surprise to the user, and is a useful intervention. It seems a bit much to expect that users must predict in advance every task that they might want to force-expire at runtime. That might just lead to some users adding expire outputs to every task in the graph as a matter of course.

---

Check List

• I have read CONTRIBUTING.md and added my name as a Code Contributor.
• Contains logically grouped changes (else tidy your branch by rebase).
• Does not contain off-topic changes (use other PRs for other changes).
• Applied any dependency changes to both setup.cfg (and conda-environment.yml if present).
• Tests are included (or explain why tests are not needed).
• CHANGES.md entry included if this is a change that can affect users
• Cylc-Doc pull request opened if required at cylc/cylc-doc/pull/XXXX.
• If this is a bug fix, PR should be raised against the relevant ?.?.x branch.

[oliver-sanders]

Questions...

"force-expiring"

I'm guessing that "force-expiring" means, setting the task to the expired state i.e: cylc set --out=expired.

Question 1: should force-expiring an active task kill the job?

The cylc set command, sets prereqs & outputs, interacting with running jobs is out of scope.

From the proposal:

[cylc set] Command for intervening in task prerequisite, output, and completion status.

Setting the succeeded or failed output on a task does not kill an active job, the expire output is no different.

To kill the job, use cylc kill, or if the use case is more aligned with question (2), use cylc remove.

Question 2: should force-expire have to be flagged in the graph?

Expiry doesn't have to be handled in the graph for every task, but if it isn't handled, and the task is expired, then the workflow will stall.

From the proposal:

Expiry is only optional if it is explicitly referenced in the graph

This is correct as it is not possible to guess the user's intention in this situation, in most cases assuming expiry is optional will either result in premature shutdown or a delayed stall which is not a desirable default.

Assuming the user's intention is aligned with optional expiry, then cylc removeis the tool for the job, it will also kill any active jobs on-route.

Remove i.e. make it as if this task never existed, rather than set the task state to expired then resolve the downstream implications.

[hjoliver]

I'm guessing that "force-expiring" means, setting the task to the expired state i.e: cylc set --out=expired.

Yes, as opposed to clock-expire which is automatic, and therefore potentially surprising if you made a mistake when writing the workflow months ago.

Force-expiring a task is by definition not surprising. You asked for it, and you got it. Unless you want to count "I didn't realize what expire does" as surprising, which would be bad form - we should provide powerful tools, and not presume that users don't understand them (the usual rm -rf example applies here!)

From the proposal:

[cylc set] Command for intervening in task prerequisite, output, and completion status.

That statement does not preclude killing an active job. "Intervening in completion status" should (at least could) have consequences for the jobs of running tasks.

Setting the succeeded or failed output on a task does not kill an active job, the expire output is no different.

Well, IF we decide that's that case. (Actually, maybe setting failed should kill the job - otherwise the failed task can be re-triggered while the original job is still running, with dire consequences for job IO).

cylc set -o expired does not just set the output, it also sets the task status to "expired" which means: "there is no need to run the task (or its children) anymore". That being the case, killing the active job, if there is one, makes perfect sense.

To kill the job, use cylc kill, or if the use case is more aligned with question (2), use cylc remove.

Those don't cover quite the same use cases, at least not without extra effort from users to get the right result, especially for scripted applications (e.g. in task scripting) where you don't necessarily know if the task is running when you want to force-expire it.

If I expire a task, by the meaning of "expire", I want the task not to run, and not to trigger anything downstream except via :expire?.

• cylc kill will result in triggering on :failed, and not on :expired? (potentially requiring additional clean-up and expire interventions)
• cylc remove will not trigger on anything, including not on :expired? (potentially requiring an additional expire intervention)

[hjoliver]

I'm more on the fence about Question 2, because otherwise (as you noted) we are in a sense assuming that expire is optional, when it's not marked as such. However, it is very easy to argue that point:

• automatic (clock) expire DOES have to be marked in the graph - no argument there
• forced expiry can only occur via deliberate use of the command, which is very different from an automatic default behaviour d
• it doesn't really make sense to require pre-flagging (in flow.cylc) of tasks that we might want to manually force to expire at run time. That could be, particularly during development say, any task in the workflow.

[As an aside: why not put the same safety net under the "default" behaviour of cylc remove? It can chop off a sub-graph just like expire, with exactly the same potential consequences, so perhaps it should cause an incomplete-task type of stall too! (The fact that there's no "remove" output doesn't change the fact that remove can cause premature shutdown or delayed stall, which is the important thing, just like expire can).]

Also, this isn't true:

This is correct as it is not possible to guess the user's intention in this situation,

We're talking about a deliberate manual intervention here, so there's no guessing needed. The user issued an "expire" command, so their intention is to expire the task, and expire has a clear meaning: there's no need to run that task or its children, except via the :expire trigger. To say we can't guess their intention here is to assume that they don't understand what they're doing (which might be true in some cases, but we can't assume that as a default!)

[hjoliver]

I'll try to distill the two main issues here. I'm arguing:

Q1

• the meaning of "expire" dictates that manually expiring an active task should cancel or kill its job
• (otherwise we end up with a completely orphaned job with the potential for bad consequences)

What exactly is your argument against that?

Q2

• for manual, unlike automatic (clock), expiry we shouldn't require :expire? in the graph
• (as a manual intervention literally any task could be the target, and we don't have to guess the intention - the user explicitly told the scheduler to do it, right now. Any manual intervention is potentially dangerous, but the user is the boss.)
• (however I'm less concerned about this than Q1. I suppose we can claim that the presence of foo:expire? in the graph shows we thought about the consequences of expiring that task, and if that's not the case forced expire should cause a stall ... BUT then so should deliberate use of cylc set -o expired).

---

(For Q1 I think it would be acceptable to have a cylc set --kill -o expired option, although I can't think of a good reason why it shouldn't just be the default).

[oliver-sanders]

What exactly is your argument against that?

• The "set" command sets things.
• The "kill" command kills things.

Makes sense, they do what they say on the tin right.

I'm not getting the "set a finished output on an active task" use requirement, it feels invalid.

Note, however, that orphaning active jobs (proposed behaviour) is a requirement, see use case 4.

---

Also, this isn't true:

This is correct as it is not possible to guess the user's intention in this situation,

No, it is very much true! Consider this example:

a[-P1] => a

And this command:

cylc set //a -o expired

What is the user's intention here?

1. Do they want the workflow to shut down?
2. Do they want to skip to the next cycle?

They haven't given you enough information to know.

IMO outcome (1) would a very surprising outcome for this command. Removing (subtractive) the task is a more intuitive way to achieve this than setting (additive) a task state.

[hjoliver]

What exactly is your argument against that?

The "set" command sets things.
The "kill" command kills things.

Makes sense, they do what they say on the tin right.

That doesn't counter my argument for what it means to expire a task. Exactly where we locate it within the command set is of secondary importance, and can be changed very easily.

I certainly agree that cylc set -o expired is not the ideal interface for "expiring a task" as opposed to simply setting the expired output. That command in particular is only in focus here because it's where we decided to locate the concept of "task expiration" before this problem came up. It already does more than it says on the tin though. It changes the task state as well setting the output. (It's the command and option names that matter here - we can easily change the docs to include killing expired active tasks, if that's what you mean by "what it says on the tin").

I'm not getting the "set a finished output on an active task" use requirement, it feels invalid.

Do you think we should not allow an active task to be expired? I'll try to be absolutely clear about the use case:

• task foo has succeed, fail, and expire triggers (the latter being what I want to happen instead of the task, if I manually expire it because I've decided it does not need to be run)
• task foo is running already when circumstances dictate that I don't need it anymore, so I want to manually expire it and run the expire sub-graph downstream of it. [There's nothing unreasonable about that!]
• so there's no reason to leave the job running now, but:
  • I don't want to kill the task (that will kick off the fail sub-graph, and not the expire sub-graph)
  • I don't want to remove the task (that won't kick off the expire sub-graph, and in any case "expire" is the appropriate concept for this scenario)
  • Just orphaning the job would be probably be OK, but clearly not the best thing to do when killing it is perfectly possible and there's a risk of getting duplicate jobs later if we don't kill it.

Note, however, that orphaning active jobs (proposed behaviour) is a requirement, see use case 4.

Yes, but that's an entirely different use case. If the scheduler cannot contact the job platform, the best we can do is set the task to failed and (potentially) orphan the job. If the scheduler could contact the job platform, we would not want to kill or orphan the job.

[hjoliver]

a[-P1] => a

cylc set //a -o expired

What is the user's intention here?

Do they want the workflow to shut down?
Do they want to skip to the next cycle?

They haven't given you enough information to know.

OK this is the essence of our disagreement.

You are essentially saying we should not respect an explicit command issued by the user because they don't understand what it means to expire a task.

I'm saying that task expiry has a clear meaning and clear consequences, and we should do what the user asks; the onus is on them to understand what they're doing when they issue a command.

The lack of an expire trigger in the graph is "information". If the graph is a[-P1] => a and I expire 1/a (say) then it should be obvious that 2/a will not be spawned. If I want to skip ahead to the next cycle, given the example graph above, then expiring 1/a without first triggering the next cycle isn't the right way to do it.

I agree that clock-expiry should be flagged in the graph because (a) it happens automatically and possibly very infrequently, and so is potentially surprising and dangerous, but also (b) it is reasonable to do that because you by definition know in advance which tasks may clock expire - that is not the case for direct manual interventions.

Removing (subtractive) the task is a more intuitive way to achieve this than setting (additive) a task state.

As per previous comments, I do agree that cylc set -o expired is not the best interface to "expire a task" as opposed to merely setting the output. But also, for the use case(s) in discussion, task expiry (wherever we put it) is the intuitive concept, not "removal".

[hjoliver]

For Q1 (job kill when expiring an active task), we more or less agree on the "set" command not being ideal for killing jobs, and you haven't really argued against the conceptual issue, so I think this could be the way forward:

cylc set sets task prerequisites and outputs (and changes state, for final outputs)

• it does not touch jobs if the target task is active

cylc kill kills jobs and sets final outputs and states accordingly

• default: kill the job and set the failed output and state
  • (use cases: kill the job and proceed as if the task failed)
• with --expire: kill the job and set the expired output and state
  • (use cases: kill the job and proceed as if we no longer need to run the task at all)

[hjoliver]

For Q2 (manually expiring tasks that have no "expired?" in the graph) I think I'll concede ☮️ I'm OK with stalling as a safety net, even though as argued above I don't really agree that it should be necessary.

BTW the extension proposal actually says this:

4. :expire cannot be required - it can only be optional or not allowed

but I think we both agree we do need to allow manually expiring tasks that do not have :expire? in the graph.

Scenario:

• I manually expire task foo for which :expired? was not declared
• the workflow stalled with foo expired and incomplete (its required "not-expired" outputs were not completed).
• now I trigger the next cycle or whatever

But how should I un-stall the workflow now? What I really want to do is expire foo (i.e. tell the scheduler we don't need to run it anymore) but it's already in the expired state!

Options:

• remove foo (but won't that make it look as if the task never existed and therefore never even expired, which is not true??)
• expire foo again, to confirm that it is truly expired and the scheduler can forget it now? (Consider this an explicit command to override the graph config that says expired is not optional).

Hmmm, actually the second option is best. We can explain it like this:

• manually expiring an incomplete task allows the scheduler to forget it, without completing it
• and that includes tasks that are incomplete because they were forced to expire "illegally"

(That fits perfectly with what it means to expire a task).

[hjoliver]

(I imagine it goes without saying, but there's no point in looking at the code on this branch until these two discussions are resolved)

[hjoliver]

[UPDATE after off-line discussions]

We've agreed:

1. extend cylc kill to allow setting of other (non-failed) options after the job kill - job kill - set task other outputs #5981
2. for manual expiry, stick with the expire-stall-remove approach

For the record: I'm not 100% happy with 2. because:

• expire as a manual intervention could in principle target any task in the workflow - i.e., it's unreasonable to expect use of :expire? for that, as opposed to automatic clock expiry; and
• a deliberate and well-defined manual intervention (expire means: no need to run the task or its dependents any more) shouldn't need the same safety net as automatic expiry

However, it's consistent and easy to explain, and a couple of extra steps won't hurt anyone. So that's OK.

Closing as superseded by #5981 ...