snakemake exit with MissingOutputException when using --immediate-submit

[sirintra]

Software Versions
Snakemake 8.16.0
snakemake-executor-plugin-slurm 0.9.0
slurm 23.11.7

Describe the bug
I have recently moved to Snakemake v8.16 and have been trying to execute Snakemake with the --immediate-submit flag in order to submit all jobs to a SLURM cluster simultaneously. However, the immediate submit functionality does not seem to function properly. A simple workflow using the snakemake-executor-plugin-slurm works as expected without the --immediate-submit flag. When the immediate-submit is enabled, Snakemake continuously checks for input files for the next step, even while a submitted job is still running. Consequently, job submission terminates prematurely with a MissingOutputException.

Is there any other configuration required to ensure that the immediate-submit flag functions correctly? I’m convinced that its current behaviour is unintended.

Logs
Execute snakemake without immediate-submit flag work fine
Command executed: snakemake -s snakefile --profile slurm

Using profile slurm for setting default command line arguments.
Building DAG of jobs...
SLURM run ID: ec9655ff-b74e-4cf5-8cc4-f10ddcbf2d9b
Using shell: /usr/bin/bash
Provided remote nodes: 999
Job stats:
job      count
-----  -------
all          1
step1        1
step2        1
step3        1
total        4

Select jobs to execute...
Execute 1 jobs...

[Tue Aug 13 18:15:41 2024]
rule step1:
    input: input.txt
    output: output1.txt
    jobid: 3
    reason: Missing output files: output1.txt
    resources: mem_mb=1000, mem_mib=954, disk_mb=1000, disk_mib=954, tmpdir=<TBD>, slurm_partition=k2-hipri, runtime=60, cpus=1, slurm_account=default

cat input.txt > output1.txt
Job 3 has been submitted with SLURM jobid 11487 (log: /mnt/scratch2/users/xxx/sm_centos8/immSub/.snakemake/slurm_logs/rule_step1/11487.log).
[Tue Aug 13 18:16:21 2024]
Finished job 3.
1 of 4 steps (25%) done
Select jobs to execute...
Execute 1 jobs...

[Tue Aug 13 18:16:21 2024]
rule step2:
    input: output1.txt
    output: output2.txt
    jobid: 2
    reason: Missing output files: output2.txt; Input files updated by another job: output1.txt
    resources: mem_mb=1000, mem_mib=954, disk_mb=1000, disk_mib=954, tmpdir=<TBD>, slurm_partition=k2-hipri, runtime=60, cpus=1, slurm_account=default

sleep 40; head -n2 output1.txt > output2.txt
Job 2 has been submitted with SLURM jobid 11488 (log: /mnt/scratch2/users/xxx/sm_centos8/immSub/.snakemake/slurm_logs/rule_step2/11488.log).
[Tue Aug 13 18:17:22 2024]
Finished job 2.
2 of 4 steps (50%) done
Select jobs to execute...
Execute 1 jobs...

[Tue Aug 13 18:17:22 2024]
rule step3:
    input: output2.txt
    output: output3.txt
    jobid: 1
    reason: Missing output files: output3.txt; Input files updated by another job: output2.txt
    resources: mem_mb=1000, mem_mib=954, disk_mb=1000, disk_mib=954, tmpdir=<TBD>, slurm_partition=k2-hipri, runtime=60, cpus=1, slurm_account=default

head -n1 output2.txt > output3.txt
Job 1 has been submitted with SLURM jobid 11489 (log: /mnt/scratch2/users/xxx/sm_centos8/immSub/.snakemake/slurm_logs/rule_step3/11489.log).
[Tue Aug 13 18:17:32 2024]
Finished job 1.
3 of 4 steps (75%) done
Select jobs to execute...
Execute 1 jobs...

[Tue Aug 13 18:17:32 2024]
localrule all:
    input: output3.txt
    jobid: 0
    reason: Input files updated by another job: output3.txt
    resources: mem_mb=1000, mem_mib=954, disk_mb=1000, disk_mib=954, tmpdir=/tmp, slurm_partition=k2-hipri, runtime=60, cpus=1, slurm_account=default

[Tue Aug 13 18:17:32 2024]
Finished job 0.
4 of 4 steps (100%) done
Complete log: .snakemake/log/2024-08-13T181541.724840.snakemake.log

Execute snakemake with immediate-submit flag failed to submit all jobs and exit prematurely with the following error:
Command executed: snakemake -s snakefile --profile slurm --immediate-submit --notemp

Using profile slurm for setting default command line arguments.
Building DAG of jobs...
SLURM run ID: e55b4617-07b9-4c35-992e-7710f5874f16
Using shell: /usr/bin/bash
Provided remote nodes: 999
Job stats:
job      count
-----  -------
all          1
step1        1
step2        1
step3        1
total        4

Select jobs to execute...
Execute 1 jobs...

[Tue Aug 13 18:19:29 2024]
rule step1:
    input: input.txt
    output: output1.txt
    jobid: 3
    reason: Missing output files: output1.txt
    resources: mem_mb=1000, mem_mib=954, disk_mb=1000, disk_mib=954, tmpdir=<TBD>, slurm_partition=k2-hipri, runtime=60, cpus=1, slurm_account=default

cat input.txt > output1.txt
Job 3 has been submitted with SLURM jobid 11490 (log: /mnt/scratch2/users/xxx/sm_centos8/immSub/.snakemake/slurm_logs/rule_step1/11490.log).
Waiting at most 30 seconds for missing files.
MissingOutputException in rule step1 in file /mnt/scratch2/users/xxx/sm_centos8/immSub/snakefile, line 4:
Job 3  completed successfully, but some output files are missing. Missing files after 30 seconds. This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait:

Removing output files of failed job step1 since they might be corrupted:
output1.txt
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2024-08-13T181929.497962.snakemake.log
WorkflowError:
At least one job did not complete successfully.

Minimal example

Content of the snakefile:

rule all:
  input:
      'output3.txt'
rule step1:
  input:
      'input.txt'
  output:
      'output1.txt'
  shell:
      'cat {input} > {output}'
rule step2:
  input:
      'output1.txt'
  output:
      'output2.txt'
  shell:
      'sleep 20; head -n2 {input} > {output}'
rule step3:
  input:
      'output2.txt'
  output:
      'output3.txt'
  shell:
      'head -n1 {input} > {output}'

Additional context

config.yaml file:

executor: slurm
jobs: 999
latency-wait: 30
max-status-checks-per-second: 10
rerun-incomplete: True
scheduler: greedy
keep-going: True
printshellcmds: True
show-failed-logs: True

default-resources:
    slurm_partition: "k2-hipri"
    mem_mb: 1000
    runtime: "1h"
    cpus: 1 
    slurm_account: "default"

[cmeesters]

This is not supposed to work, see the docs on this flag, without tinkering with job dependencies. I wonder: Is this a left-over from the pre-plugin situation?

Anyway, immediate submission is actually not possible (due to missing files) when no dependencies are defined on the SLURM site. The plugin is oblivious to the workflow. It cannot define dependencies, as it "sees" a job at a time. This is not a big issue, scheduling wise, as SLURM (which might change in the future) does not assign a higher priority to waiting jobs with a dependency, only to those without.

We really need to implement a refusal if the SLURM executor is present.

[sirintra]

Thanks for the explanation. It seems that the immediate-submit function has been removed along with the deprecated --cluster flag. As a result, I wonder if it is still possible to submit all jobs from a workflow simultaneously onto the SLURM job queue?

Most of my workflows involve several thousand jobs, many of which are processed in parallel. These workflows typically take 6-8 hours to complete. Unfortunately, keeping a terminal open and reliably connected to the cluster for such extended periods is impractical. Additionally, our login sessions have a time limit of 3 hours.

Given this context, I am unsure of the most feasible approach to run my workflow on the HPC. Could you kindly recommend a workaround? Should I consider making a snakemake job submission process as a long-running SLURM job itself? Your insights would be greatly appreciated!

Lastly, I wonder if there are any plans to reintroduce an equivalent feature to immediate-submit in Snakemake version 8. Perhaps this functionality could be added to the snakemake-executor-plugin-cluster-generic? I'm sure that such an enhancement would greatly benefit users, as it provides an efficient approach for executing workflows.

[cmeesters]

... connected to the cluster for such extended periods is impractical.

You could run your workflow in a terminal multiplexer (e.g. "screen" or "tmux"), observe, that it is starting correctly, detach, log off, go for lunch, chat with a labmate (or whatever), log in again and re-attach.

Additionally, our login sessions have a time limit of 3 hours.

I get this a lot. You could approach your friendly admins and tell them that Snakemake does not accumulate CPU time and is mostly dormant (except for localrules like plotting, download, etc. - which you would execute on a login node anyway and are hardly noticed by admins and do not hinder others).

I am trying to work on a remedy so ensure within SLURM-context submissions. Not sure, it will work reliably.

Lastly, I wonder if there are any plans to reintroduce an equivalent feature to immediate-submit

Not that I am aware of, but I asked about the stance (internally) already. Snakemake itself, see about, does handle the dependencies. Perhaps, there is a way, I will let you know. Do not close the issue until then.