Skip to content

Commit

Permalink
feat: warning if run in job (#78)
Browse files Browse the repository at this point in the history
Already, two issues (#75 and #22) seem to result from running Snakemake
in a SLURM job context and using this executor plugin. This PR
introduces detecting if triggered within a SLURM job and issuing a
warning accordingly.

In principle, the plugin may work in job context. Submitting jobs from
jobs has always been a highlight of SLURM. However, settings may lead to
unintended behaviour (and would do so without Snakemake, presumably).
Hence, we can only warn from the executor.
  • Loading branch information
cmeesters committed Jul 5, 2024
1 parent ffff12f commit b5d66fd
Show file tree
Hide file tree
Showing 2 changed files with 19 additions and 0 deletions.
6 changes: 6 additions & 0 deletions docs/further.md
Original file line number Diff line number Diff line change
Expand Up @@ -291,6 +291,12 @@ set-resources:
Be sure to use sensible settings for your cluster and make use of parallel execution (e.g. threads) and [global profiles](#using-profiles) to avoid I/O contention.


## Nesting Jobs (or Running this Plugin within a Job)

Some environments provide a shell within a SLURM job, for instance, IDEs started in on-demand context. If Snakemake attempts to use this plugin to spawn jobs on the cluster, this may work just as intended. Or it might not: depending on cluster settings or individual settings, submitted jobs may be ill-parameterized or will not find the right environment.

If the plugin detects to be running within a job, it will therefore issue a warning and stop for 5 seconds.

## Summary:

When put together, a frequent command line looks like:
Expand Down
13 changes: 13 additions & 0 deletions snakemake_executor_plugin_slurm/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -51,11 +51,24 @@
# Implementation of your executor
class Executor(RemoteExecutor):
def __post_init__(self):
# run check whether we are running in a SLURM job context
self.warn_on_jobcontext()
self.run_uuid = str(uuid.uuid4())
self.logger.info(f"SLURM run ID: {self.run_uuid}")
self._fallback_account_arg = None
self._fallback_partition = None

def warn_on_jobcontext(self, done=None):
if not done:
if "SLURM_JOB_ID" in os.environ:
self.logger.warning(
"Running Snakemake in a SLURM within a job may lead"
" to unexpected behavior. Please run Snakemake directly"
" on the head node."
)
time.sleep(5)
done = True

def additional_general_args(self):
return "--executor slurm-jobstep --jobs 1"

Expand Down

0 comments on commit b5d66fd

Please sign in to comment.