Hotfix for v0.3.1 continued #98
Conversation
| final taskCfg = task.getConfig() | ||
| final taskCores = !taskCfg.get("cpus") ? DEFAULT_CPUS : taskCfg.get("cpus") as Integer | ||
| final taskMemory = taskCfg.get("memory") ? new MemoryUnit( taskCfg.get("memory") as String ) : new MemoryUnit(DEFAULT_MEMORY) | ||
| final taskMemory = !taskCfg.get("memory") ? new MemoryUnit(DEFAULT_MEMORY) : new MemoryUnit( taskCfg.get("memory") as String ) |
There was a problem hiding this comment.
@jagedn , I think that the CPU/memory allocation needs to be updated because of a resourceLimits directive which was introduced in Nextflow v24.04.x onwards.
Here's a good overview https://nf-co.re/docs/usage/getting_started/configuration#max-resources of the older --max_cpus and --max_memory parameters and the new native directive.
Basically, as of now, if you try to run nf-core/demo pipeline on sun-nomadlab the FASTQC jobs are not being allocated anywhere since the actual resources in the generated Nomad job definition are different from the global override we provide in nextflow.config file.
- What is set in
nextflow.config
process {
executor = "nomad"
cpus=4
memory= "4.GB"
time= "1.h"
//NOTE: Basically using the new directive here, but it is a generalized version of above settings.
resourceLimits = [ cpus: 4, memory: 4.GB, time: 1.h ]
}
- What Nomad receives
NOTE: The problem seems to be that our config is not overriding the default memory for FASTQC (as assigned via a process_medium label in the pipeline config)
I think this might be related to nextflow-io/nextflow#5306 - what are your thoughts?
In the meantime, I think I'll use a fork to control that behavior 🤔
There was a problem hiding this comment.
surely
I remember I spent lot of days trying to improve the config process but there are lot of posibilities due profiles...
Signed-off-by: Jorge Aguilera <[email protected]>
Continuing the discussion from #96 (comment)