Updated the Sanger config to list more queues and cover multiple internal clusters

[muffato]

Supersedes #555

This config will first query the name of the cluster, to allow defining different values in the rest of the file depending on the cluster.

1. The list of queues has been expanded to include hugemem and teramem.
2. I've reduced the threshold of the small queue to 1 minute, as per our HPC guidelines.
3. The maximum resources can now be defined per cluster, and I've defined different values for tol and farm5.
4. For farm5, I've added /data to the list of mount points.
5. For the tol cluster, I've removed all the automatic mounting of /lustre, etc, because we want to make sure that all the inputs are properly defined in the pipelines. Pipelines that do require whole filesystems to be mounted need to define their own singularity.runOptions.

[muffato]

ping @BioinfoTongLI who I can't add as "reviewer" for an unknown reason

[DLBPointon]

This looks good to me @muffato, thanks for taking a look.

[maxulysse]

LGTM

[muffato]

@maxozo @BioinfoTongLI could you take a look ? This will affect you

[maxozo]

This looks good to me! Don’t think it will break anything for me. Thanks! On 10 Nov 2023, at 11:19, Matthieu Muffato ***@***.***> wrote: @maxozo @BioinfoTongLI could you take a look ? This will affect you —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[BioinfoTongLI]

As I am not an extensitve user of all the different queues. I am unsure if the if-else will capture all the use cases. Or is it trying to capture that at all?
I see that it is now not wrapped into withlabel and is default setting. Wouldn't it be safer to wrap this part into withlabel cpu-only?

[muffato]

As I am not an extensitve user of all the different queues. I am unsure if the if-else will capture all the use cases. Or is it trying to capture that at all?

The chain of if-else always has an else and defines a queue for all scenarios.

I see that it is now not wrapped into withlabel and is default setting. Wouldn't it be safer to wrap this part into withlabel cpu-only?

That's the way the configs work, I didn't change that. "CPU-only" is the default mode for processes and is captured by having the queue = {...} directly in the process block. Processes that need GPUs must add the gpu label and are then captured by the withLabel: gpu { ... } block which overrides the default values set above. Moving the default queue settings to a withLabel: cpu_only { ... } block would mean that all existing pipelines would instantly break as they couldn't get their queue settings. I don't want to do that 😅

[BioinfoTongLI]

thanks for the explanation @muffato! I think the override piece of inofrmation is the missing piece of my knowledge - That sounds then! All good to me!