Dynamic declaration of array size and accessing array job variables within script

[banilmohammed]

Hi all,
Im trying to write a process which runs a script that I want to submit as an array job to my cluster running SGE. Prior to using nextflow I have created a submission script that defines resources and includes the commands to run. Something akin to this:

#!/bin/bash
#$ -cwd
#$ -j y
#$ -pe shared 1
#$ -l h_rt=24:00:00,h_data=20G
# Notify when
#$ -m a
#  Job array indexes
#$ -t 1-1000:1


# Load the job environment:
. /u/local/Modules/default/init/modules.sh

module load anaconda3
conda activate env

Rscript foo.R -n ${SGE_TASK_ID}

Running this script with qsub submission.sh gets me what I want, however I have been porting this pipeline over to nextflow and struggling to get similar functionality. Currently I have my process set up as the following:

process RUN_TOOL {
        debug true

        publishDir 'results', mode: 'symlink'

        executor "sge"
        cpus 1
        memory "20GB"
        time "5h"
        array 10

        input:
                path(genotype_dir)

        output:
             path "$genotype_dir/output"

        script:
        """
        Rscript foo.R -n ${SGE_TASK_ID}
        """
}

This currently doesn't work as nextflow tells me there is no defined variable called SGE_TASK_ID.

I have 2 questions:

1. Is there a way to access the scheduler specific variable $SGE_TASK_ID?
2. Is there a way to dynamically declare array size?

In regards to 1. I looked through the following thread #1477 (comment), where there seems to be references to SGE_TASK_ID however I can't seem to access that variable. In regards to 2., I have tried to dynamically allocate array size with a closure like: array { array_size }, however get an error message saying that the array directive cannot be declared in dynamic manner with a closure. If this is currently supported, I welcome any ideas as a workaround.

Thank you all

[bentsherman]

You just need to escape the dollar sign in the script block: \${SGE_TASK_ID}. Otherwise Nextflow will treat it as a Nextflow variable (that's how string interpolation works in Nextflow).

As for dynamic array size, it is not allowed on a per-task basis because that wouldn't make any sense. But you could still set it using a param:

array params.array_size

The closure is needed only when you want to use task inputs.

[banilmohammed]

heres the .command.run:

#!/bin/bash
#$ -N nf-RUN_PIPELINE_RUN_TOOL_(1)
#$ -o /u/nmohamm/nextflow/work/d7/edc6df23c4717b9db26d1276511d0d/.command.log
#$ -j y
#$ -terse
#$ -notify

for this process:

process RUN_TOOL{
        debug true

        publishDir 'results', mode: 'symlink'

        executor "sge"
        array 5

        input:
                path(genotype_dir)

        script:
        """
        echo "testing"
        """
}

[bentsherman]

Which version of Nextflow are you using?

[banilmohammed]

24.10.4

[bentsherman]

Very strange. I can't tell why the option isn't showing up. Unfortunately I don't have an SGE environment to test it either.

You can try different versions of Nextflow to see if it stops working at a specific point. I would try 24.04 since array job support was added in that version, just to see if it ever worked

You can also build Nextflow locally and add some debug logging to test the logic. The array job option for SGE is implemented here. It's pretty straightforward, so if it isn't doing the right thing here then I would guess that the array directive isn't being picked up for some reason.

[banilmohammed]

ill give it a shot and open up a new issue if anything comes up. thanks for the help!