samtools version error message

[sci-kai]

Description of the bug

Hi, I am encountering the following bug running nf-core/rnaseq v3.15.1 on an offline HPC system with PBSPro:

ERROR ~ mapping values are not allowed here
 in 'reader', line 3, column 23:
        samtools: samtools: error while loading shared lib ... 
                          ^
 -- Check script 'nf-core-rnaseq_3.15.1/3_15_1/./workflows/rnaseq/../../subworkflows/local/utils_nfcore_rnaseq_pipeline/../../nf-core/utils_nfcore_pipeline/main.nf' at line: 105 or see '.nextflow.log' file for more details
ERROR ~ Pipeline failed. Please refer to troubleshooting docs: https://nf-co.re/docs/usage/troubleshooting

 -- Check '.nextflow.log' file for details

I already investigated a bit and think the error is coming from the ALIGN_SATR:STAR_ALIGN process. It always crashes after this process. The versions.yml within the corresponding working directory of this process looks like this, probably causing this error message:

"NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN":
    star: 2.7.10a
    samtools: samtools: error while loading shared libraries: libhts.so.3: cannot open shared object file: No such file or directory
    gawk: 5.1.0

The .command.err file gives the following message:

INFO:    Converting SIF file to temporary sandbox...
work/e3/7f6271d697ca4514d4c45ed7d2d57b/.command.run: line 51: /dev/fd/63: No such file or directory
grep: /proc/stat: No such file or directory
INFO:    Cleaning up image...

While .command.out shows Successful finish (and .exitcode is set to 0)

        STAR --genomeDir star --readFilesIn input1/sample1_1.fastp.fastq.gz input2/sample1_2.fastp.fastq.gz --runThreadN 12 --outFileNamePrefix sample1. --sjdbGTFfile Homo_sapiens.GRCh38.dna_sm.primary_assembly.filtered.gtf --outSAMattrRGline ID:sample1 SM:sample1 --quantMode TranscriptomeSAM --twopassMode Basic --outSAMtype BAM Unsorted --readFilesCommand zcat --runRNGseed 0 --outFilterMultimapNmax 20 --alignSJDBoverhangMin 1 --outSAMattributes NH HI AS NM MD --quantTranscriptomeBan Singleend --outSAMstrandField intronMotif
        STAR version: 2.7.10a   compiled: 2022-01-14T18:50:00-05:00 :/home/dobin/data/STAR/STARcode/STAR.master/source
Sep 18 15:50:10 ..... started STAR run
Sep 18 15:50:10 ..... loading genome
Sep 18 15:51:04 ..... processing annotations GTF
Sep 18 15:51:20 ..... inserting junctions into the genome indices
Sep 18 15:52:47 ..... started 1st pass mapping
Sep 18 15:55:19 ..... finished 1st pass mapping
Sep 18 15:55:20 ..... inserting junctions into the genome indices
Sep 18 15:57:00 ..... started mapping
Sep 18 16:09:25 ..... finished mapping
Sep 18 16:09:36 ..... finished successfully

This error only occurs with some samples, while other finish successfully on different nextflow runs.
I assume based on this it might be a resource-related error, but not 100 % sure. If so, it would be very helpful if the process terminates with an appropriate exit code and is rerun with higher resources in the respective error strategy.

Command used and terminal output

nextflow run nf-core-rnaseq_3.15.1/3_15_1 \
-profile singularity,singlenode \
-c config/rnaseq/run-rnaseq.config \
-params-file config/rnaseq/nf-params.json \
--input $SAMPLESHEET \
-plugins [email protected] \
-resume

Relevant files

nf-params.json
run-rnaseq.config

System information

Nextflow version: 24.04.2
Hardware: HPC Cluster (offline, no internet connection on nodes)
Executor: local, run on single node of HPC Cluster
Container engine: singularity
OS: Linux
version nf-core/rnaseq: 3.15.1

[sci-kai]

I solved this error by giving the STAR processes more memory.
It would be still good if the workflow produces an exit code so that the process restarts with the respective error strategy with more memory instead of failing like this.

I added the following lines to the run.config file:

process {
    // increase memory usage to avoid error.
    withName: 'STAR_ALIGN' {
        cpus = 16
        memory = 120.GB
    }
    withName: 'STAR_GENOMEGENERATE' {
        cpus = 16
        memory = 120.GB
    }
}

Now the processes use 16 instead of 12 cores and 120 GB instead of 70 GB memory.