Unable to use Wave with Singularity as the container engine

[marcodelapierre]

Here is a (hopefully) simple way to reproduce.

Requirements to reproduce:

• Conda
• Docker (runs OK, baseline)
• Singularity (has issues -- note I am using v3.8.7)
• Nextflow, latest edge [updated]:

    Version: 23.01.0-edge build 5838
    Created: 31-01-2023 15:12 UTC 
    System: Linux 5.15.0-58-generic
    Runtime: Groovy 3.0.14 on Java HotSpot(TM) 64-Bit Server VM 19.0.2+7-44
    Encoding: UTF-8 (UTF-8)
  

Baseline, successful:

nextflow run -r main marcodelapierre/toy-cowsay-nf -profile condawave

Attempt with Singularity:

nextflow run -r main marcodelapierre/toy-cowsay-nf -profile condawavesing

This profile has:

condawavesing {
  process.conda = 'conda-forge::cowpy'
  singularity.enabled = true
  wave {
    strategy = ['conda']
    enabled = true
  }
}

This is the error message:

[..]

Command error:
  FATAL:   could not open image wave.seqera.io/wt/9995c54e2db7/wave/build:fd8de55a0b747597234e1c02d7cf5c40: failed to retrieve path for wave.seqera.io/wt/9995c54e2db7/wave/build:fd8de55a0b747597234e1c02d7cf5c40: lstat wave.seqera.io: no such file or directory

[..]

Clue: in the .command.run of the failing process:

nxf_launch() {
    set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} singularity exec wave.seqera.io/wt/5392729906db/wave/build:fd8de55a0b747597234e1c02d7cf5c40 /bin/bash -ue /home/ubuntu/TEST/t2/work/0e/5aeadd73f4af23417c8750945d7103/.command.sh
}

[marcodelapierre]

@pditommaso this is the issue with running Wave with Singularity

[pditommaso]

Can you please try latest 23.01.0-edge ?

[marcodelapierre]

Same issue

[pditommaso]

it worked for me

$ nextflow run pditommaso/hello 
N E X T F L O W  ~  version 23.01.0-edge
Launching `https://github.com/pditommaso/hello` [curious_snyder] DSL2 - revision: 5df6c0103b [master]
[-        ] process > sayHello -
Pulling Singularity image docker://wave.seqera.io/wt/bed02b9728ee/wave/build:fd8de55a0b747597234e1c02d7cf5c40 [cachexecutor >  local (1)
[a6/d768fd] process > sayHello [100%] 1 of 1 ✔
Pulling Singularity image docker://wave.seqera.io/wt/bed02b9728ee/wave/build:fd8de55a0b747597234e1c02d7cf5c40 [cache /home/paolo/wave-showcase/sing/work/singularity/wave.seqera.io-wt-bed02b9728ee-wave-build-fd8de55a0b747597234e1c02d7cf5c40.img]
WARN: Singularity cache directory has not been defined -- Remote image will be stored in the path: /home/paolo/wave-showcase/sing/work/singularity -- Use env variable NXF_SINGULARITY_CACHEDIR to specify a different location

$ singularity --version
singularity-ce version 3.10.4-jammy

[marcodelapierre]

What env variables are you using to enable wave and the other settings?

[pditommaso]

i've just used this config file

  process.conda = 'conda-forge::cowpy'
  singularity.enabled = true
  wave {
    strategy = ['conda']
    enabled = true
  }

I'd suggest trying latest Singularity version

[marcodelapierre]

Same here, your pipeline + those settings work, my pipeline does not.
There is a difference though, your pipeline uses a pre-existing container, quay.io/nextflow/bash, whereas mine requires a build. I am wondering whether there is some fix required in the code for this latter case ..?

[marcodelapierre]

same nextflow ,.command.run with your pipeline:

nxf_launch() {
    set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} singularity exec /home/ubuntu/paolo-hello/work/singularity/wave.seqera.io-wt-35ded7dadf9c-wave-build-786daf554edcff4772a07a53f1247b56.img /bin/bash -ue /home/ubuntu/paolo-hello/work/f5/1912235d329a9998efe8e248b425c8/.command.sh
}

and with mine:

nxf_launch() {
    set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} singularity exec wave.seqera.io/wt/5392729906db/wave/build:fd8de55a0b747597234e1c02d7cf5c40 /bin/bash -ue /home/ubuntu/TEST/t2/work/0e/5aeadd73f4af23417c8750945d7103/.command.sh
}

[pditommaso]

Yep. it looks it's not converting the Wave container to a local singularity image