Funnel Tasks Fail when used as a TES Executor for Nextflow

[lbeckman314]

Overview

Current Behavior

Attempting to run Nextflow processes against Funnel results in an Executor Error soon after task creation.

Expected Behavior

Funnel should be able to successfully run Nextflow processes as a TES executor.

Possible Cause

The .command.run and .command.sh files from Nextflow are being treated as directories by Funnel.

Steps To Reproduce

 
Installing Nextflow + dependencies

curl -s "https://get.sdkman.io" | bash
source "$HOME/.sdkman/bin/sdkman-init.sh"
sdk install java 17.0.6-amzn

git clone https://github.com/nextflow-io/nextflow -b tes-update-1.1
cd nextflow
make install

export NXF_MODE=ga4gh
export NXF_EXECUTOR=tes
export NXF_EXECUTOR_TES_ENDPOINT='http://localhost:8000'

 
nextflow.config

plugins {
  id 'nf-ga4gh'
}

process.executor = 'tes'
tes.endpoint = 'http://localhost:8000'

 
Running Nextflow/nf-canary (results in Executor Error)

➜  nf-canary git:(main) ✗ ./nextflow run main.nf
N E X T F L O W  ~  version 23.08.0-edge
Launching `main.nf` [nostalgic_archimedes] DSL2 - revision: 1d6eff1807
Uploading local `bin` scripts folder to /Users/beckmanl/code/nf-canary/work/tmp/fc/897cbe65f46804b7718000d584967e/bin
executor >  tes [http://localhost:8000] (1)
[f4/d75b4f] process > NF_CANARY:TEST_SUCCESS [  0%] 0 of 1
executor >  tes [http://localhost:8000] (1)
[f4/d75b4f] process > NF_CANARY:TEST_SUCCESS [100%] 1 of 1, failed: 1 ✘
Execution cancelled -- Finishing pending tasks before exit
ERROR ~ Error executing process > 'NF_CANARY:TEST_SUCCESS'

Caused by!:
  Process `NF_CANARY:TEST_SUCCESS` terminated for an unknown reason -- Likely it has been terminated by the external system

Command executed:

  exit 0

Command exit status:
  -

Command output:
  (empty)

Work dir:  /Users/beckmanl/code/nf-canary/work/f4/d75b4f918ef545b5e541df800bc78a

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line

 -- Check '.nextflow.log' file for details

 
Nextflow work directory (with process files)

➜ head -a work/f4/d75b4f918ef545b5e541df800bc78a/.command.sh

#!/bin/bash -ue
exit 0

➜ head work/f4/d75b4f918ef545b5e541df800bc78a/.command.run

#!/bin/bash
# NEXTFLOW TASK: NF_CANARY:TEST_SUCCESS
set -e
set -u
NXF_DEBUG=${NXF_DEBUG:=0}; [[ $NXF_DEBUG > 1 ]] && set -x
NXF_ENTRY=${1:-nxf_main}

  
Funnel work directory (with process files treated as directories?)

➜ ls -a funnel-work-dir/clts53ls99bm3r94mfe0/work/.command.run/
.  ..

➜ ls -a funnel-work-dir/clts53ls99bm3r94mfe0/work/.command.sh/
.  ..

 

Environment

• Nextflow: 23.10.0.5889
• Funnel: 0.11.0-rc.2
• Java: openjdk 17.0.6 2023-01-17 LTS (Corretto)
• macOS Ventura (13.2 (22D49)

Open Questions

• Are the .command.run and .command.sh files not being shared correctly due to different work directories between Nextflow and Funnel?
• Can the Funnel (Executor) work directory be shared with Nextflow as a config setting? Alternatively can Funnel be instructed to use the Nextflow work directory?

If the executor work dir is different then I guess Nextflow should take it as a config setting

Additional Resources

• Executor work directory (‘/work’): TesTaskHandler.groovy#L185
• Nextflow GA4GH TES Docs: https://www.nextflow.io/docs/latest/executor.html#ga4gh-tes

[lbeckman314]

Nextflow PR that adds support for Funnel to be used as a TES Executor:

nextflow-io/nextflow#4195

[lbeckman314]

Resolved with following PR:
nextflow-io/nextflow#4608

See here for an example test script that runs Funnel with Nextflow:
nextflow-io/nextflow#4608 (comment)