New RFC: Switching from publishDir to workflow output

[maxulysse]

Have you read the RFC docs?

• Yes, I have read and understood the RFC docs

Summary

Workflow output are still in preview mode, but the current third syntax is stable enough and I think vastly superior to what we do with publishDir.
Because instead of publishing files from the process, we publish them from the channels.

Champion

@maxulysse

Background & Motivation

For me the new workflow output is one of the step to help go done the way to pipeline chaining.
And I think publishing files from the channels makes more sense in a Nextflow kind of way than doing it at the process level.

Goals

• Migrate selected pipelines
• Write blogpost to help migrating
• update TEMPLATE

Non-Goals

• Migrate all pipelines

References

• Ben's PR prototype: Update to Nextflow 25.04 fetchngs#347
• Nextflow docs: https://nextflow.io/docs/latest/workflow.html#publishing-outputs
• Podcast: https://www.youtube.com/watch?v=aRpsuub5BsA
• Pipelines already using this: https://github.com/nf-core/references/

[nf-core-bot]

RFC approval status: 🕐 Pending

RFC has approvals from 0/7 required @core-team quorum.

Review Status	Core Team members
❌ Rejected	@mashehu, @ewels, @FriederikeHanssen, @jfy133
🕐 Pending	@maxulysse, @mribeirodantas, @JoseEspinosa, @mirpedrol, @kenibrewer, @edmundmiller, @sateeshperi, @FranBonath, @christopher-hakkaart, @nvnieuwk

[maxulysse]

I nominate @nvnieuwk @edmundmiller and @FriederikeHanssen as co-champion

[mashehu]

/reject too soon. needs to be out of preview mode first imo

[ewels]

/reject - agree with @mashehu.

I'd be happy to see work towards POC branches for specific pipelines, early development work etc. But I do not think that we should be releasing production pipelines or touching the template with this yet.

[maxulysse]

The way I see it, this can still be planned and not put into production until it's phased outside of preview.
We used DSL2 in preview mode for some time, no-one seemed to see any issue there.

[FriederikeHanssen]

Agree with @ewels . Sorry :(

I would vote for some of the bigger pipeline with elaborate publishing logic making a serious effort to implement this though so we can catch any remaining edge cases that are clunky or unsupported (although time will be the enemy here I think). We started some of this but abandoned it quickly with v1 or v2 because of limitations.

/reject

[ewels]

We used DSL2 in preview mode for some time, no-one seemed to see any issue there.

Yeah but nf-core was probably less than 5% of the size back then 😅

If this RFC has a roadmap included where experimental work is done in branches in preparation, and full porting only happens after October, then I'm happy.

[maxulysse]

If this RFC has a roadmap included where experimental work is done in branches in preparation, and full porting only happens after October, then I'm happy.

I can do this

[jfy133]

/reject

For same reasons above about premature

However at the same time we need it already for meta-omics for the sanplehseet based pipeline chaining, so happy to volunteer those pipelines for testing :)

[ewels]

@maxulysse shout once you have fleshed out the RFC issue text a bit and we can update votes. Shout if you want any help.

[bentsherman]

Workflow outputs will be finalized in 25.10, so that should be a good time to start adopting it.

But if you are concerned about the feasibility of updating larger pipelines, we need to work through those concerns sooner rather than later. It will be much harder to changes things after 25.10.

I can get the rnaseq PR started, but someone else needs to bring it to the finish line, making sure that the pipeline is still correct.

I have a migration guide in the works that will be published soon: nextflow-io/nextflow#6162

It's based on my preview-25-04 branch for rnaseq-nf , which is obviously nowhere near the size/complexity of nf-core, but it's good enough to demonstrate the essential concepts

It would be nice to at least allow fetchngs to use latest stable + workflow outputs on dev as I have proposed in nf-core/fetchngs#347. Or at least for someone to test the PR more thoroughly than I have. It would still be a good example for people of an nf-core pipeline that uses the syntax.

[bentsherman]

Started an rnaseq PR with the latest syntax: nf-core/rnaseq#1566