Start september/october maintainer minutes

sites/main-site/src/content/blog/2024/maintainers-minutes-2024-10-25.mdx

@@ -0,0 +1,140 @@
+---
+title: 'Maintainers Minutes: September/October 2024'
+subtitle: 'Keeping you informed of the latest maintainers discussions'
+pubDate: 2024-10-28T12:00:00+01:00
+headerImage: '/assets/images/blog/maintainers-minutes-2024-07-26/maintainers_duck.png'
+headerImageAlt: 'Cartoon yellow rubber duck with nf-core logo badge on its body with the nf-core logo.'
+headerImageDim: [250, 250]
+embedHeaderImage: true
+authors:
+    - 'jfy133'
+    - 'MaxUlysse'
+label:
+    - 'maintainers'
+---
+
+import Profile from '@components/GitHubProfilePictureExtended.svelte';
+import { Image } from 'astro:assets';
+
+The 'Maintainers Minutes' aims to further give insight into the workings of the [nf-core maintainers team](/governance#maintainers)
+by providing brief summaries of the monthly team meetings.
+
+## Overview
+
+In this double post, we discussed the following topics in September and October:
+
+-   XXX SEPT POINT
+-   XXX SEPT PONINT
+-   [New way of passing parameters to modules within nf-test scripts](new-way-of-passing-parameters-to-modules-within-nf-test-scripts)
+-   [Stalled PRs due to waiting for review](stalled-prs-due-to-waiting-for-review)
+-   [Replacing monolithic `conf/modules.conf` into per module-configs](replacing-monolithic-confmodulesconf-into-per-module-configs)
+-   [Modification of modules specifications regarding channels](modifications-of-modules-specifications-regarding-channels)
+
+### XXX SEPT POINT
+
+### XXX SEPT POINT
+
+## New way of passing parameters to modules within nf-test scripts
+
+Our nf-core/modules have greatly benefited from the new nf-test implementation.
+However one pet peeve for many is the greatly increased number of files that now comes with each module when installing on a pipeline (>100 file pipeline PRs anyone 😱).
+
+Fortunately <Profile username="mahesh-panchal">Mahesh Binzer-Panchal</Profile> has made a [proof of concept](https://github.com/nf-core/modules/pull/5706) to help reduce this number.
+By moving parameters into `when` blocks within the main nf-test script file, we can drop the test-only `nextflow.config` file that many modules have 🎉.
+
+The proposal received unanimous approval.
+Mahesh will now begin to formalise this by updating the [nf-core/modules specifications](https://nf-co.re/docs/guidelines/components/modules), and we can start to begin adding additional linting tests to help transfer to this system.
+
+:::info
+This system only applies to modules nf-test, and should not be used at pipeline level!
+:::
+
+## Stalled PRs due to waiting for review
+
+It was raised that regularly get almost-complete module PRs, but sometimes they stall because a reviewer leaves comments but does not come back to re-review.
+Even if the module author receives a second approving review, but the author often (because we are nice here at nf-core) doesn't feel comfortable merging without getting the approval from the original reviewer.
+
+After a short discussion we felt that we can add a new reviewing guideline that if all (reasonable) comments have been addressed, and an approval has been given by another reviewer, a PR should be merged after 3 months if the original reviewer did not re-review.
+
+If the original reviewer was temporarily unable to review during this period (which can happen!), the module can still be corrected in a follow up review.
+
+## Replacing monolithic `conf/modules.conf` into per module-configs
+
+A contentious point was brought up by the developers of [nf-core/rnaseq](https://nf-co.re/rnaseq) and [nf-core/methylseq](https://nf-co.re/methylseq).
+
+Both pipelines recently have experimented with modifying the ways in which modules and processes can get customised by config files.
+Currently, the nf-core template has a single file - `conf/modules.conf` - where all module-level configurations occur (specifying additional arguments, file publication specifications etc.).
+
+The two pipelines have instead tried to split this single monolithic file into multiple, with one config file per module e.g. [in this nf-core/methylseq PR](https://github.com/nf-core/methylseq/commit/7b276c97589b65153ae23b34fcf4f6da86bb8d70).
+
+One of the main aims of this is to make CI testing more efficient, by modifications to a config only triggering related tests, rather than the whole suite of tests (as all modules are reliant on the one config file).
+Another reason is to make modules and configurations even more modular and thus portable.
+
+This proposal initially brought up a lot of consternation with most of the other maintainers present.
+The primary reason for this is the fear of findability by users, such as if they want to see which parameters of the tool is customised within the module.
+Currently we can point them to a single file within a single directory, and they can search that.
+By splitting into potentially hundreds of files, there was a fear it'll make it much harder for users to discover.
+Furthermore, the system is not currently 'natively' supported by Nextflow (where modules can automatically detect and load `nextflow.config` files next to the `main.nf`), meaning a lot of manual work to manually `include` the config for each module.
+
+A long debate occurred around this topic, ultimately between user readable vs developer efficiency and which should take more priority.
+For example, maybe efficiency should be pushed to the nf-core/tools or GitHub CI configuration level where we can do some clever things to serve the same purpose.
+Alternatively we can wait for Nextflow native support to make this less onerous to the developers.
+
+A general agreement was that this new system is not yet palatable to enough maintainers to propose push to the template, however that nf-core/rnaseq and nf-core/methylseq can continue to refine the approach.
+
+## Modification of modules specifications regarding channels
+
+Finally, we debated pros- and cons- of different ways of structuring input channels in modules.
+
+In the initial design of DSL2 nf-core modules, it was decided to require one input channel per file type, and not support 'mega-tuples', due to readability and portability.
+
+For example, this was found to be less readable:
+
+```nextflow
+input:
+tuple value(meta), path(bam), path(bai), path(fasta), path(fai), path(dict)
+```
+
+Compared to:
+
+```nextflow
+input:
+tuple value(meta), path(bam)
+tuple value(meta), path(bai)
+tuple value(meta), path(fasta)
+tuple value(meta), path(fai)
+tuple value(meta), path(dict)
+```
+
+Where ensuring each file was associated with it's companion could be facilitated with (relatively) standardised `multiMap()` calls.
+
+By forcing a particular structure of a singular input tuple (former example), it restricted pipeline developers in how they can do their own file shuttling between processes, potentially requiring many custom `map` functions to restructure channels.
+
+On the flipside, there are some instances (think `samtools`) where a particular file will _always_ be accompanied by a secondary file such as index files.
+By combining these into a single tuple would make it easier to chain such modules with extremely uniform input file requirements.
+
+Recently the module specifications had been relaxed to allow such cases of the former, but also in particular that different file format variants serving the same function (`.bai` vs `.csi` files for example), could be represented in the same element of a 'collapsed' input channel.
+
+For example, an output channel could be like so:
+
+```nextflow
+output:
+tuple value(meta), path("*bam"), path(".{bai,csi}"),
+path(fasta)
+path(fai)
+path(dict)
+```
+
+Some maintainers did not like this, and rather wanted to go back to strict and always requiring each input file to be in it's own input channel.
+Their reasoning was that it can be dangerous e.g. if a misconfiguration in a pipeline could result in the _wrong_ type of index file, and cause the pipeline to fail when a downstream process received the wrong type of index file.
+
+We discussed wishes for 'typed' inputs (possibly coming natively to Nextflow in the future), versus code clarity (not constantly having to constantly `.join` and `.combine` after each module call).
+Different philosophies came to the fore, whether trying to cover _all_ cases of errors of users is reasonable, versus pipelines only supporting specific configurations and parameters, and modifications away from this (e.g. via `ext.args`) being entirely at a users risk.
+
+This discussion remains unresolved, and will likely rear it's head again in the future.
+
+## The end
+
+As always, if you want to get involved and give your input, join the discussion on relevant PRs and slack threads!
+
+\- :heart: from your #maintainers team!