Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Move process config into pipeline code #1081

Closed
wants to merge 5 commits into from
Closed

Move process config into pipeline code #1081

wants to merge 5 commits into from

Conversation

bentsherman
Copy link

This PR moves the process config from modules.config into the pipeline code, using an experimental Nextflow feature (nextflow-io/nextflow#4375) which allows process directives to be defined in the workflow logic alongside the process invocation.

@github-actions
Copy link

github-actions bot commented Oct 5, 2023

This PR is against the master branch ❌

  • Do not close this PR
  • Click Edit and change the base to dev
  • This CI test will remain failed until you push a new commit

Hi @bentsherman,

It looks like this pull-request is has been made against the bentsherman/rnaseq master branch.
The master branch on nf-core repositories should always contain code from the latest release.
Because of this, PRs to master are only allowed if they come from the bentsherman/rnaseq dev branch.

You do not need to close this PR, you can change the target branch to dev by clicking the "Edit" button at the top of this page.
Note that even after this, the test will continue to show as failing until you push a new commit.

Thanks again for your contribution!

@bentsherman
Copy link
Author

Leave me alone CI! 😆

@github-actions
Copy link

github-actions bot commented Oct 5, 2023 •
edited
Loading

nf-core lint overall result: Failed ❌

Posted for pipeline commit 143af87

+| ✅ 140 tests passed       |+
#| ❔   4 tests were ignored |#
!| ❗   4 tests had warnings |!
-| ❌  22 tests failed       |-

❌ Test failures:

  • nextflow_config - Config variable not found: params.validationShowHiddenParams
  • nextflow_config - Config variable not found: params.validationSchemaIgnoreParams
  • files_unchanged - CODE_OF_CONDUCT.md does not match the template
  • files_unchanged - .github/CONTRIBUTING.md does not match the template
  • files_unchanged - .github/ISSUE_TEMPLATE/bug_report.yml does not match the template
  • files_unchanged - .github/workflows/linting.yml does not match the template
  • files_unchanged - assets/nf-core-rnaseq_logo_light.png does not match the template
  • schema_params - Param schema_ignore_params from nextflow config not found in nextflow_schema.json
  • schema_params - Param prepare_genome_untar_args from nextflow config not found in nextflow_schema.json
  • schema_params - Param prepare_genome_gffread_args from nextflow config not found in nextflow_schema.json
  • schema_params - Param prepare_genome_rsem_args from nextflow config not found in nextflow_schema.json
  • schema_params - Param prepare_genome_bbsplit_args from nextflow config not found in nextflow_schema.json
  • schema_params - Param subsample_fq_args from nextflow config not found in nextflow_schema.json
  • schema_params - Param subsample_salmon_args from nextflow config not found in nextflow_schema.json
  • schema_params - Param fastqc_args from nextflow config not found in nextflow_schema.json
  • schema_params - Param bbsplit_args from nextflow config not found in nextflow_schema.json
  • schema_params - Param sortmerna_args from nextflow config not found in nextflow_schema.json
  • schema_params - Param picard_args from nextflow config not found in nextflow_schema.json
  • schema_params - Param bedtools_args from nextflow config not found in nextflow_schema.json
  • schema_params - Param samtools_sort_args from nextflow config not found in nextflow_schema.json
  • schema_params - Param preseq_lcextrap_args from nextflow config not found in nextflow_schema.json
  • multiqc_config - 'assets/multiqc_config.yml' does not contain a matching 'report_comment'.

❗ Test warnings:

  • files_exist - File not found: .github/workflows/awstest.yml
  • files_exist - File not found: .github/workflows/awsfulltest.yml
  • nextflow_config - Config manifest.version should end in dev: 3.12.0
  • pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your prefered methods description, e.g. add publication citation for this pipeline

❔ Tests ignored:

  • files_unchanged - File ignored due to lint config: assets/email_template.html
  • files_unchanged - File ignored due to lint config: assets/email_template.txt
  • files_unchanged - File ignored due to lint config: lib/NfcoreTemplate.groovy
  • actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/rnaseq/rnaseq/.github/workflows/awstest.yml

✅ Tests passed:

Run details

  • nf-core/tools version 2.10
  • Run at 2023-10-16 12:55:58

@bentsherman
Copy link
Author

Most processes were straightforward, just move the process config into the pipeline code.

For withName rules that apply to multiple processes, you can keep things DRY by defining e.g. a publishDir once and re-using it throughout the workflow. If the rule applies across different workflows, well... sometimes it's okay to get a little wet 😄

For subworkflows that are used multiple times with different config in different contexts, the config can be passed through to the processes as workflow inputs. We'll find out how pretty or ugly that looks tomorrow...

But in any case, I think it's an improvement overall since the config is right next to the code it applies to, and you don't have to duplicate all of the conditional logic in modules.config.

@bentsherman
Copy link
Author

@drpatelh @ewels I managed to move all of modules.config into the pipeline code and perform a successful run with -profile test using the linked Nextflow PR.

Let me know what you guys think. Maybe we can spend some time on this at the hackathon. I'm curious to see if the config can be simplified any further now that it is defined in the proper context.

There is definitely a lot of repeated publishDir settings, could be simplified by nextflow-io/nextflow#4186.

@bentsherman
Copy link
Author

After discussing at the hackathon, we are going to try a different approach. See nextflow-io/nextflow#4422

@bentsherman bentsherman closed this Nov 9, 2023
@bentsherman bentsherman deleted the programmatic-process-config branch November 9, 2023 18:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@bentsherman