Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Split variantannotation in its own pipeline and import it #1561

Open
wants to merge 24 commits into
base: dev
Choose a base branch
from
Open

Split variantannotation in its own pipeline and import it #1561

wants to merge 24 commits into from

Conversation

maxulysse
Copy link
Member

PR checklist

  • This comment contains a description of changes (with reason).
  • If you've fixed a bug or added code that should be tested, add tests!
  • If you've added a new tool - have you followed the pipeline conventions in the contribution docs
  • If necessary, also make a PR on the nf-core/sarek branch on the nf-core/test-datasets repository.
  • Make sure your code lints (nf-core lint).
  • Ensure the test suite passes (nf-test test tests/ --verbose --profile +docker).
  • Check for unexpected warnings in debug mode (nextflow run . -profile debug,test,docker --outdir <OUTDIR>).
  • Usage Documentation in docs/usage.md is updated.
  • Output Documentation in docs/output.md is updated.
  • CHANGELOG.md is updated.
  • README.md is updated (including new tool citations and authors/contributors).

@maxulysse maxulysse changed the base branch from master to dev June 11, 2024 13:45
@nf-core nf-core deleted a comment from github-actions bot Jun 11, 2024
@github-actions GitHub Actions
Copy link

github-actions bot commented Jun 11, 2024 •
edited
Loading

nf-core lint overall result: Passed ✅ ⚠️

Posted for pipeline commit 50d6441

+| ✅ 200 tests passed       |+
#| ❔  12 tests were ignored |#
!| ❗   5 tests had warnings |!

❗ Test warnings:

  • pipeline_todos - TODO string in main.nf: Optionally add in-text citation tools to this list.
  • pipeline_todos - TODO string in main.nf: Optionally add bibliographic entries to this list.
  • pipeline_todos - TODO string in main.nf: Only uncomment below if logic in toolCitationText/toolBibliographyText has been filled!
  • schema_params - Schema param input_variantannotation not found from nextflow config
  • schema_params - Schema param input_restart_variantannotation not found from nextflow config

❔ Tests ignored:

  • files_exist - File is ignored: .github/workflows/awsfulltest.yml
  • files_exist - File is ignored: .github/workflows/awstest.yml
  • files_exist - File is ignored: conf/modules.config
  • files_unchanged - File ignored due to lint config: .github/PULL_REQUEST_TEMPLATE.md
  • files_unchanged - File ignored due to lint config: assets/nf-core-sarek_logo_light.png
  • files_unchanged - File ignored due to lint config: docs/images/nf-core-sarek_logo_light.png
  • files_unchanged - File ignored due to lint config: docs/images/nf-core-sarek_logo_dark.png
  • files_unchanged - File ignored due to lint config: .gitignore or .prettierignore
  • actions_ci - actions_ci
  • actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/sarek/sarek/.github/workflows/awstest.yml
  • template_strings - template_strings
  • modules_config - modules_config

✅ Tests passed:

Run details

  • nf-core/tools version 2.14.1
  • Run at 2024-06-13 08:21:33

@maxulysse maxulysse changed the title Add variantannotation Split variantannotation in its own pipeline Jun 12, 2024
@maxulysse maxulysse changed the title Split variantannotation in its own pipeline Split variantannotation in its own pipeline and import it Jun 12, 2024
@maxulysse
Copy link
Member Author

Relevant code has been copy pasted from
https://github.com/maxulysse/nf-core_variantannotation

@maxulysse
Copy link
Member Author

maxulysse commented Jun 12, 2024 •
edited
Loading

cf: https://github.com/maxulysse/nf-core_variantannotation

My idea was to split out the pipeline specific params in nextflow.config in its own config, that way, it's easy enough to move around and import (or deal with it nicely when we go fully for another system, as what is left in nextflow.config is mainly boilerplate).

Following that idea, I similarly split out the pipeline specifics from nextflow_schema.json, but as we cannot deal with multiple nextflow_schema.json, I basically stitch it back together manually, I tried later on with jq and it was working well. Only issue is that order is a bit scrambled, but nothing that bad.

For the assets/schema_input.json, I switched from a highly flexible samplesheet to a flexible way to handle multiple samplesheets, but using the step params to assign the right samplesheet and still using a single input.

All the pipeline only relevant that should be imported into the other pipeline is in my opinion, whatever we have in workflows/<pipeline>/, and subworfklows/local/prepare_<pipeline>/.

And then dealing nicely with importing all in the main.nf.

So basically I would want a nf-core tools command like nf-core pipeline import <pipeline> -r <tag> that would do:

  • Copy nextflow_schema_<pipeline>.json and merge all nextflow_schema_*.json in a single nextflow_schema.json (using something like jq)
  • Copy nextflow_<pipeline>.config
  • Copy code actual pipeline code:
    • <pipeline>/subworkflows/local/prepare_<pipeline>/*
    • <pipeline>/workflows/<pipeline>/*
  • Copy the assets/schema_input.json as assets/schema_input_<pipeline>.json
  • A smart update of the modules.json file

EDIT: Now left with

  • Think of something smart and nice for the docs/
  • Add proper tests

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@FriederikeHanssen FriederikeHanssen Awaiting requested review from FriederikeHanssen FriederikeHanssen is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@maxulysse @FriederikeHanssen