Template merge #15

ksenia-krasheninnikova · 2023-06-15T10:32:24Z

PR checklist

priyanka-surana

Main points - documentation, template issues and code - input mismatch. Can you please make the changes? Then we can review a cleaner copy.

priyanka-surana · 2023-06-23T09:32:04Z

.nf-core.yml

Can you add Code of Conduct back in the repository and then you should be able to remove from here? You can get the latest copy from https://github.com/nf-core/tools/blob/master/nf_core/pipeline-template/CODE_OF_CONDUCT.md

priyanka-surana · 2023-06-23T09:33:01Z

LICENSE

@@ -1,6 +1,6 @@
 MIT License

-Copyright (c) 2022 Genome Research Ltd.
+Copyright (c) 2023 Genome Research Ltd


Suggested change

Copyright (c) 2023 Genome Research Ltd

Copyright (c) 2022-2023 Genome Research Ltd.

priyanka-surana · 2023-06-23T09:37:44Z

README.md

This has a lot of left over from the older version. It might be easier to make a copy of README from the template version and then fill in the TODOs.

priyanka-surana · 2023-06-23T09:38:21Z

assets/email_template.html

+  <meta name="description" content="sanger-tol/genomeassembly: A bioinformatics best-practice analysis pipeline for genome assembly from P
+acBio CCS and HiC reads">


Is the line break intentional?

priyanka-surana · 2023-06-23T09:40:05Z

assets/methods_description_template.yml

+## You inject any metadata in the Nextflow '${workflow}' object
+data: |
+  <h4>Methods</h4>
+  <p>Data was processed using sanger-tol/genomeassembly v${workflow.manifest.version} ${doi_text} of the nf-core collection of workflows (<a href="https://doi.org/10.1038/s41587-020-0439-x">Ewels <em>et al.</em>, 2020</a>).</p>


Suggested change

Data was processed using sanger-tol/genomeassembly v${workflow.manifest.version} ${doi_text} of the nf-core collection of workflows (<a href="https://doi.org/10.1038/s41587-020-0439-x">Ewels et al., 2020</a>).

Data was processed using sanger-tol/genomeassembly v${workflow.manifest.version} ${doi_text} of the sanger-tol collection of workflows, created using nf-core (<a href="https://doi.org/10.1038/s41587-020-0439-x">Ewels et al., 2020</a>).

priyanka-surana · 2023-07-18T11:18:45Z

nextflow.config

    // References
    genome                     = null
    igenomes_base              = 's3://ngi-igenomes/igenomes'
    igenomes_ignore            = false


I thought you removed igenomes...

nextflow.config

priyanka-surana · 2023-07-18T11:21:51Z

pipeline_template.yml

@@ -0,0 +1,2 @@
+prefix: sanger-tol
+skip: []


I thought you skipped igenomes

priyanka-surana · 2023-07-18T11:22:44Z

subworkflows/local/input_check.nf

The code in this file does not make sense to me, considering your input and stuff. Can you please explain/fix?

priyanka-surana · 2023-07-18T11:25:31Z

workflows/genomeassembly.nf

 // Check input path parameters to see if they exist
-def checkPathParamList = [ params.input ]
+def checkPathParamList = [ params.input, params.multiqc_config, params.fasta ]


You test for params.fasta both here and in main.nf but I don't see a params.fasta in your input. I am confused about this.

github-actions · 2023-08-16T09:51:53Z

`nf-core lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit 1788133

+| ✅ 131 tests passed       |+
#| ❔  18 tests were ignored |#
!| ❗  15 tests had warnings |!

❗ Test warnings:

files_exist - File not found: conf/igenomes.config
readme - README contains the placeholder zenodo.XXXXXXX. This should be replaced with the zenodo doi (after the first release).
pipeline_todos - TODO string in WorkflowMain.groovy: Add Zenodo DOI for pipeline after first release
pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your prefered methods description, e.g. add publication citation for this pipeline
pipeline_todos - TODO string in genomeassembly.nf: Add all file path parameters for the pipeline to the list below
pipeline_todos - TODO string in usage.md: Add documentation about anything specific to running your pipeline. For general topics, please point to (and add to) the main nf-core website.
pipeline_todos - TODO string in output.md: Write this documentation describing your workflow's output
pipeline_todos - TODO string in base.config: Check the defaults for all processes
pipeline_todos - TODO string in base.config: Customise requirements for specific processes.
pipeline_todos - TODO string in ci.yml: You can customise CI pipeline run tests as required
pipeline_todos - TODO string in awsfulltest.yml: You can customise AWS full pipeline tests as required
system_exit - System.exit in WorkflowSanger-tol-genomeassembly.groovy: System.exit(1) [line 51]
schema_description - No description provided in schema for parameter: timestamp
schema_description - No description provided in schema for parameter: hifiasm
schema_description - No description provided in schema for parameter: hifiasmhic

❔ Tests ignored:

files_exist - File is ignored: assets/nf-core-genomeassembly_logo_light.png
files_exist - File is ignored: docs/images/nf-core-genomeassembly_logo_light.png
files_exist - File is ignored: docs/images/nf-core-genomeassembly_logo_dark.png
files_exist - File is ignored: .github/ISSUE_TEMPLATE/config.yml
files_exist - File is ignored: .github/workflows/awstest.yml
files_exist - File is ignored: .github/workflows/awsfulltest.yml
nextflow_config - Config variable ignored: manifest.name
nextflow_config - Config variable ignored: manifest.homePage
files_unchanged - File ignored due to lint config: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_unchanged - File ignored due to lint config: .github/CONTRIBUTING.md
files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/bug_report.yml
files_unchanged - File ignored due to lint config: .github/workflows/linting.yml
files_unchanged - File ignored due to lint config: assets/sendmail_template.txt
files_unchanged - File does not exist: assets/nf-core-genomeassembly_logo_light.png
files_unchanged - File does not exist: docs/images/nf-core-genomeassembly_logo_light.png
files_unchanged - File does not exist: docs/images/nf-core-genomeassembly_logo_dark.png
files_unchanged - File ignored due to lint config: lib/NfcoreTemplate.groovy
files_unchanged - File ignored due to lint config: .gitignore or .prettierignore or pyproject.toml

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: conf/modules.config
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: lib/nfcore_external_java_deps.jar
files_exist - File found: lib/NfcoreSchema.groovy
files_exist - File found: lib/NfcoreTemplate.groovy
files_exist - File found: lib/Utils.groovy
files_exist - File found: lib/WorkflowMain.groovy
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: lib/WorkflowGenomeassembly.groovy
files_exist - File found: modules.json
files_exist - File found: pyproject.toml
files_exist - File not found check: Singularity
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: docs/images/nf-core-genomeassembly_logo.png
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: .travis.yml
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: params.show_hidden_params
nextflow_config - Config variable found: params.schema_ignore_params
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: '1.0dev'
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - lib/nfcore_external_java_deps.jar matches the template
files_unchanged - lib/NfcoreSchema.groovy matches the template
actions_ci - '.github/workflows/ci.yml' is triggered on expected events
actions_ci - '.github/workflows/ci.yml' checks minimum NF version
actions_awstest - '.github/workflows/awstest.yml' is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml does not use -profile test
readme - README Nextflow minimum version badge matched config. Badge: 22.10.1, Config: 22.10.1
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (208 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/yaml'
schema_params - Schema matched params returned from nextflow config
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: awstest.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
actions_schema_validation - Workflow validation passed: awsfulltest.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: branch.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
multiqc_config - 'assets/multiqc_config.yml' follows the ordering scheme of the minimally required plugins.
multiqc_config - 'assets/multiqc_config.yml' contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'

Run details

nf-core/tools version 2.8
Run at 2023-09-14 10:32:20

Changes to README according to the TEMPLATE updates

priyanka-surana

I am unable to test the pipeline at this point. This is just a documentation review.
The nf-core linting error might be fixed by modules update.

priyanka-surana · 2023-09-06T09:08:46Z

README.md


 The pipeline is built using [Nextflow](https://www.nextflow.io), a workflow tool to run tasks across multiple compute infrastructures in a very portable manner. It uses Docker/Singularity containers making installation trivial and results highly reproducible. The [Nextflow DSL2](https://www.nextflow.io/docs/latest/dsl2.html) implementation of this pipeline uses one container per process which makes it much easier to maintain and update software dependencies. Where possible, these processes have been submitted to and installed from [nf-core/modules](https://github.com/nf-core/modules) in order to make them available to all nf-core pipelines, and to everyone within the Nextflow community!

-<!-- TODO nf-core: Add full-sized test dataset and amend the paragraph below if applicable -->
-
 On release, automated continuous integration tests run the pipeline on a full-sized dataset on the AWS cloud infrastructure. This ensures that the pipeline runs on AWS, has sensible resource allocation defaults set to run on real-world datasets, and permits the persistent storage of results to benchmark between pipeline releases and other analysis sources.


This section should have automatically been removed as part of the template merge. Seems like the merging was just messy.

Suggested change

On release, automated continuous integration tests run the pipeline on a full-sized dataset on the AWS cloud infrastructure. This ensures that the pipeline runs on AWS, has sensible resource allocation defaults set to run on real-world datasets, and permits the persistent storage of results to benchmark between pipeline releases and other analysis sources.

priyanka-surana · 2023-09-06T09:10:39Z

assets/schema_input.json

I cannot test this right now.

priyanka-surana · 2023-09-06T09:11:33Z

conf/base.config

    cpus   = { check_max( 1    * task.attempt, 'cpus'   ) }
    memory = { check_max( 6.GB * task.attempt, 'memory' ) }
    time   = { check_max( 4.h  * task.attempt, 'time'   ) }

-    errorStrategy = { task.exitStatus in [143,137,104,134,139] ? 'retry' : 'finish' }
+    errorStrategy = { task.exitStatus in ((130..145) + 104) ? 'retry' : 'finish' }
    maxRetries    = 1


You might want to up this number as it is no longer controlled by the main sanger.config

Sorry, I don't quite get this comment. This is the list of error codes on which the pipelines should be retried. Do you mean to top it up with some other error codes?

Figured it out. Asked Matthieu.

conf/modules.config

priyanka-surana · 2023-09-06T09:18:47Z

nextflow_schema.json

                    "schema": "assets/schema_input.json",
-                    "description": "Path to YAML file containing information about the samples in the experiment.",
+                    "description": "Path to comma-separated file containing information about the samples in the experiment.",


Suggested change

"description": "Path to comma-separated file containing information about the samples in the experiment.",

"description": "Path to YAML file containing information about the samples in the experiment.",

You can use the nf-core schema build tool to check the rest of the document via the online GUI.

muffato · 2023-09-08T15:30:38Z

@muffato Does this make sense? Is this too Sanger specific to be here?

See my comments and my suggestion in #19

Updated config for running longranger with and without LSF

… into template_merge

priyanka-surana

Documentation will be done in separate PR. This is good to go.

Ksenia Krasheninnikova added 2 commits June 15, 2023 10:25

Initial template commit

27fafd1

Merged vanilla TEMPLATE branch into main pipeline

6f23969

ksenia-krasheninnikova requested a review from priyanka-surana June 15, 2023 10:32

priyanka-surana suggested changes Jul 18, 2023

View reviewed changes

Ksenia Krasheninnikova added 6 commits August 3, 2023 10:39

Add changes based on code review

20d5f31

Fixes according to code review

73ed8a5

Modify input configuration

b27dc2f

Set default value for hifiasm hic mode

ddb7086

Add formatting fixes and updated test file

4cfbd5e

Update test full

20f78da

ksenia-krasheninnikova force-pushed the template_merge branch from 72409da to 20f78da Compare August 16, 2023 09:49

ksenia-krasheninnikova requested a review from priyanka-surana August 16, 2023 09:53

ksenia-krasheninnikova and others added 6 commits August 25, 2023 11:28

Update README.md

e0f2eeb

Changes to README according to the TEMPLATE updates

Correcting nf-core version

5fad421

Linting change

67730e7

Prettier fixes

d5549b2

Remove extra file

5e3012f

Ignore files when doing prettier

48b9d89

priyanka-surana reviewed Sep 6, 2023

View reviewed changes

Ksenia Krasheninnikova and others added 9 commits September 6, 2023 11:24

Remove leftovers of the old template in README

012daa0

Update nf-core modules

7b9c3f6

Prettier fix

e51360f

Remove diff

591f758

Fix prettier

dfd215f

Fixes for parameters descriptions

943242f

Fix liniting

904347b

Prettify

3768dec

Updated config for running longranger with and without LSF

dfafb26

Merge pull request #19 from sanger-tol/longranger_lsf

c6ce6fb

Updated config for running longranger with and without LSF

ksenia-krasheninnikova requested a review from priyanka-surana September 11, 2023 13:35

Ksenia Krasheninnikova added 4 commits September 14, 2023 10:22

Set max retries

efeb7ac

Merge branch 'template_merge' of github.com:sanger-tol/genomeassembly…

f703ac3

… into template_merge

Linting checks

5821aef

Prettier checks

1788133

priyanka-surana approved these changes Sep 18, 2023

View reviewed changes

priyanka-surana merged commit 3bf48e2 into dev Sep 20, 2023
4 of 6 checks passed

priyanka-surana deleted the template_merge branch September 20, 2023 12:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Template merge #15

Template merge #15

ksenia-krasheninnikova commented Jun 15, 2023

priyanka-surana left a comment •

edited

Loading

priyanka-surana Jun 23, 2023

priyanka-surana Jun 23, 2023

priyanka-surana Jun 23, 2023

priyanka-surana Jun 23, 2023

priyanka-surana Jun 23, 2023

priyanka-surana Jul 18, 2023

priyanka-surana Jul 18, 2023

priyanka-surana Jul 18, 2023 •

edited

Loading

priyanka-surana Jul 18, 2023 •

edited

Loading

github-actions bot commented Aug 16, 2023 •

edited

Loading

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

priyanka-surana left a comment •

edited

Loading

priyanka-surana Sep 6, 2023 •

edited

Loading

priyanka-surana Sep 6, 2023

priyanka-surana Sep 6, 2023

ksenia-krasheninnikova Sep 8, 2023

ksenia-krasheninnikova Sep 8, 2023

priyanka-surana Sep 6, 2023

muffato commented Sep 8, 2023

priyanka-surana left a comment

	Copyright (c) 2023 Genome Research Ltd
	Copyright (c) 2022-2023 Genome Research Ltd.

		<meta name="description" content="sanger-tol/genomeassembly: A bioinformatics best-practice analysis pipeline for genome assembly from P
		acBio CCS and HiC reads">

	<p>Data was processed using sanger-tol/genomeassembly v${workflow.manifest.version} ${doi_text} of the nf-core collection of workflows (<a href="https://doi.org/10.1038/s41587-020-0439-x">Ewels <em>et al.</em>, 2020</a>).</p>
	<p>Data was processed using sanger-tol/genomeassembly v${workflow.manifest.version} ${doi_text} of the sanger-tol collection of workflows, created using nf-core (<a href="https://doi.org/10.1038/s41587-020-0439-x">Ewels <em>et al.</em>, 2020</a>).</p>

	"description": "Path to comma-separated file containing information about the samples in the experiment.",
	"description": "Path to YAML file containing information about the samples in the experiment.",

Template merge #15

Template merge #15

Conversation

ksenia-krasheninnikova commented Jun 15, 2023

PR checklist

priyanka-surana left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

priyanka-surana Jul 18, 2023 • edited Loading

Choose a reason for hiding this comment

priyanka-surana Jul 18, 2023 • edited Loading

Choose a reason for hiding this comment

github-actions bot commented Aug 16, 2023 • edited Loading

nf-core lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

priyanka-surana left a comment • edited Loading

Choose a reason for hiding this comment

priyanka-surana Sep 6, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

muffato commented Sep 8, 2023

priyanka-surana left a comment

Choose a reason for hiding this comment

priyanka-surana left a comment •

edited

Loading

priyanka-surana Jul 18, 2023 •

edited

Loading

priyanka-surana Jul 18, 2023 •

edited

Loading

github-actions bot commented Aug 16, 2023 •

edited

Loading

`nf-core lint` overall result: Passed ✅ ⚠️

priyanka-surana left a comment •

edited

Loading

priyanka-surana Sep 6, 2023 •

edited

Loading