Result Retrieval from FastMatch Pipeline #3

emarinier · 2024-12-11T18:12:00Z

Description:

As a Data Analyst, I want the FastMatch pipeline to return a file with results that meet my specified similarity threshold, and include the similarity scores for each match, so that I can effectively analyze and compare the results.

Acceptance criteria:

Acceptance Criteria:

The FastMatch pipeline generates a linelist excel file with query results.
- Columns: Query ID, Query Sample Name, Reference ID, Reference Sample Name, Distance, Reference Metadata 1, Reference Metadata 2, ...
The results in the file meet the user-specified similarity threshold.
The file includes the similarity scores for each match.
The user can analyze and compare the matches based on the provided similarity scores.

samplesheet.csv

sample,sample_name,mlst_alleles,metadata_1,metadata_2,metadata_3,metadata_4,metadata_5,metadata_6,metadata_7,metadata_8
sample1,sample_name1,https://raw.githubusercontent.com/phac-nml/gasclustering/dev/tests/data/reports/sample1.mlst.json,1.1,1.2,1.3,1.4,1.5,1.6,1.7,1.8
sample2,sample_name2,https://raw.githubusercontent.com/phac-nml/gasclustering/dev/tests/data/reports/sample2.mlst.json,2.1,2.2,2.3,2.4,2.5,2.6,2.7,2.8
sample3,sample_name3,https://raw.githubusercontent.com/phac-nml/gasclustering/dev/tests/data/reports/sample3.mlst.json,3.1,3.2,3.3,3.4,3.5,3.6,3.7,3.8

distances/profile_dists.results.text

query_id	ref_id	dist
sample_name1	sample_name1	0
sample_name1	sample_name2	0
sample_name1	sample_name3	1
sample_name2	sample_name1	0
sample_name2	sample_name2	0
sample_name2	sample_name3	1
sample_name3	sample_name3	0
sample_name3	sample_name1	1
sample_name3	sample_name2	1

append/distances_and_metadata.tsv

Query ID	Query Sample Name	Reference ID	Reference Sample Name	Distance	metadata_1	metadata_2	metadata_3	metadata_4	metadata_5	metadata_6	metadata_7	metadata_8
sample1	sample_name1	sample1	sample_name1	0	1.1	1.2	1.3	1.4	1.5	1.6	1.7	1.8
sample1	sample_name1	sample2	sample_name2	0	2.1	2.2	2.3	2.4	2.5	2.6	2.7	2.8
sample1	sample_name1	sample3	sample_name3	1	3.1	3.2	3.3	3.4	3.5	3.6	3.7	3.8
sample2	sample_name2	sample1	sample_name1	0	1.1	1.2	1.3	1.4	1.5	1.6	1.7	1.8
sample2	sample_name2	sample2	sample_name2	0	2.1	2.2	2.3	2.4	2.5	2.6	2.7	2.8
sample2	sample_name2	sample3	sample_name3	1	3.1	3.2	3.3	3.4	3.5	3.6	3.7	3.8
sample3	sample_name3	sample3	sample_name3	0	3.1	3.2	3.3	3.4	3.5	3.6	3.7	3.8
sample3	sample_name3	sample1	sample_name1	1	1.1	1.2	1.3	1.4	1.5	1.6	1.7	1.8
sample3	sample_name3	sample2	sample_name2	1	2.1	2.2	2.3	2.4	2.5	2.6	2.7	2.8

process/results.tsv

Query ID	Query Sample Name	Reference ID	Reference Sample Name	Distance	metadata_1	metadata_2	metadata_3	metadata_4	metadata_5	metadata_6	metadata_7	metadata_8
sample1	sample_name1	sample1	sample_name1	0	1.1	1.2	1.3	1.4	1.5	1.6	1.7	1.8
sample1	sample_name1	sample2	sample_name2	0	2.1	2.2	2.3	2.4	2.5	2.6	2.7	2.8
sample2	sample_name2	sample1	sample_name1	0	1.1	1.2	1.3	1.4	1.5	1.6	1.7	1.8
sample2	sample_name2	sample2	sample_name2	0	2.1	2.2	2.3	2.4	2.5	2.6	2.7	2.8
sample3	sample_name3	sample3	sample_name3	0	3.1	3.2	3.3	3.4	3.5	3.6	3.7	3.8

github-actions · 2024-12-11T18:13:42Z

`nf-core pipelines lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit 4c18be3

+| ✅ 145 tests passed       |+
#| ❔  28 tests were ignored |#
!| ❗   4 tests had warnings |!

❗ Test warnings:

files_exist - File not found: conf/igenomes_ignored.config
nextflow_config - nf-validation has been detected in the pipeline. Please migrate to nf-schema: https://nextflow-io.github.io/nf-schema/latest/migration_guide/
nextflow_config - Config manifest.version should end in dev: 0.1.0
schema_lint - Schema $id should be https://raw.githubusercontent.com/phac-nml/fastmatchirida/master/nextflow_schema.json
Found https://raw.githubusercontent.com/phac-nml/fastmatchirida/main/nextflow_schema.json

❔ Tests ignored:

files_exist - File is ignored: assets/nf-core-fastmatchirida_logo_light.png
files_exist - File is ignored: docs/images/nf-core-fastmatchirida_logo_light.png
files_exist - File is ignored: docs/images/nf-core-fastmatchirida_logo_dark.png
files_exist - File is ignored: .github/workflows/awstest.yml
files_exist - File is ignored: .github/workflows/awsfulltest.yml
files_exist - File is ignored: lib/Utils.groovy
files_exist - File is ignored: lib/WorkflowMain.groovy
files_exist - File is ignored: lib/NfcoreTemplate.groovy
files_exist - File is ignored: lib/Workflowfastmatchirida.groovy
nextflow_config - Config variable ignored: manifest.name
nextflow_config - Config variable ignored: manifest.homePage
nextflow_config - Config variable ignored: params.max_cpus
files_unchanged - File ignored due to lint config: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_unchanged - File ignored due to lint config: .github/CONTRIBUTING.md
files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/bug_report.yml
files_unchanged - File ignored due to lint config: .github/PULL_REQUEST_TEMPLATE.md
files_unchanged - File ignored due to lint config: .github/workflows/branch.yml
files_unchanged - File ignored due to lint config: assets/email_template.html
files_unchanged - File ignored due to lint config: assets/email_template.txt
files_unchanged - File ignored due to lint config: assets/sendmail_template.txt
files_unchanged - File does not exist: assets/nf-core-fastmatchirida_logo_light.png
files_unchanged - File does not exist: docs/images/nf-core-fastmatchirida_logo_light.png
files_unchanged - File does not exist: docs/images/nf-core-fastmatchirida_logo_dark.png
files_unchanged - File ignored due to lint config: docs/README.md
files_unchanged - File ignored due to lint config: .gitignore or .prettierignore
actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/fastmatchirida/fastmatchirida/.github/workflows/awstest.yml
actions_awsfulltest - actions_awsfulltest
pipeline_name_conventions - pipeline_name_conventions

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: conf/modules.config
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: conf/igenomes.config
files_exist - File found: modules.json
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: docs/images/nf-core-fastmatchirida_logo.png
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: lib/WorkflowFastmatchirida.groovy
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: Singularity
files_exist - File not found check: lib/nfcore_external_java_deps.jar
files_exist - File not found check: .travis.yml
nextflow_config - Found nf-validation plugin
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - nextflow.config contains configuration profile test
nextflow_config - Config default value correct: params.metadata_1_header= metadata_1
nextflow_config - Config default value correct: params.metadata_2_header= metadata_2
nextflow_config - Config default value correct: params.metadata_3_header= metadata_3
nextflow_config - Config default value correct: params.metadata_4_header= metadata_4
nextflow_config - Config default value correct: params.metadata_5_header= metadata_5
nextflow_config - Config default value correct: params.metadata_6_header= metadata_6
nextflow_config - Config default value correct: params.metadata_7_header= metadata_7
nextflow_config - Config default value correct: params.metadata_8_header= metadata_8
nextflow_config - Config default value correct: params.pd_distm= hamming
nextflow_config - Config default value correct: params.pd_missing_threshold= 1.0
nextflow_config - Config default value correct: params.pd_sample_quality_threshold= 1.0
nextflow_config - Config default value correct: params.pd_file_type= text
nextflow_config - Config default value correct: params.gm_thresholds= 10,5,0
nextflow_config - Config default value correct: params.gm_method= average
nextflow_config - Config default value correct: params.gm_delimiter= .
nextflow_config - Config default value correct: params.max_cpus= 4
nextflow_config - Config default value correct: params.max_memory= 2.GB
nextflow_config - Config default value correct: params.max_time= 1.h
nextflow_config - Config default value correct: params.publish_dir_mode= copy
nextflow_config - Config default value correct: params.validate_params= true
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - .github/workflows/linting.yml matches the template
actions_ci - '.github/workflows/ci.yml' is triggered on expected events
actions_ci - '.github/workflows/ci.yml' checks minimum NF version
readme - README Zenodo placeholder was replaced with DOI.
pipeline_todos - No TODO strings found
plugin_includes - No wrong validation plugin imports have been found
template_strings - Did not find any Jinja template strings (0 files)
schema_lint - Schema lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
multiqc_config - assets/multiqc_config.yml found and not ignored.
multiqc_config - assets/multiqc_config.yml contains report_section_order
multiqc_config - assets/multiqc_config.yml contains export_plots
multiqc_config - assets/multiqc_config.yml contains report_comment
multiqc_config - assets/multiqc_config.yml follows the ordering scheme of the minimally required plugins.
multiqc_config - assets/multiqc_config.yml contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'
base_config - conf/base.config found and not ignored.
base_config - CUSTOM_DUMPSOFTWAREVERSIONS found in conf/base.config and Nextflow scripts.
modules_config - conf/modules.config found and not ignored.
modules_config - INPUT_ASSURE found in conf/modules.config and Nextflow scripts.
modules_config - LOCIDEX_MERGE found in conf/modules.config and Nextflow scripts.
modules_config - PROFILE_DISTS found in conf/modules.config and Nextflow scripts.
modules_config - CUSTOM_DUMPSOFTWAREVERSIONS found in conf/modules.config and Nextflow scripts.
nfcore_yml - Repository type in .nf-core.yml is valid: pipeline
nfcore_yml - nf-core version in .nf-core.yml is set to the latest version: 3.0.1

Run details

nf-core/tools version 3.0.1
Run at 2024-12-12 17:59:32

apetkau

This looks amazing. Thanks so much @emarinier 😄

Some in-line comments below.

bin/process_output.py

workflows/fastmatchirida.nf

sgsutcliffe

Looks good to me! I didn't run it myself but I figure I can take a closer look when me merge all three stories.

apetkau

Everything looks great to me. Thanks so much Eric 😄

emarinier added 3 commits December 11, 2024 11:20

Very basic output framework.

4448ac1

pairwise output format, appending metadata

d579c35

Cleaning up, process_output.py outputs the input for now.

6ee6358

emarinier self-assigned this Dec 11, 2024

emarinier marked this pull request as draft December 11, 2024 18:12

emarinier added 6 commits December 11, 2024 14:44

Both sample names and sample/Irida IDs in output.

2f08dd1

Distance threshold.

2121cfd

Removing profile dist output format from schema.

4a2138c

threshold as a parameter

34b8373

Newline at end of file.

eb173c2

Updated script example.

8ade704

emarinier changed the title ~~WIP: Result Retrieval from FastMatch Pipeline~~ Result Retrieval from FastMatch Pipeline Dec 11, 2024

emarinier marked this pull request as ready for review December 11, 2024 22:34

emarinier added 2 commits December 11, 2024 16:38

Updating container

75013de

Updating test data after switching to pairwise output.

0699fca

emarinier requested review from apetkau and sgsutcliffe December 12, 2024 16:14

apetkau requested changes Dec 12, 2024

View reviewed changes

bin/process_output.py Outdated Show resolved Hide resolved

bin/process_output.py Outdated Show resolved Hide resolved

sgsutcliffe reviewed Dec 12, 2024

View reviewed changes

workflows/fastmatchirida.nf Show resolved Hide resolved

sgsutcliffe approved these changes Dec 12, 2024

View reviewed changes

xlsx, float threshold

4c18be3

apetkau approved these changes Dec 12, 2024

View reviewed changes

emarinier merged commit 70962d5 into dev Dec 12, 2024
4 checks passed

emarinier deleted the output-stub branch December 13, 2024 17:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Result Retrieval from FastMatch Pipeline #3

Result Retrieval from FastMatch Pipeline #3

emarinier commented Dec 11, 2024 •

edited

Loading

github-actions bot commented Dec 11, 2024 •

edited

Loading

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

apetkau left a comment

sgsutcliffe left a comment

apetkau left a comment

Result Retrieval from FastMatch Pipeline #3

Result Retrieval from FastMatch Pipeline #3

Conversation

emarinier commented Dec 11, 2024 • edited Loading

github-actions bot commented Dec 11, 2024 • edited Loading

nf-core pipelines lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

apetkau left a comment

Choose a reason for hiding this comment

sgsutcliffe left a comment

Choose a reason for hiding this comment

apetkau left a comment

Choose a reason for hiding this comment

emarinier commented Dec 11, 2024 •

edited

Loading

github-actions bot commented Dec 11, 2024 •

edited

Loading

`nf-core pipelines lint` overall result: Passed ✅ ⚠️