Add genotype filtering Terra workflow configs and documentation #695

mwalker174 · 2024-07-02T13:53:18Z

Adds JoinRawCalls, SVConcordance, and FilterGenotypes Terra workflow configurations. Note that the workflow numbering in Dockstore and Terra are not yet updated.
Updates genotype filtering documentation

Tested on our "bwa-melt" benchmarking workspace, which had a recent cleaned vcf to work from.

VJalili · 2024-07-08T15:33:57Z

README.md

Would it be helpful to add this documentation to the website, too?

Yes, we'll want everything on the website by the time we release the featured workspace. I think it makes sense to me to get the README and dashboard updated so we can update the template Terra workspace, then update the website after - does that sound good?

epiercehoffman

Thanks for writing all of this up. Mostly looks good, though I flagged a few places to fix. In addition, the pipeline diagram should be updated to include these workflows.

epiercehoffman · 2024-09-13T17:24:37Z

README.md

+    * [JoinRawCalls](#join-raw-calls) - Merges unfiltered calls across batches
+    * [SVConcordance](#svconcordance) - Calculates genotype concordance with raw calls
+    * [FilterGenotypes](#filter-genotypes) - Performs genotype filtering
+    * [AnnotateVcf](#annotate-vcf) - Functional and allele frequency annotation
    * [Module 09](#module09) - QC and Visualization


Should we delete this?

Yeah I will clean out the readme with other unrelated changes when I do a full readme->website transfer later.

README.md

epiercehoffman · 2024-09-13T17:28:48Z

README.md

+Computes genotype concordance metrics between all variants in the joint call set and raw calls.
+
+#### Prerequisites:
+* [MakeCohortVcf](#make-cohort-vcf)


We should probably update the README with the four subsections of MakeCohortVcf and change this prereq to CleanVcf. At the very least, the MakeCohortVcf section should describe the subworkflows.

You're right. I think this is a bit beyond the scope of this PR but I am going to create a new ticket for the website update and will mention this.

epiercehoffman · 2024-09-13T17:54:53Z

inputs/templates/terra_workspaces/cohort_mode/workflow_configurations/JoinRawCalls.json.tmpl

+  "JoinRawCalls.clustered_depth_vcfs" : "${this.clustered_depth_vcf}",
+  "JoinRawCalls.clustered_depth_vcf_indexes" : "${this.clustered_depth_vcf_index}",
+
+  "JoinRawCalls.clustered_manta_vcfs" : "${this.clustered_manta_vcf}",
+  "JoinRawCalls.clustered_manta_vcf_indexes" : "${this.clustered_manta_vcf_index}",
+
+  "JoinRawCalls.clustered_wham_vcfs" : "${this.clustered_wham_vcf}",
+  "JoinRawCalls.clustered_wham_vcf_indexes" : "${this.clustered_wham_vcf_index}",
+
+  "JoinRawCalls.clustered_melt_vcfs" : "${this.clustered_melt_vcf}",
+  "JoinRawCalls.clustered_melt_vcf_indexes" : "${this.clustered_melt_vcf_index}",


Suggested change

"JoinRawCalls.clustered_depth_vcfs" : "${this.clustered_depth_vcf}",

"JoinRawCalls.clustered_depth_vcf_indexes" : "${this.clustered_depth_vcf_index}",

"JoinRawCalls.clustered_manta_vcfs" : "${this.clustered_manta_vcf}",

"JoinRawCalls.clustered_manta_vcf_indexes" : "${this.clustered_manta_vcf_index}",

"JoinRawCalls.clustered_wham_vcfs" : "${this.clustered_wham_vcf}",

"JoinRawCalls.clustered_wham_vcf_indexes" : "${this.clustered_wham_vcf_index}",

"JoinRawCalls.clustered_melt_vcfs" : "${this.clustered_melt_vcf}",

"JoinRawCalls.clustered_melt_vcf_indexes" : "${this.clustered_melt_vcf_index}",

"JoinRawCalls.clustered_depth_vcfs" : "${this.sample_sets.clustered_depth_vcf}",

"JoinRawCalls.clustered_depth_vcf_indexes" : "${this.sample_sets.clustered_depth_vcf_index}",

"JoinRawCalls.clustered_manta_vcfs" : "${this.sample_sets.clustered_manta_vcf}",

"JoinRawCalls.clustered_manta_vcf_indexes" : "${this.sample_sets.clustered_manta_vcf_index}",

"JoinRawCalls.clustered_wham_vcfs" : "${this.sample_sets.clustered_wham_vcf}",

"JoinRawCalls.clustered_wham_vcf_indexes" : "${this.sample_sets.clustered_wham_vcf_index}",

"JoinRawCalls.clustered_melt_vcfs" : "${this.sample_sets.clustered_melt_vcf}",

"JoinRawCalls.clustered_melt_vcf_indexes" : "${this.sample_sets.clustered_melt_vcf_index}",

Can you cross-reference these JSONs with the ones in the existing Terra workspace to avoid issues like this? Or were they all configured for a single batch? In that case AoU Phase 2 could work for checking, though I hope I got them all

Also, probably should add scramble VCF inputs and double check that the workflow runs ok when an empty array is provided for one set of caller VCFs

Good catch. This looks consistent with the AoU workspace and I've changed the default to Scramble even though that won't be default behavior until #722 goes in. I think this has been running with empty Scramble inputs so that shouldn't be a problem.

inputs/templates/terra_workspaces/cohort_mode/workflow_configurations/SVConcordance.json.tmpl

epiercehoffman · 2024-09-13T17:55:36Z

inputs/templates/terra_workspaces/cohort_mode/workflow_configurations/FilterGenotypes.json.tmpl

@@ -0,0 +1,41 @@
+{
+  "FilterGenotypes.vcf": "${this.concordance_vcf}",
+  "FilterGenotypes.output_prefix": "${this.sample_set_id}",


Suggested change

"FilterGenotypes.output_prefix": "${this.sample_set_id}",

"FilterGenotypes.output_prefix": "${this.sample_set_set_id}",

epiercehoffman · 2024-09-13T17:57:31Z

inputs/templates/terra_workspaces/cohort_mode/workspace.tsv.tmpl

@@ -5,6 +5,7 @@ gatk_docker	{{ dockers.gatk_docker }}
 gatk_docker_pesr_override	{{ dockers.gatk_docker_pesr_override }}
 gcnv_gatk_docker	{{ dockers.gatk_docker }}
 genomes_in_the_cloud_docker	{{ dockers.genomes_in_the_cloud_docker }}
+gq_recalibrator_docker	{{ dockers.gq_recalibrator_docker }}


Can we get this merged with the other GATK docker soon?

Unlikely since this tool is still on a branch

inputs/templates/test/FilterGenotypes/FilterGenotypes.fixed_cutoffs.json.tmpl

Typo Typo 2 Add to overview Round out readme Typo 3 Add detail Minor changes Fix workspace table Update inputs Fix templates

epiercehoffman

Thanks for addressing my comments! This is looking good. We should also get these workflows added to the dockstore.yml and the template workspace - I can also do that in my upcoming ManualReview PR though.

epiercehoffman · 2024-10-18T19:24:50Z

README.md

@@ -541,7 +541,7 @@ See the SV "Genotype Filter" section on page 34 of the [All of Us Genomic Qualit

 All valid genotypes are annotated with a "scaled logit" (SL) score, which is rescaled to non-negative adjusted GQs on [1, 99]. Note that the rescaled GQs should *not* be interpreted as probabilities. Original genotype qualities are retained in the OGQ field. 

-A more positive SL score indicates higher probability of correctness of the given genotype. Genotypes are therefore filtered using SL thresholds that depend on SV type and size. This workflow also generates QC plots using the [MainVcfQc](https://github.com/broadinstitute/gatk-sv/blob/main/wdl/MainVcfQc.wdl) workflow to review call set quality (see below for recommended practices). 
+A more positive SL score indicates higher probability that the give genotype is not homozygous for the reference allele. Genotypes are therefore filtered using SL thresholds that depend on SV type and size. This workflow also generates QC plots using the [MainVcfQc](https://github.com/broadinstitute/gatk-sv/blob/main/wdl/MainVcfQc.wdl) workflow to review call set quality (see below for recommended practices). 


Suggested change

A more positive SL score indicates higher probability that the give genotype is not homozygous for the reference allele. Genotypes are therefore filtered using SL thresholds that depend on SV type and size. This workflow also generates QC plots using the [MainVcfQc](https://github.com/broadinstitute/gatk-sv/blob/main/wdl/MainVcfQc.wdl) workflow to review call set quality (see below for recommended practices).

A more positive SL score indicates higher probability that the given genotype is not homozygous for the reference allele. Genotypes are therefore filtered using SL thresholds that depend on SV type and size. This workflow also generates QC plots using the [MainVcfQc](https://github.com/broadinstitute/gatk-sv/blob/main/wdl/MainVcfQc.wdl) workflow to review call set quality (see below for recommended practices).

mwalker174 requested a review from epiercehoffman July 2, 2024 13:53

VJalili reviewed Jul 8, 2024

View reviewed changes

mwalker174 mentioned this pull request Aug 20, 2024

I want to combine data using MakeCohortVcf after processing 500 samples multiple times in cohort mode #715

Open

epiercehoffman reviewed Sep 13, 2024

View reviewed changes

mwalker174 added 2 commits September 30, 2024 10:10

Add filtering documentation

d63d082

Typo Typo 2 Add to overview Round out readme Typo 3 Add detail Minor changes Fix workspace table Update inputs Fix templates

Fix FilterGenotypes config template

ad9360a

mwalker174 mentioned this pull request Sep 30, 2024

Consolidate documentation #730

Open

Reviewer comments

05b8025

mwalker174 force-pushed the mw_terra_genotype_filtering branch from 7237b4a to 05b8025 Compare September 30, 2024 15:03

epiercehoffman approved these changes Oct 18, 2024

View reviewed changes

Typo

9482cb3

mwalker174 merged commit c844d78 into main Oct 21, 2024
5 checks passed

mwalker174 deleted the mw_terra_genotype_filtering branch October 21, 2024 15:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add genotype filtering Terra workflow configs and documentation #695

Add genotype filtering Terra workflow configs and documentation #695

mwalker174 commented Jul 2, 2024

VJalili Jul 8, 2024

epiercehoffman Sep 13, 2024

epiercehoffman left a comment

epiercehoffman Sep 13, 2024

mwalker174 Sep 30, 2024

epiercehoffman Sep 13, 2024

mwalker174 Sep 30, 2024

epiercehoffman Sep 13, 2024 •

edited

Loading

mwalker174 Sep 30, 2024

epiercehoffman Sep 13, 2024

epiercehoffman Sep 13, 2024

mwalker174 Oct 9, 2024

epiercehoffman left a comment

epiercehoffman Oct 18, 2024

	"FilterGenotypes.output_prefix": "${this.sample_set_id}",
	"FilterGenotypes.output_prefix": "${this.sample_set_set_id}",

Add genotype filtering Terra workflow configs and documentation #695

Add genotype filtering Terra workflow configs and documentation #695

Conversation

mwalker174 commented Jul 2, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

epiercehoffman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

epiercehoffman Sep 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

epiercehoffman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

epiercehoffman Sep 13, 2024 •

edited

Loading