TASK-6324 - Simplify QC analysis by launching a single job #2469

jtarraga · 2024-06-19T09:36:26Z

TASK-6324 Simplify QC analysis by launching a single job:

Alignment QC analysis
Sample QC analysis

…b that executes sequentially samtools, plot-bamstats and fastqc, #TASK-6325, #TASK-6324 On branch TASK-6324 Changes to be committed: modified: opencga-analysis/src/main/java/org/opencb/opencga/analysis/alignment/qc/AlignmentFastQcMetricsAnalysis.java modified: opencga-analysis/src/main/java/org/opencb/opencga/analysis/alignment/qc/AlignmentQcAnalysis.java modified: opencga-analysis/src/main/java/org/opencb/opencga/analysis/tools/OpenCgaTool.java modified: opencga-analysis/src/main/java/org/opencb/opencga/analysis/wrappers/executors/DockerWrapperAnalysisExecutor.java modified: opencga-analysis/src/main/java/org/opencb/opencga/analysis/wrappers/samtools/SamtoolsWrapperAnalysisExecutor.java modified: opencga-analysis/src/test/java/org/opencb/opencga/analysis/alignment/AlignmentAnalysisTest.java modified: opencga-core/src/main/java/org/opencb/opencga/core/tools/OpenCgaToolExecutor.java modified: opencga-core/src/main/java/org/opencb/opencga/core/tools/result/ExecutionResultManager.java

…TASK-6324 On branch TASK-6324 Changes to be committed: new file: opencga-app/app/analysis/genome-plot/circos.R

…nused parameters, #TASK-6325, #TASK-6324 On branch TASK-6324 Changes to be committed: modified: opencga-analysis/src/main/java/org/opencb/opencga/analysis/wrappers/fastqc/FastqcWrapperAnalysis.java modified: opencga-analysis/src/main/java/org/opencb/opencga/analysis/wrappers/fastqc/FastqcWrapperAnalysisExecutor.java

… unused parameters, #TASK-6325, #TASK-6324 On branch TASK-6324 Changes to be committed: modified: opencga-analysis/src/main/java/org/opencb/opencga/analysis/wrappers/samtools/SamtoolsWrapperAnalysis.java modified: opencga-analysis/src/main/java/org/opencb/opencga/analysis/wrappers/samtools/SamtoolsWrapperAnalysisExecutor.java

…tools stats/flagstats, fastqc), #TASK-6325, #TASK-6324 On branch TASK-6324 Changes to be committed: modified: opencga-analysis/src/main/java/org/opencb/opencga/analysis/alignment/qc/AlignmentQcAnalysis.java

…r sonnar issues, #TASK-6325, #TASK-6324 On branch TASK-6324 Changes to be committed: modified: opencga-analysis/src/main/java/org/opencb/opencga/analysis/alignment/qc/AlignmentQcAnalysis.java

On branch TASK-6324 Changes to be committed: modified: opencga-analysis/src/test/java/org/opencb/opencga/analysis/alignment/AlignmentAnalysisTest.java

…pos and sonnar issues, #TASK-6326, #TASK-6324 On branch TASK-6324 Changes to be committed: modified: opencga-analysis/src/main/java/org/opencb/opencga/analysis/variant/mutationalSignature/MutationalSignatureAnalysis.java modified: opencga-analysis/src/main/java/org/opencb/opencga/analysis/variant/mutationalSignature/MutationalSignatureLocalAnalysisExecutor.java modified: opencga-core/src/main/java/org/opencb/opencga/core/tools/variant/MutationalSignatureAnalysisExecutor.java

On branch TASK-6324 Changes to be committed: modified: opencga-analysis/src/main/java/org/opencb/opencga/analysis/variant/stats/SampleVariantStatsAnalysis.java modified: opencga-analysis/src/main/java/org/opencb/opencga/analysis/variant/stats/SampleVariantStatsLocalAnalysisExecutor.java modified: opencga-core/src/main/java/org/opencb/opencga/core/tools/variant/SampleVariantStatsAnalysisExecutor.java

…lot analysis), fix some typos and sonnar issues, #TASK-6326, #TASK-6324 On branch TASK-6324 Changes to be committed: modified: opencga-analysis/src/main/java/org/opencb/opencga/analysis/variant/genomePlot/GenomePlotAnalysis.java modified: opencga-analysis/src/main/java/org/opencb/opencga/analysis/variant/genomePlot/GenomePlotLocalAnalysisExecutor.java modified: opencga-core/src/main/java/org/opencb/opencga/core/tools/variant/GenomePlotAnalysisExecutor.java

On branch TASK-6324 Changes to be committed: modified: opencga-analysis/src/main/java/org/opencb/opencga/analysis/tools/OpenCgaTool.java

…e QC step, #TASK-6326, #TASK-6324 On branch TASK-6324 Changes to be committed: modified: opencga-analysis/src/main/java/org/opencb/opencga/analysis/sample/qc/SampleQcAnalysis.java

…ASK-6324 On branch TASK-6324 Changes to be committed: deleted: opencga-analysis/src/main/R/genome-plot/circos.R deleted: opencga-analysis/src/main/R/mutational-signature/mutational-signature.r modified: opencga-analysis/src/main/java/org/opencb/opencga/analysis/alignment/qc/AlignmentQcAnalysis.java modified: opencga-analysis/src/main/java/org/opencb/opencga/analysis/tools/OpenCgaTool.java

…lysis folder, #TASK-6326, #TASK-6324 On branch TASK-6324 Changes to be committed: modified: opencga-analysis/src/main/java/org/opencb/opencga/analysis/variant/circos/CircosLocalAnalysisExecutor.java modified: opencga-analysis/src/main/java/org/opencb/opencga/analysis/variant/genomePlot/GenomePlotLocalAnalysisExecutor.java

On branch TASK-6324 Changes to be committed: modified: opencga-analysis/src/test/java/org/opencb/opencga/analysis/variant/OpenCGATestExternalResource.java modified: opencga-analysis/src/test/java/org/opencb/opencga/analysis/variant/VariantAnalysisTest.java deleted: opencga-storage/opencga-storage-core/src/test/resources/AR2.10039966-01T.copynumber.caveman.vcf.gz deleted: opencga-storage/opencga-storage-core/src/test/resources/AR2.10039966-01T_vs_AR2.10039966-01G.annot.brass.vcf.gz deleted: opencga-storage/opencga-storage-core/src/test/resources/AR2.10039966-01T_vs_AR2.10039966-01G.annot.pindel.vcf.gz new file: opencga-storage/opencga-storage-core/src/test/resources/cancer-cnvs.vcf.gz new file: opencga-storage/opencga-storage-core/src/test/resources/cancer-indels.vcf.gz new file: opencga-storage/opencga-storage-core/src/test/resources/cancer-rearrs.vcf.gz new file: opencga-storage/opencga-storage-core/src/test/resources/cancer-snvs.vcf.gz new file: opencga-storage/opencga-storage-core/src/test/resources/genome-plot-config.json

...ava/org/opencb/opencga/analysis/variant/mutationalSignature/MutationalSignatureAnalysis.java

-        logger.info("Signagture fit max. rare sigs.: {}", signatureParams.getFitMaxRareSigs());
-        logger.info("Signagture fit signatures file: {}", signaturesFile);
-        logger.info("Signagture fit rare signatures file: {}", rareSignaturesFile);
+        logger.info("Signature id: {}", signatureParams.getId());


...ava/org/opencb/opencga/analysis/variant/mutationalSignature/MutationalSignatureAnalysis.java

-        logger.info("Signagture fit signatures file: {}", signaturesFile);
-        logger.info("Signagture fit rare signatures file: {}", rareSignaturesFile);
+        logger.info("Signature id: {}", signatureParams.getId());
+        logger.info("Signature description: {}", signatureParams.getDescription());


...cga-analysis/src/main/java/org/opencb/opencga/analysis/alignment/qc/AlignmentQcAnalysis.java

j-coll · 2024-06-19T13:01:15Z

...cga-analysis/src/main/java/org/opencb/opencga/analysis/alignment/qc/AlignmentQcAnalysis.java

            }
+        } else {
+            throw new ToolException("Something wrong happened running Samtools flagstat analysis");


Your future self will appreciate if you add some extra information here. Like the number of lines, and/or the actual first line

...cga-analysis/src/main/java/org/opencb/opencga/analysis/alignment/qc/AlignmentQcAnalysis.java

j-coll · 2024-06-19T13:05:44Z

opencga-analysis/src/main/java/org/opencb/opencga/analysis/sample/qc/SampleQcAnalysis.java

                                    + "' is already used");
                        }
                    }
                }
            }
+        } else {
+            String msg = "Skipping sample variant stats analysis by user";
+            addWarning(msg);


Is this even a warning, or should it be just an info event?

hard to say!

opencga-analysis/src/main/java/org/opencb/opencga/analysis/tools/OpenCgaTool.java

j-coll · 2024-06-19T13:13:38Z

opencga-analysis/src/main/java/org/opencb/opencga/analysis/tools/OpenCgaTool.java

@@ -495,6 +499,20 @@ protected final void addAttribute(String key, Object value) throws ToolException
        erm.addAttribute(key, value);
    }

+    protected final void addStepAttributes(ExecutionResult executionResult) throws ToolException {


You might want to include more information from the nested ExecutionResult, like events (maybe that's the only interesting thing)

do you mean to add the "nested" ExecutionResult events into the list of "parent" Execution events?,

Yes, something like that. Otherwise, they're lost (not technically, as you can always read the nested job result file, but they'd be quite hidden).

opencga-core/src/main/java/org/opencb/opencga/core/tools/result/ExecutionResultManager.java

… #TASK-6324 On branch TASK-6324 Changes to be committed: modified: opencga-analysis/src/main/java/org/opencb/opencga/analysis/alignment/qc/AlignmentQcAnalysis.java

…-6324 On branch TASK-6324 Changes to be committed: modified: opencga-analysis/src/main/java/org/opencb/opencga/analysis/sample/qc/SampleQcAnalysis.java modified: opencga-analysis/src/main/java/org/opencb/opencga/analysis/tools/OpenCgaTool.java modified: opencga-core/src/main/java/org/opencb/opencga/core/tools/result/ExecutionResultManager.java

…as wrong running dockers (samtools stats/flagstats and fastqc), #TASK-6325, #TASK-6324 On branch TASK-6324 Changes to be committed: modified: opencga-analysis/src/main/java/org/opencb/opencga/analysis/alignment/qc/AlignmentFastQcMetricsAnalysis.java modified: opencga-analysis/src/main/java/org/opencb/opencga/analysis/alignment/qc/AlignmentQcAnalysis.java modified: opencga-analysis/src/main/java/org/opencb/opencga/analysis/wrappers/executors/DockerWrapperAnalysisExecutor.java modified: opencga-analysis/src/test/java/org/opencb/opencga/analysis/alignment/AlignmentAnalysisTest.java

j-coll · 2024-06-20T14:18:55Z

...cga-analysis/src/main/java/org/opencb/opencga/analysis/alignment/qc/AlignmentQcAnalysis.java

        return steps;
    }

    @Override
    protected void run() throws ToolException {
        // Create the tool runner
-        toolRunner = new ToolRunner(opencgaHome, catalogManager, StorageEngineFactory.get(variantStorageManager.getStorageConfiguration()));
+        toolRunner = new ToolRunner(getOpencgaHome().toString(), catalogManager,


You can better use the constructor

public ToolRunner(String opencgaHome, CatalogManager catalogManager, VariantStorageManager variantStorageManager)

instead of

ToolRunner(String opencgaHome, CatalogManager catalogManager, StorageEngineFactory storageEngineFactory)

… attributes, #TASK-6325, #TASK-6324 On branch TASK-6324 Changes to be committed: modified: opencga-analysis/src/main/java/org/opencb/opencga/analysis/alignment/qc/AlignmentQcAnalysis.java modified: opencga-analysis/src/main/java/org/opencb/opencga/analysis/sample/qc/SampleQcAnalysis.java modified: opencga-analysis/src/main/java/org/opencb/opencga/analysis/tools/OpenCgaTool.java

On branch TASK-6324 Changes to be committed: modified: opencga-analysis/src/main/java/org/opencb/opencga/analysis/alignment/qc/AlignmentQcAnalysis.java

On branch TASK-6324 Changes to be committed: modified: opencga-analysis/src/main/java/org/opencb/opencga/analysis/sample/qc/SampleQcAnalysis.java

The base branch was changed.

halender · 2024-07-22T10:17:13Z

Task linked: TASK-6324 Simplify QC analysis by launching a single job

jtarraga added 16 commits May 31, 2024 13:38

Merge branch 'develop' into TASK-6324

483c1be

app: fix circos.R and move to app/analysis/genome-plot, #TASK-6326, #…

b0aea7b

…TASK-6324 On branch TASK-6324 Changes to be committed: new file: opencga-app/app/analysis/genome-plot/circos.R

analysis: use ToolRunner to execute each alignment QC step (i.e.: sam…

512c17d

…tools stats/flagstats, fastqc), #TASK-6325, #TASK-6324 On branch TASK-6324 Changes to be committed: modified: opencga-analysis/src/main/java/org/opencb/opencga/analysis/alignment/qc/AlignmentQcAnalysis.java

analysis: support the flag overwrite, and forward excetions, and mino…

65a1bdd

…r sonnar issues, #TASK-6325, #TASK-6324 On branch TASK-6324 Changes to be committed: modified: opencga-analysis/src/main/java/org/opencb/opencga/analysis/alignment/qc/AlignmentQcAnalysis.java

test: add JUnit tests for alignment QC, #TASK-6325, #TASK-6324

55c81ff

On branch TASK-6324 Changes to be committed: modified: opencga-analysis/src/test/java/org/opencb/opencga/analysis/alignment/AlignmentAnalysisTest.java

analysis: add the method addStepAttributes, #TASK-6326, #TASK-6324

39c2f6a

On branch TASK-6324 Changes to be committed: modified: opencga-analysis/src/main/java/org/opencb/opencga/analysis/tools/OpenCgaTool.java

analysis: use the ToolRunner instead of launching jobs for each sampl…

7e3ca47

…e QC step, #TASK-6326, #TASK-6324 On branch TASK-6324 Changes to be committed: modified: opencga-analysis/src/main/java/org/opencb/opencga/analysis/sample/qc/SampleQcAnalysis.java

github-advanced-security bot found potential problems Jun 19, 2024

View reviewed changes

jtarraga requested a review from j-coll June 19, 2024 09:36

Merge branch 'develop' into TASK-6324

11a8653

j-coll requested changes Jun 19, 2024

View reviewed changes

jtarraga added 3 commits June 19, 2024 17:11

analysis: add a new tool step to update the alignment QC, #TASK-6325,…

9f6913b

… #TASK-6324 On branch TASK-6324 Changes to be committed: modified: opencga-analysis/src/main/java/org/opencb/opencga/analysis/alignment/qc/AlignmentQcAnalysis.java

jtarraga requested a review from j-coll June 20, 2024 07:53

j-coll reviewed Jun 20, 2024

View reviewed changes

jtarraga requested a review from j-coll June 21, 2024 09:40

jtarraga added 2 commits June 21, 2024 11:43

analysis: use a simpler ToolRunner constructor, #TASK-6325, #TASK-6324

0531c48

On branch TASK-6324 Changes to be committed: modified: opencga-analysis/src/main/java/org/opencb/opencga/analysis/alignment/qc/AlignmentQcAnalysis.java

analysis: use a simpler ToolRunner constructor, #TASK-6326, #TASK-6324

a53bd04

On branch TASK-6324 Changes to be committed: modified: opencga-analysis/src/main/java/org/opencb/opencga/analysis/sample/qc/SampleQcAnalysis.java

jtarraga changed the title ~~Simplify QC analysis by launching a single job~~ TASK-6324 Simplify QC analysis by launching a single job Jun 21, 2024

j-coll previously approved these changes Jun 25, 2024

View reviewed changes

j-coll changed the title ~~TASK-6324 Simplify QC analysis by launching a single job~~ TASK-6324 - Simplify QC analysis by launching a single job Jun 25, 2024

Merge branch 'release-3.2.x' into TASK-6324

f77ca2e

jtarraga changed the base branch from develop to release-3.2.x July 22, 2024 10:17

Merge branch 'release-3.2.x' into TASK-6324

8666ba9

juanfeSanahuja self-requested a review August 14, 2024 16:48

juanfeSanahuja approved these changes Aug 14, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TASK-6324 - Simplify QC analysis by launching a single job #2469

TASK-6324 - Simplify QC analysis by launching a single job #2469

jtarraga commented Jun 19, 2024 •

edited

Loading

j-coll Jun 19, 2024

j-coll Jun 19, 2024

jtarraga Jun 19, 2024

j-coll Jun 19, 2024

jtarraga Jun 19, 2024

j-coll Jun 20, 2024

j-coll Jun 20, 2024

halender commented Jul 22, 2024

TASK-6324 - Simplify QC analysis by launching a single job #2469

Are you sure you want to change the base?

TASK-6324 - Simplify QC analysis by launching a single job #2469

Conversation

jtarraga commented Jun 19, 2024 • edited Loading

j-coll Jun 19, 2024

Choose a reason for hiding this comment

j-coll Jun 19, 2024

Choose a reason for hiding this comment

jtarraga Jun 19, 2024

Choose a reason for hiding this comment

j-coll Jun 19, 2024

Choose a reason for hiding this comment

jtarraga Jun 19, 2024

Choose a reason for hiding this comment

j-coll Jun 20, 2024

Choose a reason for hiding this comment

j-coll Jun 20, 2024

Choose a reason for hiding this comment

halender commented Jul 22, 2024

jtarraga commented Jun 19, 2024 •

edited

Loading