Adding single-read functionality to RAW and CLEAN #80

simonleandergrimm · 2024-10-28T16:45:36Z

This PR adds support for single-read (single-end) sequencing data to the RAW and CLEAN stages of the pipeline while maintaining existing paired-end functionality. This allows the pipeline to process both single-end and paired-end sequencing data using the same workflow infrastructure.

Key Changes

Amended generate_samplesheet.sh so it can also take in single-read data.
Added the run_dev_se.nf workflow, which will be the workflow in which single-read functionality is added up until all steps of the pipeline have single-read functionality. At that point, we can replace run.nf with run_dev_se.nf.
Added read_type parameter ("single_end" or "paired_end") in run_dev_se.config to control pipeline behavior
Split FASTP process into FASTP_SINGLE and FASTP_PAIRED variants
Split TRUNCATE_CONCAT process into _SINGLE and _PAIRED variants
Edited RAW, CLEAN, QC, and HV_SCREEN subworkflows to either take in the single_end or paired_end version of processes.
- Edited HV_SCREEN only because it would otherwise fail due to not identifying the fastp process correctly.
Updated bin/summarize-multiqc-single.R so it takes in a read-type variable, which triggers if/else branches throughout the script to change data processing accordingly.

Testing

I added test directories with example data for both single and paired-end cases

test-single-read/ - Contains single-end test data and configuration
test-paired-end/ - Contains paired-end test data and configuration

I validated the pipeline changes in this notebook: https://data.securebio.org/simons-notebook/posts/2024-10-24-mgs-single-read-eval/

… for single read analyses.

…-multiqc-paired.R

…on. Renamed Multiqc to not be confusing regardings its naming as "Single"

…d version

… or paired-end read version of fastp

…read or paired-end read version of fastp

… paired-end read version of SUMMARIZE_MULTIQC

…for paired-end vs single read runs.

…n outputs

…d paired end read runs

…on/*/raw

…book/notebooks/2024-10-17_crits-christoph-2-4-0.html to create analyses of single and paired-end read data.

…samples being paired-end or not.

… pair information for single read data. Also dropped some code which combines values across read pairs, for single read data. I dropped the renaming of tab_tsv to tab_tsv_2 for paired end data, so I didn't have to create two different versions of the combine step at the end of the subscript. ``` tab <- tab_json %>% inner_join(tab_tsv, by="sample") ```

…ead data, as I instead amended the existing script to be able to handle both single read paired end data.

…ad version. Renamed Multiqc to not be confusing regardings its naming as "Single"" This reverts commit 01ea0c5.

…rizeMultiqc" This reverts commit ad8faf9.

harmonbhasin

Looks good to me. I think this is ready for Will.

… single-read-raw-clean

…ysing read pairs!

harmonbhasin

Left some feedback, looking overall pretty good.

.github/workflows/end-to-end.yml

harmonbhasin · 2024-12-09T15:27:24Z

test-data/nextflow.config

@@ -7,12 +7,15 @@ params {

    // Directories
    base_dir = "s3://nao-mgs-wb/test-batch" // Parent for working and output directories (can be S3)
-    ref_dir = "s3://nao-mgs-wb/index-20241113/output" // Reference/index directory (generated by index workflow)
+ref_dir = "s3://nao-mgs-wb/index-20241113/output" // Reference/index directory (generated by index workflow)


outdated index

waiting for Will to create new index.

test-data/nextflow.config

workflows/run_dev_se.nf

modules/local/fastp/main.nf

harmonbhasin

Make this update, then this looks good to me.

workflows/run_dev_se.nf

simonleandergrimm added 30 commits October 21, 2024 19:19

Adding single read option to raw/main.nf

15354f6

Adding WIP version of run.nf to enable testing raw and clean versions…

ad2115d

… for single read analyses.

Created separate versions of summarize-multiqc-single.R and summarize…

03ee37a

…-multiqc-paired.R

Split processes in fastp to a single read and paired-end read version.

b517340

Split processes in MultiQC to a single read and paired-end read versi…

01ea0c5

…on. Renamed Multiqc to not be confusing regardings its naming as "Single"

Deleted summarizeMultiqcSingle, which was superseded by summarizeMultiqc

ad8faf9

Split processes in truncateConcat to a single read and paired-end rea…

ef0e9c8

…d version

Created a single_end if clause in Clean to either use the single read…

2535ccd

… or paired-end read version of fastp

Created a single_end if clause in hv_screen to either use the single …

cbcb109

…read or paired-end read version of fastp

Created a single_end if clause in qc to either use the single read or…

c7f8c83

… paired-end read version of SUMMARIZE_MULTIQC

Renamed test dir to test-paired-end. Added clause in nextflow.config …

ff0a8be

…for paired-end vs single read runs.

Edited gitignore to leave out test-paired-end and test-single-read ru…

6048dd3

…n outputs

Fixed name of test-single-end dir to test-single-read

92270e5

Created a version of test dir that allows the run of single-read data.

b13ac94

Added script to quickly download the s3 output of test single read an…

dff2302

…d paired end read runs

Added nextflow config for test paired and test single read.

64bb7f4

Fixed if clause in main.nf

5bd1aec

Updated gen samplesheet scripts to pull in data from s3://nao-mgs-sim…

c8fd3ac

…on/*/raw

Updated gitignore

578fde0

Activated CLEAN subworkflow in run.nf

59218b9

Starting to adapt Will's https://data.securebio.org/wills-public-note…

fd9dc1e

…book/notebooks/2024-10-17_crits-christoph-2-4-0.html to create analyses of single and paired-end read data.

Adding ignoring mgs-results to gitignore

81ff0ba

Adding Will's auxiliary scripts to run his quarto notebooks.

590b2c3

Merge branch 'master' into single-read-raw

6a650b4

Amended qmd somewhat so data imports work.

9f1eb03

Added a flag to summarize-multiqc-single.R that provides info on the …

9622004

…samples being paired-end or not.

Deleting seperate version of summarize-multiqc I created for paired r…

f8d9c28

…ead data, as I instead amended the existing script to be able to handle both single read paired end data.

Revert "Split processes in MultiQC to a single read and paired-end re…

8e1c7b5

…ad version. Renamed Multiqc to not be confusing regardings its naming as "Single"" This reverts commit 01ea0c5.

Revert "Deleted summarizeMultiqcSingle, which was superseded by summa…

0ba0552

…rizeMultiqc" This reverts commit ad8faf9.

simonleandergrimm requested review from willbradshaw and harmonbhasin November 29, 2024 02:42

harmonbhasin approved these changes Dec 2, 2024

View reviewed changes

simonleandergrimm assigned willbradshaw and unassigned harmonbhasin Dec 2, 2024

simonleandergrimm and others added 9 commits December 3, 2024 17:21

Update nextflow.config

ad640c6

Merge remote-tracking branch 'origin/harmon_fix_gh_actions_test' into…

e85dd45

… single-read-raw-clean

Merge remote-tracking branch 'origin/harmon_fix_gh_actions_test' into…

3a6f6b5

… single-read-raw-clean

Put single_end into profiles.config

a0f5f32

fixed run-dev-se config in tests

d14da14

Creating a new config for read_type flag.

3fe2bd2

added run dev se to end-to-end yml

d0375ab

Made rundevse index and outputs look the same as run.nf

f5cf80a

Fixing setup of run_dev_se test config.

3dc323e

simonleandergrimm changed the base branch from dev to harmon_fix_gh_actions_test December 5, 2024 18:47

simonleandergrimm and others added 3 commits December 5, 2024 13:47

Update .gitignore (dropped new line)

1904931

Setting profiles.config back to original

e24d79e

Updated comments in main.nf to represent the posiblity of not not ala…

b38b93d

…ysing read pairs!

harmonbhasin self-requested a review December 9, 2024 15:23

harmonbhasin requested changes Dec 9, 2024

View reviewed changes

simonleandergrimm and others added 5 commits December 9, 2024 11:50

Fixed duplicate par statement in fastp.

21b15b8

Responding to Harmon's comments.

9d717b7

dropped unncessary single-end variable.

ee7baf4

fixed faulty paired-end fastp

034914b

added end-to-end-se.yml

c5454b9

harmonbhasin approved these changes Dec 10, 2024

View reviewed changes

harmonbhasin requested changes Dec 10, 2024

View reviewed changes

workflows/run_dev_se.nf Outdated Show resolved Hide resolved

simonleandergrimm added 3 commits December 11, 2024 15:51

adedd subworkflow to create samplesheet

4b966d8

split truncate concat into two processes/

7a3a59b

removed run dev se from end to end yml.

6ad3ce2

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding single-read functionality to RAW and CLEAN #80

Adding single-read functionality to RAW and CLEAN #80

simonleandergrimm commented Oct 28, 2024 •

edited

Loading

harmonbhasin left a comment

harmonbhasin left a comment

harmonbhasin Dec 9, 2024

simonleandergrimm Dec 9, 2024

harmonbhasin left a comment

Adding single-read functionality to RAW and CLEAN #80

Are you sure you want to change the base?

Adding single-read functionality to RAW and CLEAN #80

Conversation

simonleandergrimm commented Oct 28, 2024 • edited Loading

Key Changes

Testing

harmonbhasin left a comment

Choose a reason for hiding this comment

harmonbhasin left a comment

Choose a reason for hiding this comment

harmonbhasin Dec 9, 2024

Choose a reason for hiding this comment

simonleandergrimm Dec 9, 2024

Choose a reason for hiding this comment

harmonbhasin left a comment

Choose a reason for hiding this comment

simonleandergrimm commented Oct 28, 2024 •

edited

Loading