Add samplesheet generation for taxprofiler #39

Joon-Klaps · 2024-10-10T10:39:30Z

PR checklist

Joon-Klaps · 2024-10-10T12:25:14Z

Note:

The default test profile outputs empty nanopore files which makes taxprofiler's Porechop crash.

log -profile test | taxprofiler

[-        ] NFCORE_TAXPROFILER:TAXPROFILER:UNTAR                                                                                       [  0%] 0 of 8
[dc/7bb954] NFCORE_TAXPROFILER:TAXPROFILER:FASTQC (test_paired-end_plus_long-reads_longReads)                                          [100%] 5 of 5 ✔
[0b/72b269] NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_PREPROCESSING:SHORTREAD_FASTP:FASTP_SINGLE (test_single-end_short)                [100%] 1 of 1 ✔
[01/7fb7f6] NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_PREPROCESSING:SHORTREAD_FASTP:FASTP_PAIRED (test_paired-end_plus_long-reads)      [100%] 2 of 2 ✔
[-        ] NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_PREPROCESSING:FASTQC_PROCESSED                                                    -
[6c/c4aa69] NFC…E_TAXPROFILER:TAXPROFILER:LONGREAD_PREPROCESSING:LONGREAD_ADAPTERREMOVAL:PORECHOP_ABI (test_single-end_long_longReads) [100%] 2 of 2, failed: 2 ✘
[-        ] NFCORE_TAXPROFILER:TAXPROFILER:LONGREAD_PREPROCESSING:LONGREAD_FILTERING:NANOQ                                             -
[-        ] NFCORE_TAXPROFILER:TAXPROFILER:LONGREAD_PREPROCESSING:FASTQC_PROCESSED                                                     -
[-        ] NFCORE_TAXPROFILER:TAXPROFILER:NONPAREIL:NONPAREIL_NONPAREIL                                                               -
[-        ] NFCORE_TAXPROFILER:TAXPROFILER:NONPAREIL:NONPAREIL_CURVE                                                                   -
[-        ] NFCORE_TAXPROFILER:TAXPROFILER:NONPAREIL:NONPAREIL_SET                                                                     -
[-        ] NFCORE_TAXPROFILER:TAXPROFILER:NONPAREIL:NONPAREIL_NONPAREILCURVESR                                                        -
[-        ] NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_COMPLEXITYFILTERING:BBMAP_BBDUK                                                   -
[-        ] NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_HOSTREMOVAL:BOWTIE2_BUILD                                                         [  0%] 0 of 1
[-        ] NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_HOSTREMOVAL:BOWTIE2_ALIGN                                                         -
[-        ] NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_HOSTREMOVAL:SAMTOOLS_INDEX                                                        -
[-        ] NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_HOSTREMOVAL:SAMTOOLS_STATS                                                        -
[-        ] NFCORE_TAXPROFILER:TAXPROFILER:LONGREAD_HOSTREMOVAL:MINIMAP2_INDEX                                                         [  0%] 0 of 1
[-        ] NFCORE_TAXPROFILER:TAXPROFILER:LONGREAD_HOSTREMOVAL:MINIMAP2_ALIGN                                                         -
Plus 35 more processes waiting for tasks…
Execution cancelled -- Finishing pending tasks before exit
-[nf-core/taxprofiler] Pipeline completed with errors-
ERROR ~ Error executing process > 'NFCORE_TAXPROFILER:TAXPROFILER:LONGREAD_PREPROCESSING:LONGREAD_ADAPTERREMOVAL:PORECHOP_ABI (test_paired-end_plus_long-reads_longReads)'

Caused by:
  Process `NFCORE_TAXPROFILER:TAXPROFILER:LONGREAD_PREPROCESSING:LONGREAD_ADAPTERREMOVAL:PORECHOP_ABI (test_paired-end_plus_long-reads_longReads)` terminated with an error exit status (1)


Command executed:

  porechop_abi \
      --input test_paired-end_plus_long-reads_longReads_filtered.fastq.gz \
      --threads 2 \
       \
      --output test_paired-end_plus_long-reads_longReads_test_paired-end_plus_long-reads_porechop_abi.fastq.gz \
      | tee test_paired-end_plus_long-reads_longReads_test_paired-end_plus_long-reads_porechop_abi.log
  cat <<-END_VERSIONS > versions.yml
  "NFCORE_TAXPROFILER:TAXPROFILER:LONGREAD_PREPROCESSING:LONGREAD_ADAPTERREMOVAL:PORECHOP_ABI":
      porechop_abi: $( porechop_abi --version )
  END_VERSIONS

Command exit status:
  1

Command output:
  
  Loading reads
  test_paired-end_plus_long-reads_longReads_filtered.fastq.gz

Command error:
  Traceback (most recent call last):
    File "/usr/local/bin/porechop_abi", line 11, in <module>
  
  Loading reads
  test_paired-end_plus_long-reads_longReads_filtered.fastq.gz
      sys.exit(main())
    File "/usr/local/lib/python3.10/site-packages/porechop_abi/porechop_abi.py", line 63, in main
      reads, check_reads, read_type = load_reads(args.input,
    File "/usr/local/lib/python3.10/site-packages/porechop_abi/porechop_abi.py", line 136, in load_reads
      reads, read_type = load_fasta_or_fastq(input_file_or_directory)
    File "/usr/local/lib/python3.10/site-packages/porechop_abi/misc.py", line 114, in load_fasta_or_fastq
      file_type = get_sequence_file_type(filename)
    File "/usr/local/lib/python3.10/site-packages/porechop_abi/misc.py", line 106, in get_sequence_file_type
      raise ValueError('File is neither FASTA or FASTQ')
  ValueError: File is neither FASTA or FASTQ

Work dir:
  /workspace/detaxizer/testing/work/9e/fe127faf0ee71dc856849d6b73507e

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`

 -- Check '.nextflow.log' file for details
ERROR ~ Pipeline failed. Please refer to troubleshooting docs: https://nf-co.re/docs/usage/troubleshooting

 -- Check '.nextflow.log' file for details

detaxizer -profile test_blastn doesn't but crashes at TAXPROFILER:NONPAREIL:NONPAREIL_CURVE

log -profile test_blastn | taxprofiler

cc/a04a1e] NFCORE_TAXPROFILER:TAXPROFILER:UNTAR (test-db-centrifuge.tar.gz)                                                            [100%] 8 of 8 ✔
[b4/d62ad8] NFCORE_TAXPROFILER:TAXPROFILER:FASTQC (test_paired-end_plus_long-reads_longReads)                                           [100%] 5 of 5 ✔
[0b/6e35f0] NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_PREPROCESSING:SHORTREAD_FASTP:FASTP_SINGLE (test_single-end_short)                 [100%] 1 of 1 ✔
[23/b28ca8] NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_PREPROCESSING:SHORTREAD_FASTP:FASTP_PAIRED (test_paired-end_plus_long-reads)       [100%] 2 of 2 ✔
[93/f63330] NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_PREPROCESSING:FASTQC_PROCESSED (test_single-end_short)                             [100%] 3 of 3 ✔
[d5/75d96c] NFCORE_TAXPROFILER:TAXPROFILER:LONGREAD_PREPROCESSING:LONGREAD_ADAPTERREMOVAL:PORECHOP_ABI (test_single-end_long_longReads) [100%] 2 of 2 ✔
[84/9469f9] NFCORE_TAXPROFILER:TAXPROFILER:LONGREAD_PREPROCESSING:LONGREAD_FILTERING:NANOQ (test_single-end_long_longReads)             [100%] 2 of 2 ✔
[95/97cbfe] NFCORE_TAXPROFILER:TAXPROFILER:LONGREAD_PREPROCESSING:FASTQC_PROCESSED (test_single-end_long_longReads)                     [100%] 2 of 2 ✔
[dc/4c58dc] NFCORE_TAXPROFILER:TAXPROFILER:NONPAREIL:NONPAREIL_NONPAREIL (test_single-end_short)                                        [100%] 3 of 3 ✔
[25/579fab] NFCORE_TAXPROFILER:TAXPROFILER:NONPAREIL:NONPAREIL_CURVE (test_paired-end)                                                  [100%] 2 of 2, failed: 2
[-        ] NFCORE_TAXPROFILER:TAXPROFILER:NONPAREIL:NONPAREIL_SET                                                                      [  0%] 0 of 1
[-        ] NFCORE_TAXPROFILER:TAXPROFILER:NONPAREIL:NONPAREIL_NONPAREILCURVESR                                                         [  0%] 0 of 1
[dc/9bde9f] NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_COMPLEXITYFILTERING:BBMAP_BBDUK (test_single-end_short)                            [100%] 3 of 3 ✔
[4d/1d618c] NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_HOSTREMOVAL:BOWTIE2_BUILD (genome.fasta)                                           [100%] 1 of 1 ✔
[-        ] NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_HOSTREMOVAL:BOWTIE2_ALIGN                                                          [  0%] 0 of 3
[-        ] NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_HOSTREMOVAL:SAMTOOLS_INDEX                                                         -
[-        ] NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_HOSTREMOVAL:SAMTOOLS_STATS                                                         -
[af/4afeca] NFCORE_TAXPROFILER:TAXPROFILER:LONGREAD_HOSTREMOVAL:MINIMAP2_INDEX                                                          [100%] 1 of 1 ✔
[95/9819b6] NFCORE_TAXPROFILER:TAXPROFILER:LONGREAD_HOSTREMOVAL:MINIMAP2_ALIGN (test_single-end_long_longReads)                         [100%] 2 of 2 ✔
[-        ] NFCORE_TAXPROFILER:TAXPROFILER:MULTIQC                                                                                      [  0%] 0 of 1
Plus 34 more processes waiting for tasks…
Execution cancelled -- Finishing pending tasks before exit
-[nf-core/taxprofiler] Pipeline completed with errors-
ERROR ~ Error executing process > 'NFCORE_TAXPROFILER:TAXPROFILER:NONPAREIL:NONPAREIL_CURVE (test_paired-end_plus_long-reads)'

Caused by:
  Missing output file(s) `*.png` expected by process `NFCORE_TAXPROFILER:TAXPROFILER:NONPAREIL:NONPAREIL_CURVE (test_paired-end_plus_long-reads)`


Command executed:

  #!/usr/bin/env Rscript
  library(Nonpareil)
  
  png(file='test_paired-end_plus_long-reads_test_paired-end_plus_long-reads.png')
  Nonpareil.curve('test_paired-end_plus_long-reads_test_paired-end_plus_long-reads.npo')
  dev.off()
  
  version_file_path <- "versions.yml"
  version_nonpareil <- paste(unlist(packageVersion("Nonpareil")), collapse = ".")
  f <- file(version_file_path, "w")
  writeLines('"NFCORE_TAXPROFILER:TAXPROFILER:NONPAREIL:NONPAREIL_CURVE":', f)
  writeLines("    nonpareil: ", f, sep = "")
  writeLines(version_nonpareil, f)
  close(f)

Command exit status:
  0

Command output:
  null device 
            1 

Command error:
  Warning message:
  In Nonpareil.curve("test_paired-end_plus_long-reads_test_paired-end_plus_long-reads.npo") :
    Insufficient resolution below 90% coverage, check parameters and re-run (e.g., increase the value of -d in nonpareil).

Work dir:
  /workspace/detaxizer/testing/work/54/03c55ab469331fb1edf02b8b2754c4

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`

 -- Check '.nextflow.log' file for details
ERROR ~ Pipeline failed. Please refer to troubleshooting docs: https://nf-co.re/docs/usage/troubleshooting

 -- Check '.nextflow.log' file for details

*Edit, adding logs & different profiles

Joon-Klaps · 2024-10-10T12:29:08Z

Linting with tools 2.14.1 results in 0 errors.

$ nf-core lint

                                      ,--./,-.
      ___     __   __   __   ___     /,-._.--~\
|\ | |__  __ /  ` /  \ |__) |__         }  {
| \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                      `._,._,'

nf-core/tools version 2.14.1 - https://nf-co.re
There is a new version of nf-core/tools available! (3.0.1)

INFO Testing pipeline: .

╭─ [?] 2 Pipeline Tests Ignored ───────────────────────────────────────────────────╮
│                                                                                  │
│ files_unchanged: File ignored due to lint config: .github/CONTRIBUTING.md        │
│ files_unchanged: File ignored due to lint config:                                │
│ .github/PULL_REQUEST_TEMPLATE.md                                                 │
│                                                                                  │
╰──────────────────────────────────────────────────────────────────────────────────╯
╭─ [!] 3 Pipeline Test Warnings ───────────────────────────────────────────────────╮
│                                                                                  │
│ pipeline_todos: TODO string in awsfulltest.yml: You can customise AWS full       │
│ pipeline tests as required                                                       │
│ schema_lint: Schema $schema should be http://json-schema.org/draft-07/schema     │
│ Found https://json-schema.org/draft-07/schema                                    │
│ schema_description: No description provided in schema for parameter:             │
│ saveReference                                                                    │
│                                                                                  │
╰──────────────────────────────────────────────────────────────────────────────────╯

╭─ [!] 8 Module Test Warnings ─────────────────────────────────────────────────────╮
│                             ╷                         ╷                          │
│ Module name                 │ File path               │ Test message             │
│╶────────────────────────────┼─────────────────────────┼─────────────────────────╴│
│ kraken2preparation          │ modules/local/kraken2p… │ 'meta' map not emitted   │
│                             │                         │ in output channel(s)     │
│ bbmap/bbduk                 │ modules/nf-core/bbmap/… │ New version available    │
│ blast/blastn                │ modules/nf-core/blast/… │ New version available    │
│ blast/makeblastdb           │ modules/nf-core/blast/… │ New version available    │
│ fastp                       │ modules/nf-core/fastp   │ New version available    │
│ fastqc                      │ modules/nf-core/fastqc  │ New version available    │
│ kraken2/kraken2             │ modules/nf-core/kraken… │ New version available    │
│ multiqc                     │ modules/nf-core/multiqc │ New version available    │
│                             ╵                         ╵                          │
╰──────────────────────────────────────────────────────────────────────────────────╯

╭─ [!] 3 Subworkflow Test Warnings ────────────────────────────────────────────────╮
│                            ╷                          ╷                          │
│ Subworkflow name           │ File path                │ Test message             │
│╶───────────────────────────┼──────────────────────────┼─────────────────────────╴│
│ utils_nextflow_pipeline    │ subworkflows/nf-core/ut… │ New version available    │
│ utils_nfcore_pipeline      │ subworkflows/nf-core/ut… │ New version available    │
│ utils_nfvalidation_plugin  │ subworkflows/nf-core/ut… │ New version available    │
│                            ╵                          ╵                          │
╰──────────────────────────────────────────────────────────────────────────────────╯
╭───────────────────────╮
│ LINT RESULTS SUMMARY  │
├───────────────────────┤
│ [✔] 487 Tests Passed  │
│ [?]   2 Tests Ignored │
│ [!]  14 Test Warnings │
│ [✗]   0 Tests Failed  │
╰───────────────────────╯

Joon-Klaps · 2024-10-10T15:23:42Z

✅ It runs with no erros when the input sheet is

sample,short_reads_fastq_1,short_reads_fastq_2,long_reads_fastq_1
ERR5766180,https://raw.githubusercontent.com/nf-core/test-datasets/taxprofiler/data/fastq/ERX5474936_ERR5766180_1.fastq.gz,,
ERR5766176,https://raw.githubusercontent.com/nf-core/test-datasets/taxprofiler/data/fastq/ERX5474932_ERR5766176_1.fastq.gz,https://raw.githubusercontent.com/nf-core/test-datasets/taxprofiler/data/fastq/ERX5474932_ERR5766176_2.fastq.gz,https://raw.githubusercontent.com/nf-core/test-datasets/taxprofiler/data/fastq/ERR3201952.fastq.gz
ERR5766176_B,https://raw.githubusercontent.com/nf-core/test-datasets/taxprofiler/data/fastq/ERX5474932_ERR5766176_B_1.fastq.gz,https://raw.githubusercontent.com/nf-core/test-datasets/taxprofiler/data/fastq/ERX5474932_ERR5766176_B_2.fastq.gz,
ERR3201952,,,https://raw.githubusercontent.com/nf-core/test-datasets/taxprofiler/data/fastq/ERR3201952.fastq.gz

detaxizer:

nextflow run . -profile docker,test_blastn --filter_trimmed false --outdir ./results --input ./testing/samplesheet.csv -resume

taxprofiler:

mkdir -p testing & cd testing;
nextflow run nf-core/taxprofiler --input ../results/downstream_samplesheet/taxprofiler.csv -profile docker,test --outdir ./results

Joon-Klaps · 2024-10-10T15:26:11Z

subworkflows/local/generate_downstream_samplesheets/main.nf

+//
+// Subworkflow with functionality specific to the nf-core/createtaxdb pipeline
+//
+workflow GENERATE_DOWNSTREAM_SAMPLESHEETS {


Should be made a sub workflow specifically for taxprofiler, as done in https://github.com/nf-core/mag/pull/688/files

Joon-Klaps · 2024-10-10T15:26:26Z

subworkflows/local/generate_downstream_samplesheets/main.nf

+
+    // Constructs the header string and then the strings of each row, and
+    // finally concatenates for saving.
+    ch_colnames


Try & make a function out of it

initial setup detaxizer -> taxprofiler

6a51981

Joon-Klaps linked an issue Oct 10, 2024 that may be closed by this pull request

Add samplesheet generation for Taxprofiler #38

Open

Joon-Klaps added 2 commits October 10, 2024 12:45

fix nanopore logic

c9bc9dd

add docs

97756ab

jfy133 mentioned this pull request Oct 10, 2024

Non pareil fails if data is insufficient for calculation due to a missing png nf-core/taxprofiler#543

Open

Joon-Klaps commented Oct 10, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add samplesheet generation for taxprofiler #39

Add samplesheet generation for taxprofiler #39

Joon-Klaps commented Oct 10, 2024

Joon-Klaps commented Oct 10, 2024 •

edited

Loading

Joon-Klaps commented Oct 10, 2024

Joon-Klaps commented Oct 10, 2024

Joon-Klaps Oct 10, 2024

Joon-Klaps Oct 10, 2024

Add samplesheet generation for taxprofiler #39

Are you sure you want to change the base?

Add samplesheet generation for taxprofiler #39

Conversation

Joon-Klaps commented Oct 10, 2024

PR checklist

Joon-Klaps commented Oct 10, 2024 • edited Loading

Joon-Klaps commented Oct 10, 2024

Joon-Klaps commented Oct 10, 2024

Joon-Klaps Oct 10, 2024

Choose a reason for hiding this comment

Joon-Klaps Oct 10, 2024

Choose a reason for hiding this comment

Joon-Klaps commented Oct 10, 2024 •

edited

Loading