From 8f91dd5d101e29ef5d1f010bb8f0a86b2b121a7d Mon Sep 17 00:00:00 2001 From: Richard Lupat Date: Wed, 5 Jun 2024 11:14:16 +1000 Subject: [PATCH] Built site for gh-pages --- .nojekyll | 2 +- search.json | 22 +----- sitemap.xml | 24 +++--- workshops/4.1_modules.html | 149 +++++++++++++++++++++++++++---------- 4 files changed, 125 insertions(+), 72 deletions(-) diff --git a/.nojekyll b/.nojekyll index 4126527..1ef0283 100644 --- a/.nojekyll +++ b/.nojekyll @@ -1 +1 @@ -722e3291 \ No newline at end of file +fa94a7b1 \ No newline at end of file diff --git a/search.json b/search.json index e2cc705..80e99c9 100644 --- a/search.json +++ b/search.json @@ -228,7 +228,7 @@ "href": "workshops/4.1_modules.html#if-statement", "title": "Nextflow Development - Developing Modularised Workflows", "section": "6.1.3 If statement", - "text": "6.1.3 If statement\nThe if statement uses the same syntax common in other programming languages, such as Java, C, and JavaScript.\nif (< boolean expression >) {\n // true branch\n}\nelse {\n // false branch\n}\nThe else branch is optional. Also, the curly brackets are optional when the branch defines just a single statement.\nx = 1\nif (x > 10)\n println 'Hello'\nIn some cases it can be useful to replace the if statement with a ternary expression (aka a conditional expression):\nprintln list ? list : 'The list is empty'\nThe previous statement can be further simplified using the Elvis operator:\nprintln list ?: 'The list is empty'" + "text": "6.1.3 If statement\nThe if statement uses the same syntax common in other programming languages, such as Java, C, and JavaScript.\nif (< boolean expression >) {\n // true branch\n}\nelse {\n // false branch\n}\nThe else branch is optional. Also, the curly brackets are optional when the branch defines just a single statement.\nx = 1\nif (x > 10)\n println 'Hello'\nIn some cases it can be useful to replace the if statement with a ternary expression (aka a conditional expression):\nprintln list ? list : 'The list is empty'\nThe previous statement can be further simplified using the Elvis operator:\nprintln list ?: 'The list is empty'\nExercise\nWe are going to turn the rnaseq.nf into a conditional workflow with an additional params.qc_enabled to set an on/off trigger for the QC parts of the workflow.\nparams.qc_enabled = false\n\nworkflow {\n QUANT_WF(Channel.of(params.transcriptome_file), reads_ch)\n\n if (params.qc_enabled) {\n QC_WF(reads_ch, QUANT_WF.out)\n }\n}\nRun the workflow again:\nnextflow run rnaseq.nf --outdir \"results\"\nWe should only see the following two stages being executed.\nN E X T F L O W ~ version 23.04.1\nLaunching `rnaseq.nf` [hopeful_gautier] DSL2 - revision: 7c50056656\nexecutor > local (2)\n[c3/91f695] process > QUANT_WF:INDEX (1) [100%] 1 of 1 ✔\n[1d/fac0d9] process > QUANT_WF:QT (1) [100%] 1 of 1 ✔\nThe params.qc_enabled can be turn on during execution.\nnextflow run rnaseq.nf --outdir \"results\" --qc_enabled true\n\n\n\n\n\n\nChallenge\nThe trimgalore currently only supports paired-end read. How do we update this so the same process can be used for both single-end and paired-end?\nFor reference, the (simplified) command that we can use for single-end can be as follow:\n trim_galore \\\\\n --gzip \\\\\n $reads" }, { "objectID": "workshops/4.1_modules.html#functions", @@ -237,13 +237,6 @@ "section": "6.1.4 Functions", "text": "6.1.4 Functions\nIt is possible to define a custom function into a script:\ndef fib(int n) {\n return n < 2 ? 1 : fib(n - 1) + fib(n - 2)\n}\n\nassert fib(10)==89\nA function can take multiple arguments separating them with a comma.\nThe return keyword can be omitted and the function implicitly returns the value of the last evaluated expression. Also, explicit types can be omitted, though not recommended:\ndef fact(n) {\n n > 1 ? n * fact(n - 1) : 1\n}\n\nassert fact(5) == 120" }, - { - "objectID": "workshops/4.1_modules.html#grooovy-library", - "href": "workshops/4.1_modules.html#grooovy-library", - "title": "Nextflow Development - Developing Modularised Workflows", - "section": "6.2 Grooovy Library", - "text": "6.2 Grooovy Library" - }, { "objectID": "workshops/4.1_modules.html#testing", "href": "workshops/4.1_modules.html#testing", @@ -256,21 +249,14 @@ "href": "workshops/4.1_modules.html#stub", "title": "Nextflow Development - Developing Modularised Workflows", "section": "7.1 Stub", - "text": "7.1 Stub\nYou can define a command stub, which replaces the actual process command when the -stub-run or -stub command-line option is enabled:\n\nprocess INDEX {\n input:\n path transcriptome\n\n output:\n path 'index'\n\n script:\n \"\"\"\n salmon index --threads $task.cpus -t $transcriptome -i index\n \"\"\"\n\n stub:\n \"\"\"\n mkdir index\n touch index/seq.bin\n touch index/info.json\n touch index/refseq.bin\n \"\"\"\n}\nThe stub block can be defined before or after the script block. When the pipeline is executed with the -stub-run option and a process’s stub is not defined, the script block is executed.\nThis feature makes it easier to quickly prototype the workflow logic without using the real commands. The developer can use it to provide a dummy script that mimics the execution of the real one in a quicker manner. In other words, it is a way to perform a dry-run." - }, - { - "objectID": "workshops/4.1_modules.html#test-profile", - "href": "workshops/4.1_modules.html#test-profile", - "title": "Nextflow Development - Developing Modularised Workflows", - "section": "7.2 Test profile", - "text": "7.2 Test profile" + "text": "7.1 Stub\nYou can define a command stub, which replaces the actual process command when the -stub-run or -stub command-line option is enabled:\n\nprocess INDEX {\n container \"/config/binaries/singularity/containers_devel/nextflow/depot.galaxyproject.org-singularity-salmon-1.10.1--h7e5ed60_0.img\"\n\n input:\n path transcriptome\n\n output:\n path \"salmon_idx\"\n\n script:\n \"\"\"\n salmon index --threads $task.cpus -t $transcriptome -i salmon_idx\n \"\"\"\n\n stub:\n \"\"\"\n mkdir salmon_idx\n touch salmon_idx/seq.bin\n touch salmon_idx/info.json\n touch salmon_idx/refseq.bin\n \"\"\"\n}\nThe stub block can be defined before or after the script block. When the pipeline is executed with the -stub-run option and a process’s stub is not defined, the script block is executed.\nThis feature makes it easier to quickly prototype the workflow logic without using the real commands. The developer can use it to provide a dummy script that mimics the execution of the real one in a quicker manner. In other words, it is a way to perform a dry-run.\nExercise\nTry modifying modules.nf to add stub for the INDEX process.\n \"\"\"\n mkdir salmon_idx\n touch salmon_idx/seq.bin\n touch salmon_idx/info.json\n touch salmon_idx/refseq.bin\n \"\"\"\nLet’s keep the workflow to only run the INDEX process, as a new rnaseq_stub.nf\nworkflow {\n index_ch = INDEX(params.transcriptome_file)\n}\nAnd run the rnaseq_stub.nf with -stub-run\nnextflow run rnaseq_stub.nf -stub-run\nN E X T F L O W ~ version 23.04.1\nLaunching `rnaseq.nf` [lonely_albattani] DSL2 - revision: 11fb1399f0\nexecutor > local (1)\n[a9/7d3084] process > INDEX [100%] 1 of 1 ✔\nThe process should look like it is running as normal. But if we inspect the work folder a9/7d3084, you will notice that the salmon_idx folder actually consists of three empty files that we touch as part of stub.\nls -la work/a9/7d3084636d95cba6b81a9ce8125289/salmon_idx/\ntotal 1\ndrwxrwxr-x 2 rlupat rlupat 4096 Jun 5 11:05 .\ndrwxrwxr-x 3 rlupat rlupat 4096 Jun 5 11:05 ..\n-rw-rw-r-- 1 rlupat rlupat 0 Jun 5 11:05 info.json\n-rw-rw-r-- 1 rlupat rlupat 0 Jun 5 11:05 refseq.bin\n-rw-rw-r-- 1 rlupat rlupat 0 Jun 5 11:05 seq.bin\n\n\n\n\n\n\nChallenge\nAdd stubs to all modules in modules.nf and try running the full workflow in a stub." }, { "objectID": "workshops/4.1_modules.html#nf-test", "href": "workshops/4.1_modules.html#nf-test", "title": "Nextflow Development - Developing Modularised Workflows", - "section": "7.3. nf-test", - "text": "7.3. nf-test\nIt is critical for reproducibility and long-term maintenance to have a way to systematically test that every part of your workflow is doing what it’s supposed to do. To that end, people often focus on top-level tests, in which the workflow is un on some test data from start to finish. This is useful but unfortunately incomplete. You should also implement module-level tests (equivalent to what is called ‘unit tests’ in general software engineering) to verify the functionality of individual components of your workflow, ensuring that each module performs as expected under different conditions and inputs.\nThe nf-test package provides a testing framework that integrates well with Nextflow and makes it straightforward to add both module-level and workflow-level tests to your pipeline. For more background information, read the blog post about nf-test on the nf-core blog.\nSee this tutorial for some examples.\n\nThis workshop is adapted from Fundamentals Training, Advanced Training, Developer Tutorials, and Nextflow Patterns materials from Nextflow and nf-core" + "section": "7.2. nf-test", + "text": "7.2. nf-test\nIt is critical for reproducibility and long-term maintenance to have a way to systematically test that every part of your workflow is doing what it’s supposed to do. To that end, people often focus on top-level tests, in which the workflow is un on some test data from start to finish. This is useful but unfortunately incomplete. You should also implement module-level tests (equivalent to what is called ‘unit tests’ in general software engineering) to verify the functionality of individual components of your workflow, ensuring that each module performs as expected under different conditions and inputs.\nThe nf-test package provides a testing framework that integrates well with Nextflow and makes it straightforward to add both module-level and workflow-level tests to your pipeline. For more background information, read the blog post about nf-test on the nf-core blog.\nSee this tutorial for some examples.\n\nThis workshop is adapted from Fundamentals Training, Advanced Training, Developer Tutorials, and Nextflow Patterns materials from Nextflow and nf-core" }, { "objectID": "workshops/00_setup.html", diff --git a/sitemap.xml b/sitemap.xml index d404978..9a03f0c 100644 --- a/sitemap.xml +++ b/sitemap.xml @@ -2,50 +2,50 @@ https://PMCC-BioinformaticsCore.github.io/nextflow-intro-workshop/sessions/2_nf_dev_intro.html - 2024-06-04T18:14:29.561Z + 2024-06-05T01:14:15.516Z https://PMCC-BioinformaticsCore.github.io/nextflow-intro-workshop/index.html - 2024-06-04T18:14:28.760Z + 2024-06-05T01:14:14.739Z https://PMCC-BioinformaticsCore.github.io/nextflow-intro-workshop/workshops/2.3_tips_and_tricks.html - 2024-06-04T18:14:27.132Z + 2024-06-05T01:14:13.126Z https://PMCC-BioinformaticsCore.github.io/nextflow-intro-workshop/workshops/1.2_intro_nf_core.html - 2024-06-04T18:14:26.201Z + 2024-06-05T01:14:12.280Z https://PMCC-BioinformaticsCore.github.io/nextflow-intro-workshop/workshops/2.2_troubleshooting.html - 2024-06-04T18:14:24.589Z + 2024-06-05T01:14:10.690Z https://PMCC-BioinformaticsCore.github.io/nextflow-intro-workshop/workshops/1.1_intro_nextflow.html - 2024-06-04T18:14:23.511Z + 2024-06-05T01:14:09.608Z https://PMCC-BioinformaticsCore.github.io/nextflow-intro-workshop/workshops/3.1_creating_a_workflow.html - 2024-06-04T18:14:22.899Z + 2024-06-05T01:14:08.992Z https://PMCC-BioinformaticsCore.github.io/nextflow-intro-workshop/workshops/4.1_draft_future_sess.html - 2024-06-04T18:14:23.880Z + 2024-06-05T01:14:09.996Z https://PMCC-BioinformaticsCore.github.io/nextflow-intro-workshop/workshops/4.1_modules.html - 2024-06-04T18:14:25.400Z + 2024-06-05T01:14:11.525Z https://PMCC-BioinformaticsCore.github.io/nextflow-intro-workshop/workshops/00_setup.html - 2024-06-04T18:14:26.633Z + 2024-06-05T01:14:12.669Z https://PMCC-BioinformaticsCore.github.io/nextflow-intro-workshop/workshops/2.1_customise_and_run.html - 2024-06-04T18:14:28.419Z + 2024-06-05T01:14:14.390Z https://PMCC-BioinformaticsCore.github.io/nextflow-intro-workshop/sessions/1_intro_run_nf.html - 2024-06-04T18:14:29.158Z + 2024-06-05T01:14:15.120Z diff --git a/workshops/4.1_modules.html b/workshops/4.1_modules.html index 24ab4fb..2eccb3e 100644 --- a/workshops/4.1_modules.html +++ b/workshops/4.1_modules.html @@ -210,11 +210,9 @@

On this page

  • 6.1.2 Maps
  • 6.1.3 If statement
  • 6.1.4 Functions
    • -
  • 6.2 Grooovy Library
  • 7. Testing
  • 7.1 Stub
    • -
  • 7.2 Test profile
    • -
  • 7.3. nf-test
    • +
  • 7.2. nf-test
    • @@ -1005,25 +1003,58 @@

    6.1.3 If statement

    println list ? list : 'The list is empty'

    The previous statement can be further simplified using the Elvis operator:

    println list ?: 'The list is empty'
    +

    Exercise

    +

    We are going to turn the rnaseq.nf into a conditional workflow with an additional params.qc_enabled to set an on/off trigger for the QC parts of the workflow.

    +
    params.qc_enabled = false
+
+workflow {
+  QUANT_WF(Channel.of(params.transcriptome_file), reads_ch)
+
+  if (params.qc_enabled) {
+    QC_WF(reads_ch, QUANT_WF.out)
+  }
+}
    +

    Run the workflow again:

    +
    nextflow run rnaseq.nf --outdir "results"
    +

    We should only see the following two stages being executed.

    +
    N E X T F L O W  ~  version 23.04.1
+Launching `rnaseq.nf` [hopeful_gautier] DSL2 - revision: 7c50056656
+executor >  local (2)
+[c3/91f695] process > QUANT_WF:INDEX (1) [100%] 1 of 1 ✔
+[1d/fac0d9] process > QUANT_WF:QT (1)    [100%] 1 of 1 ✔
    +

    The params.qc_enabled can be turn on during execution.

    +
    nextflow run rnaseq.nf --outdir "results" --qc_enabled true
    +
    +
    +
    + +
    +
    +

    Challenge

    +

    The trimgalore currently only supports paired-end read. How do we update this so the same process can be used for both single-end and paired-end?

    +

    For reference, the (simplified) command that we can use for single-end can be as follow:

    +
      trim_galore \\
+    --gzip \\
+    $reads
    +
    +
    +

    6.1.4 Functions

    It is possible to define a custom function into a script:

    -
    def fib(int n) {
-    return n < 2 ? 1 : fib(n - 1) + fib(n - 2)
-}
-
-assert fib(10)==89
    +
    def fib(int n) {
+    return n < 2 ? 1 : fib(n - 1) + fib(n - 2)
+}
+
+assert fib(10)==89

    A function can take multiple arguments separating them with a comma.

    The return keyword can be omitted and the function implicitly returns the value of the last evaluated expression. Also, explicit types can be omitted, though not recommended:

    -
    def fact(n) {
-    n > 1 ? n * fact(n - 1) : 1
-}
-
-assert fact(5) == 120
    -
    -
    -

    6.2 Grooovy Library

    +
    def fact(n) {
+    n > 1 ? n * fact(n - 1) : 1
+}
+
+assert fact(5) == 120

    7. Testing

    @@ -1031,35 +1062,71 @@

    7. Testing

    7.1 Stub

    You can define a command stub, which replaces the actual process command when the -stub-run or -stub command-line option is enabled:

    -
    
-process INDEX {
-  input:
-    path transcriptome
-
-  output:
-    path 'index'
-
-  script:
-    """
-    salmon index --threads $task.cpus -t $transcriptome -i index
-    """
-
-  stub:
-    """
-    mkdir index
-    touch index/seq.bin
-    touch index/info.json
-    touch index/refseq.bin
-    """
-}
    +
    
+process INDEX {
+    container "/config/binaries/singularity/containers_devel/nextflow/depot.galaxyproject.org-singularity-salmon-1.10.1--h7e5ed60_0.img"
+
+    input:
+    path transcriptome
+
+    output:
+    path "salmon_idx"
+
+    script:
+    """
+    salmon index --threads $task.cpus -t $transcriptome -i salmon_idx
+    """
+
+    stub:
+    """
+    mkdir salmon_idx
+    touch salmon_idx/seq.bin
+    touch salmon_idx/info.json
+    touch salmon_idx/refseq.bin
+    """
+}

    The stub block can be defined before or after the script block. When the pipeline is executed with the -stub-run option and a process’s stub is not defined, the script block is executed.

    This feature makes it easier to quickly prototype the workflow logic without using the real commands. The developer can use it to provide a dummy script that mimics the execution of the real one in a quicker manner. In other words, it is a way to perform a dry-run.

    -
    -
    -

    7.2 Test profile

    +

    Exercise

    +

    Try modifying modules.nf to add stub for the INDEX process.

    +
        """
+    mkdir salmon_idx
+    touch salmon_idx/seq.bin
+    touch salmon_idx/info.json
+    touch salmon_idx/refseq.bin
+    """
    +

    Let’s keep the workflow to only run the INDEX process, as a new rnaseq_stub.nf

    +
    workflow {
+  index_ch = INDEX(params.transcriptome_file)
+}
    +

    And run the rnaseq_stub.nf with -stub-run

    +
    nextflow run rnaseq_stub.nf -stub-run
    +
    N E X T F L O W  ~  version 23.04.1
+Launching `rnaseq.nf` [lonely_albattani] DSL2 - revision: 11fb1399f0
+executor >  local (1)
+[a9/7d3084] process > INDEX [100%] 1 of 1 ✔
    +

    The process should look like it is running as normal. But if we inspect the work folder a9/7d3084, you will notice that the salmon_idx folder actually consists of three empty files that we touch as part of stub.

    +
    ls -la work/a9/7d3084636d95cba6b81a9ce8125289/salmon_idx/
+total 1
+drwxrwxr-x 2 rlupat rlupat 4096 Jun  5 11:05 .
+drwxrwxr-x 3 rlupat rlupat 4096 Jun  5 11:05 ..
+-rw-rw-r-- 1 rlupat rlupat    0 Jun  5 11:05 info.json
+-rw-rw-r-- 1 rlupat rlupat    0 Jun  5 11:05 refseq.bin
+-rw-rw-r-- 1 rlupat rlupat    0 Jun  5 11:05 seq.bin
    +
    +
    +
    + +
    +
    +

    Challenge

    +

    Add stubs to all modules in modules.nf and try running the full workflow in a stub.

    +
    +
    +
    -

    7.3. nf-test

    +

    7.2. nf-test

    It is critical for reproducibility and long-term maintenance to have a way to systematically test that every part of your workflow is doing what it’s supposed to do. To that end, people often focus on top-level tests, in which the workflow is un on some test data from start to finish. This is useful but unfortunately incomplete. You should also implement module-level tests (equivalent to what is called ‘unit tests’ in general software engineering) to verify the functionality of individual components of your workflow, ensuring that each module performs as expected under different conditions and inputs.

    The nf-test package provides a testing framework that integrates well with Nextflow and makes it straightforward to add both module-level and workflow-level tests to your pipeline. For more background information, read the blog post about nf-test on the nf-core blog.

    See this tutorial for some examples.