Merging the stable version of the pipeline for the first release (#1)

This PR includes all the development commits for v1.0 of the Shallow-mapping pipeline: * Updating README * Updating schema * Pipeline setting up * Raw-reads preprocesing: qc, trimming and decont * sourmash module added * Post-processing modules added * BWA mapping postprocessing modules * Cleaning modules and adding bam2cov.py script * dram module and core_mode function added * Integrate annotations steps * Refactoring pangenomeDB_builder_codon.py to use exisiting files in codon * nextflow.config and schema update * Script to download databases added * dram distill debug and test data added * Updating decontamination module * Adding pipeline nf-test * Reformat python scripts with black * Diagram updated * Fix linting problems. * Fixes on modules * Test info added to README
EBI-Metagenomics · Apr 16, 2024 · bc33f93 · bc33f93
1 parent 90d217f
commit bc33f93
Show file tree

Hide file tree

Showing 78 changed files with 3,991 additions and 847 deletions.
diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md
@@ -23,6 +23,12 @@ If you're not used to this workflow with git, you can start with some [docs from
 
 ## Tests
 
+You have the option to test your changes locally by running the pipeline. For receiving warnings about process selectors and other `debug` information, it is recommended to use the debug profile. Execute all the tests with the following command:
+
+```bash
+nf-test test --profile debug,test,docker --verbose
+```
+
 When you create a pull request with changes, [GitHub Actions](https://github.com/features/actions) will run automatic tests.
 Typically, pull-requests are only fully reviewed when these tests are passing, though of course we can help out before then.
 
@@ -79,7 +85,7 @@ Once there, use `nf-core schema build` to add to `nextflow_schema.json`.
 
 Sensible defaults for process resource requirements (CPUs / memory / time) for a process should be defined in `conf/base.config`. These should generally be specified generic with `withLabel:` selectors so they can be shared across multiple processes/steps of the pipeline. A nf-core standard set of labels that should be followed where possible can be seen in the [nf-core pipeline template](https://github.com/nf-core/tools/blob/master/nf_core/pipeline-template/conf/base.config), which has the default process as a single core-process, and then different levels of multi-core configurations for increasingly large memory requirements defined with standardised labels.
 
-The process resources can be passed on to the tool dynamically within the process with the `${task.cpu}` and `${task.memory}` variables in the `script:` block.
+The process resources can be passed on to the tool dynamically within the process with the `${task.cpus}` and `${task.memory}` variables in the `script:` block.
 
 ### Naming schemes
 

diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml
@@ -2,53 +2,53 @@ name: Bug report
 description: Report something that is broken or incorrect
 labels: bug
 body:
-- type: textarea
-  id: description
-  attributes:
-    label: Description of the bug
-    description: A clear and concise description of what the bug is.
-  validations:
-    required: true
-- type: textarea
-  id: command_used
-  attributes:
-    label: Command used and terminal output
-    description: Steps to reproduce the behaviour. Please paste the command you used
-      to launch the pipeline and the output from your terminal.
-    render: console
-    placeholder: '$ nextflow run ...
-
-
-      Some output where something broke
-
-      '
-- type: textarea
-  id: files
-  attributes:
-    label: Relevant files
-    description: 'Please drag and drop the relevant files here. Create a `.zip` archive
-      if the extension is not allowed.
-
-      Your verbose log file `.nextflow.log` is often useful _(this is a hidden file
-      in the directory where you launched the pipeline)_ as well as custom Nextflow
-      configuration files.
-
-      '
-- type: textarea
-  id: system
-  attributes:
-    label: System information
-    description: '* Nextflow version _(eg. 23.04.0)_
-
-      * Hardware _(eg. HPC, Desktop, Cloud)_
-
-      * Executor _(eg. slurm, local, awsbatch)_
-
-      * Container engine: _(e.g. Docker, Singularity, Conda, Podman, Shifter, Charliecloud,
-      or Apptainer)_
-
-      * OS _(eg. CentOS Linux, macOS, Linux Mint)_
-
-      * Version of ebi-metagenomics/shallowmapping _(eg. 1.1, 1.5, 1.8.2)_
-
-      '
+  - type: textarea
+    id: description
+    attributes:
+      label: Description of the bug
+      description: A clear and concise description of what the bug is.
+    validations:
+      required: true
+  - type: textarea
+    id: command_used
+    attributes:
+      label: Command used and terminal output
+      description: Steps to reproduce the behaviour. Please paste the command you used
+        to launch the pipeline and the output from your terminal.
+      render: console
+      placeholder: "$ nextflow run ...
+
+
+        Some output where something broke
+
+        "
+  - type: textarea
+    id: files
+    attributes:
+      label: Relevant files
+      description: "Please drag and drop the relevant files here. Create a `.zip` archive
+        if the extension is not allowed.
+
+        Your verbose log file `.nextflow.log` is often useful _(this is a hidden file
+        in the directory where you launched the pipeline)_ as well as custom Nextflow
+        configuration files.
+
+        "
+  - type: textarea
+    id: system
+    attributes:
+      label: System information
+      description: "* Nextflow version _(eg. 23.04.0)_
+
+        * Hardware _(eg. HPC, Desktop, Cloud)_
+
+        * Executor _(eg. slurm, local, awsbatch)_
+
+        * Container engine: _(e.g. Docker, Singularity, Conda, Podman, Shifter, Charliecloud,
+        or Apptainer)_
+
+        * OS _(eg. CentOS Linux, macOS, Linux Mint)_
+
+        * Version of ebi-metagenomics/shallowmapping _(eg. 1.1, 1.5, 1.8.2)_
+
+        "
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
@@ -17,7 +17,8 @@ Learn more about contributing: [CONTRIBUTING.md](https://github.com/ebi-metageno
 - [ ] If you've fixed a bug or added code that should be tested, add tests!
 - [ ] If you've added a new tool - have you followed the pipeline conventions in the [contribution docs](https://github.com/ebi-metagenomics/shallowmapping/tree/master/.github/CONTRIBUTING.md)
 - [ ] Make sure your code lints (`nf-core lint`).
-- [ ] Ensure the test suite passes (`nextflow run . -profile test,docker --outdir <OUTDIR>`).
+- [ ] Ensure the test suite passes (`nf-test test main.nf.test -profile test,docker`).
+- [ ] Check for unexpected warnings in debug mode (`nextflow run . -profile debug,test,docker --outdir <OUTDIR>`).
 - [ ] Usage Documentation in `docs/usage.md` is updated.
 - [ ] Output Documentation in `docs/output.md` is updated.
 - [ ] `CHANGELOG.md` is updated.

diff --git a/.github/workflows/branch.yml b/.github/workflows/branch.yml
@@ -19,7 +19,7 @@ jobs:
       # NOTE - this doesn't currently work if the PR is coming from a fork, due to limitations in GitHub actions secrets
       - name: Post PR comment
         if: failure()
-        uses: mshick/add-pr-comment@v1
+        uses: mshick/add-pr-comment@b8f338c590a895d50bcbfa6c5859251edc8952fc # v2
         with:
           message: |
             ## This PR is against the `master` branch :x:

diff --git a/.github/workflows/ci.yml → .github/workflows/ci.yml.bak b/.github/workflows/ci.yml → .github/workflows/ci.yml.bak
diff --git a/.github/workflows/linting.yml b/.github/workflows/linting.yml
@@ -11,72 +11,33 @@ on:
     types: [published]
 
 jobs:
-  EditorConfig:
+  pre-commit:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4
 
-      - uses: actions/setup-node@v3
-
-      - name: Install editorconfig-checker
-        run: npm install -g editorconfig-checker
-
-      - name: Run ECLint check
-        run: editorconfig-checker -exclude README.md $(find .* -type f | grep -v '.git\|.py\|.md\|json\|yml\|yaml\|html\|css\|work\|.nextflow\|build\|nf_core.egg-info\|log.txt\|Makefile')
-
-  Prettier:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v3
-
-      - uses: actions/setup-node@v3
-
-      - name: Install Prettier
-        run: npm install -g prettier
-
-      - name: Run Prettier --check
-        run: prettier --check ${GITHUB_WORKSPACE}
-
-  PythonBlack:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v3
-
-      - name: Check code lints with Black
-        uses: psf/black@stable
-
-      # If the above check failed, post a comment on the PR explaining the failure
-      - name: Post PR comment
-        if: failure()
-        uses: mshick/add-pr-comment@v1
+      - name: Set up Python 3.11
+        uses: actions/setup-python@0a5c61591373683505ea898e09a3ea4f39ef2b9c # v5
         with:
-          message: |
-            ## Python linting (`black`) is failing
-
-            To keep the code consistent with lots of contributors, we run automated code consistency checks.
-            To fix this CI test, please run:
-
-            * Install [`black`](https://black.readthedocs.io/en/stable/): `pip install black`
-            * Fix formatting errors in your pipeline: `black .`
-
-            Once you push these changes the test should pass, and you can hide this comment :+1:
+          python-version: 3.11
+          cache: "pip"
 
-            We highly recommend setting up Black in your code editor so that this formatting is done automatically on save. Ask about it on Slack for help!
+      - name: Install pre-commit
+        run: pip install pre-commit
 
-            Thanks again for your contribution!
-          repo-token: ${{ secrets.GITHUB_TOKEN }}
-          allow-repeats: false
+      - name: Run pre-commit
+        run: pre-commit run --all-files
 
   nf-core:
     runs-on: ubuntu-latest
     steps:
       - name: Check out pipeline code
-        uses: actions/checkout@v3
+        uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4
 
       - name: Install Nextflow
         uses: nf-core/setup-nextflow@v1
 
-      - uses: actions/setup-python@v4
+      - uses: actions/setup-python@0a5c61591373683505ea898e09a3ea4f39ef2b9c # v5
         with:
           python-version: "3.11"
           architecture: "x64"
@@ -99,7 +60,7 @@ jobs:
 
       - name: Upload linting log file artifact
         if: ${{ always() }}
-        uses: actions/upload-artifact@v3
+        uses: actions/upload-artifact@5d5d22a31266ced268874388b861e4b58bb5c2f3 # v4
         with:
           name: linting-logs
           path: |

diff --git a/.github/workflows/linting_comment.yml b/.github/workflows/linting_comment.yml
@@ -11,7 +11,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Download lint results
-        uses: dawidd6/action-download-artifact@v2
+        uses: dawidd6/action-download-artifact@f6b0bace624032e30a85a8fd9c1a7f8f611f5737 # v3
         with:
           workflow: linting.yml
           workflow_conclusion: completed
@@ -21,7 +21,7 @@ jobs:
         run: echo "pr_number=$(cat linting-logs/PR_number.txt)" >> $GITHUB_OUTPUT
 
       - name: Post PR comment
-        uses: marocchino/sticky-pull-request-comment@v2
+        uses: marocchino/sticky-pull-request-comment@331f8f5b4215f0445d3c07b4967662a32a2d3e31 # v2
         with:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           number: ${{ steps.pr_number.outputs.pr_number }}

diff --git a/.gitignore b/.gitignore
@@ -1,6 +1,5 @@
 .nextflow*
 work/
-data/
 results/
 .DS_Store
 testing/

diff --git a/.nf-core.yml b/.nf-core.yml
@@ -1,32 +1,53 @@
 lint:
   files_exist:
-  - CODE_OF_CONDUCT.md
-  - assets/nf-core-shallowmapping_logo_light.png
-  - docs/images/nf-core-shallowmapping_logo_light.png
-  - docs/images/nf-core-shallowmapping_logo_dark.png
-  - .github/ISSUE_TEMPLATE/config.yml
-  - .github/workflows/awstest.yml
-  - .github/workflows/awsfulltest.yml
-  - conf/igenomes.config
-  - conf/igenomes.config
+    - CODE_OF_CONDUCT.md
+    - assets/nf-core-shallowmapping_logo_light.png
+    - docs/
+    - docs/output.md
+    - docs/README.md
+    - docs/README.md
+    - docs/usage.md
+    - docs/images/nf-core-shallowmapping_logo_dark.png
+    - docs/images/nf-core-shallowmapping_logo_light.png
+    - pyproject.toml
+    - .gitignore
+    - .github/ISSUE_TEMPLATE/config.yml
+    - .github/workflows/*
+    - .github/CONTRIBUTING.md
+    - .github/PULL_REQUEST_TEMPLATE.md
+    - conf/igenomes.config
+    - conf/igenomes.config
+    - conf/test.config
+    - conf/test_full.config
+    - .github/workflows/ci.yml
+    - lib/Utils.groovy
+    - lib/WorkflowMain.groovy
+    - lib/NfcoreTemplate.groovy
+    - lib/WorkflowShallowmapping.groovy
+    - lib/nfcore_external_java_deps.jar
   files_unchanged:
-  - CODE_OF_CONDUCT.md
-  - assets/nf-core-shallowmapping_logo_light.png
-  - docs/images/nf-core-shallowmapping_logo_light.png
-  - docs/images/nf-core-shallowmapping_logo_dark.png
-  - .github/ISSUE_TEMPLATE/bug_report.yml
+    - CODE_OF_CONDUCT.md
+    - assets/nf-core-shallowmapping_logo_light.png
+    - .github/ISSUE_TEMPLATE/bug_report.yml
+    - .github/CONTRIBUTING.md does not match the template
+    - .github/PULL_REQUEST_TEMPLATE.md does not match the template
+    - .github/workflows/branch.yml does not match the template
+    - .github/workflows/linting_comment.yml does not match the template
+    - .github/workflows/linting.yml does not match the template
+    - .gitignore does not match the template
+    - pyproject.toml does not match the template
   multiqc_config:
-  - report_comment
+    - report_comment
   nextflow_config:
-  - manifest.name
-  - manifest.homePage
-  - process.cpus
-  - process.memory
-  - process.time
-  - custom_config
+    - manifest.name
+    - manifest.homePage
+    - process.cpus
+    - process.memory
+    - process.time
+    - custom_config
 repository_type: pipeline
 template:
   prefix: ebi-metagenomics
   skip:
-  - igenomes
-  - nf_core_configs
+    - igenomes
+    - nf_core_configs
diff --git a/LICENSE b/LICENSE
@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) Alejandra Escobar
+Copyright (c) Microbiome Informatics team
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal