Merge pull request #194 from BU-ISCIII/main

Merge changes from main into develop
BU-ISCIII · Feb 1, 2024 · c1a51c1 · c1a51c1
2 parents f4efe59 + 3accca9
commit c1a51c1
Show file tree

Hide file tree

Showing 21 changed files with 768 additions and 117 deletions.
diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md
@@ -0,0 +1,72 @@
+# buisciii-tools: Contributing Guidelines
+
+## Contribution workflow
+
+If you'd like to write or modify some code for buisciii-tools, the standard workflow is as follows:
+
+1. Check that there isn't already an issue about your idea in the [buisciii-tools issues](https://github.com/BU-ISCIII/buisciii-tools/issues) to avoid duplicating work. **If there isn't one already, please create one so that others know you're working on this**.
+2. [Fork](https://help.github.com/en/github/getting-started-with-github/fork-a-repo) the [buisciii-tools repository](https://github.com/BU-ISCIII/buisciii-tools/) to your GitHub account.
+3. Make the necessary changes / additions within your forked repository following the [code style guidelines](#code-style-guidelines).
+4. Modify the [`CHANGELOG`](../CHANGELOG.md) file according to your changes in the appropiate section ([X.X.Xhot] or [X.X.Xdev]), you should register your changes regarding:
+   1. Added enhancements
+   2. Template changes
+   3. Fixes
+   4. Removed stuff
+   5. Requirements added or version update
+5. Update any documentation as needed.
+6. [Submit a Pull Request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request) against the `develop` or `hotfix` branch and send the url to the #pipelines-dev channel in slack (if you are not in the slack channel just wait fot the PR to be reviewed and rebased).
+
+If you're not used to this workflow with git, you can start with:
+
+- Some [docs in the bu-isciii wiki](https://github.com/BU-ISCIII/BU-ISCIII/wiki/Github--gitflow).
+- [some slides](https://docs.google.com/presentation/d/1PruqGxPQVxtNcuEbOd86mylXorgYIU5a/edit?pli=1#slide=id.p1) (in spanish).
+- some github generic docs [docs from GitHub](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests).
+- even their [excellent `git` resources](https://try.github.io/).
+
+### buisciii-tools repo branches
+
+buisciii-tools repo works with a three branching scheme. Two regular branches `main` and `develop`, and a third one created for hot fixes `hotfix`. This last one is created for changes in the **services templates**.
+
+- `main`: stable code only for releases.
+- `develop`: new code development for the diferente modules.
+- `hotfix`: bug fixing and/or templates addition/modification (bash scripts in: `templates` folder).
+
+You need to submit your PR always against `develop` or `hotfix` depending on the nature of your changes. Once approbed, this changes must be **`rebased`** so we do not create empty unwanted merges.
+
+## Tests
+
+When you create a pull request with changes, [GitHub Actions](https://github.com/features/actions) will run automatic tests.
+Typically, pull-requests are only fully reviewed when these tests are passing, though of course we can help out before then.
+
+There are typically two types of tests that run:
+
+### Lint tests
+
+We use black and flake8 linting based on PEP8 guidelines for python coding. You can check more information [here](https://github.com/BU-ISCIII/BU-ISCIII/wiki/Python#linting).
+
+### Code tests
+
+TODO. NOT YET IMPLEMENTED.
+Anyhow you should always submit locally tested code!!
+
+### New version bumping and release
+
+In order to create a new release you need to follow the next steps:
+
+1. Set the new version according to [semantic versioning](https://semver.org/), in our particular case, changes in the `hotfix` branch will change the PATCH version (third one), and changes in develop will typicaly change the MINOR version, unless the developing team decides otherwise.
+2. Create a PR bumping the new version against `hotfix` or `develop`. For bumping a new version just change [this line](https://github.com/BU-ISCIII/buisciii-tools/blob/615f1390d96cd6c8168acebc384289520a3cd728/setup.py#L5) with the new version.
+3. Once that PR is merged, create via web another PR against `main` (origin `develop` or `hotfix` accordingly). This PR would need 2 approvals.
+4. [Create a new release](https://docs.github.com/en/repositories/releasing-projects-on-github/managing-releases-in-a-repository) copying the appropiate notes from the `CHANGELOG`.
+5. Once the release is approved and merged, you're all set!
+
+PRs from one branch to another, like in a release should be **`merged`** not rebased, so we avoid conflicts and the branch merge is correctly visualize in the commits history.
+
+> A new PR for `develop` branch will be automatically generated if the changes came from `hotfix` so everything is properly sync.
+
+### Code style guidelines
+
+We follow PEP8 conventions as code style guidelines, please check [here](https://github.com/BU-ISCIII/BU-ISCIII/wiki/Python#pep-8-guidelines-read-the-full-pep-8-documentation) for more detail.
+
+## Getting help
+
+For further information/help, please ask on the  `#pipelines-dev` slack channel or write us an email! ([[email protected]](emailto:[email protected])).
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
@@ -0,0 +1,21 @@
+<!--
+# bu-isciii tools pull request
+
+Based on nf-core/viralrecon pull request template
+
+Fill in the appropriate checklist below and delete whatever is not relevant.
+
+PRs should be made against the develop of hotfix branch, unless you're preparing a software release.
+-->
+
+## PR checklist
+
+- [ ] This comment contains a description of changes (with reason).
+- [ ] Make sure your code lints (`black and flake8`).
+- If a new tamplate was added make sure:
+    - [ ] Template's schema is added in `templates/services.json`.
+    - [ ] Template's pipeline's documentation in `assets/reports/md/template.md` is added.
+    - [ ] Results Documentation in `assets/reports/results/template.md` is updated.
+- [ ] `CHANGELOG.md` is updated.
+- [ ] `README.md` is updated (including new tool citations and authors/contributors).
+- [ ] If you know a new user was added to the SFTP, make sure you added it to `templates/sftp_user.json`
diff --git a/.github/workflows/update_branches.yml b/.github/workflows/update_branches.yml
@@ -0,0 +1,30 @@
+name: Create Pull Request from main to develop and/or hotfix
+
+on:
+  push:
+    branches:
+      - main
+
+env: 
+  GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} # setting GH_TOKEN for the entire workflow
+
+jobs:
+  create-and-auto-merge-pr:
+    runs-on: ubuntu-latest
+
+    steps:
+    - name: Checkout code
+      uses: actions/checkout@v2
+
+    - name: Install GitHub CLI
+      run: |
+        sudo apt update
+        sudo apt install gh
+
+    - name: Create Pull Request to develop
+      run: |
+        gh pr create --base develop --head main --title "Merge changes from main into develop" --body "Automatically created pull request to merge changes from main into develop."
+
+    - name: Create Pull Request to hotfix
+      run: |
+        gh pr create --base hotfix --head main --title "Merge changes from main into hotfix" --body "Automatically created pull request to merge changes from main into hotfix."
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -0,0 +1,126 @@
+# bu-isciii tools Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [2.0.0dev] - 2024-0X-0X : https://github.com/BU-ISCIII/buisciii-tools/releases/tag/2.0.0
+
+### Credits
+
+Code contributions to the release:
+
+- [Sara Monzón](https://github.com/saramonzon)
+- [Sarai Varona](https://github.com/svarona)
+- [Pablo Mata](https://github.com/Shettland)
+- [Guillermo Gorines](https://github.com/GuilleGorines)
+
+
+### Template fixes and updates
+
+- Added templates:
+    - freebayes
+
+
+### Modules
+
+#### Added enhancements
+
+- Added credential parameters: --api_user, --api_password and --cred_file
+- Make modules to create folder's paths automatically from DB
+- Added finish module
+- Added json files: sftp_user.json
+- Added delivery jinja templates
+
+#### Fixes
+
+#### Changed
+
+- Fixed API requests to fit in the new database format
+- Updated README
+
+#### Removed
+
+### Requirements
+
+- Added PyYAML
+
+## [1.0.2hot] - 2024-0X-0X : https://github.com/BU-ISCIII/buisciii-tools/releases/tag/1.0.2
+
+### Credits
+
+Code contributions to the hotfix:
+
+### Template fixes and updates
+
+### Modules
+
+#### Added enhancements
+
+#### Fixes
+
+#### Changed
+
+#### Removed
+
+### Requirements
+
+## [1.0.1] - 2024-02-01 : https://github.com/BU-ISCIII/buisciii-tools/releases/tag/1.0.1
+
+### Credits
+
+Code contributions to the hotfix:
+
+- [Pablo Mata](https://github.com/Shettland)
+- [Jaime Ozaez](https://github.com/jaimeozaez)
+- [Sara Monzón](https://github.com/saramonzon)
+- [Sarai Varona](https://github.com/svarona)
+- [Daniel Valle](https://github.com/Daniel-VM)
+
+### Template fixes and updates
+
+- Added new line in `buisciii_tools/bu_isciii/templates/viralrecon/ANALYSIS/lablog_viralrecon`, in order to automatically rename `ANALYSIS0X_MAG` directory with the current date. 
+- Introduced handling of flu-C in `buisciii_tools/bu_isciii/templates/IRMA/ANALYSIS/ANALYSIS01_FLU_IRMA/04-irma/` `lablog` and `create_irma_stats.sh`
+- Small changes to `buisciii_tools/bu_isciii/templates/viralrecon/RESULTS/viralrecon_results` for blast and new excel_generator.py
+- Introduced better error handling in excel_generator.py. Now it can also be used for single files
+- Brought back `PASS_ONLY` to exometrio's `exomiser_configfile.yml`
+- [#187](https://github.com/BU-ISCIII/buisciii-tools/pull/187) - Added new template for bacterial assembly. Allowing for short, long and hybrid assembly.
+- [#190](https://github.com/BU-ISCIII/buisciii-tools/pull/190) - renamed some variables in create-summary_report from viralrecon template as their name was misleading and fixed a small typo in regex finding in excel_generator.py
+- [#192](https://github.com/BU-ISCIII/buisciii-tools/pull/192) - Small changes in excel_generator.py to automatically merge pangolin/nextclade tables when more than 1 reference is found
+
+### Modules
+
+#### Added enhancements
+- Added CHANGELOG
+- Added template for Pull Request
+- Added Contributing guidelines
+- Added github action to sync branches
+
+#### Fixes
+
+#### Changed
+
+#### Removed
+
+
+### Requirements
+
+
+## [1.0.0] - 2024-01-08 : https://github.com/BU-ISCIII/buisciii-tools/releases/tag/1.0.0
+
+### Credits
+
+Code contributions to the inital release:
+
+- [Sara Monzón](https://github.com/saramonzon)
+- [Saria Varona](https://github.com/svarona)
+- [Guillermo Gorines](https://github.com/GuilleGorines)
+- [Pablo Mata](https://github.com/Shettland)
+- [Luis Chapado](https://github.com/luissian)
+- [Erika Kvalem](https://github.com/ErikaKvalem)
+- [Alberto Lema](https://github.com/Alema91)
+- [Daniel Valle](https://github.com/Daniel-VM)
+- [Fernando Gomez](https://github.com/FGomez-Aldecoa)
+
diff --git a/bu_isciii/__main__.py b/bu_isciii/__main__.py
@@ -55,7 +55,7 @@ def run_bu_isciii():
     )
 
     # stderr.print("[green]                                          `._,._,'\n", highlight=False)
-    __version__ = "1.0.0"
+    __version__ = "1.0.1"
     stderr.print(
         "[grey39]    BU-ISCIII-tools version {}".format(__version__), highlight=False
     )

diff --git a/bu_isciii/autoclean_sftp.py b/bu_isciii/autoclean_sftp.py
@@ -120,9 +120,9 @@ def get_sftp_services(self):
                     # Get sftp-service last modification
                     service_finder = LastMofdificationFinder(sftp_service_fullPath)
                     service_last_modification = service_finder.find_last_modification()
-                    self.sftp_services[
-                        sftp_service_fullPath
-                    ] = service_last_modification
+                    self.sftp_services[sftp_service_fullPath] = (
+                        service_last_modification
+                    )
         if len(self.sftp_services) == 0:
             sys.exit(f"No services found in {self.path}")
 

diff --git a/bu_isciii/templates/IRMA/ANALYSIS/ANALYSIS01_FLU_IRMA/04-irma/create_irma_stats.sh b/bu_isciii/templates/IRMA/ANALYSIS/ANALYSIS01_FLU_IRMA/04-irma/create_irma_stats.sh
@@ -1 +1,33 @@
-echo -e "sample_ID\tTotalReads\tMappedReads\tFlu_type\tReads_HA\tReads_MP\tReads_NA\tReads_NP\tReads_NS\tReads_PA\tReads_PB1\tReads_PB2" > irma_stats.txt; cat ../samples_id.txt | while read in; do paste <(echo ${in}) <(grep '1-initial' ${in}/tables/READ_COUNTS.txt | cut -f2) <(grep '3-match' ${in}/tables/READ_COUNTS.txt | cut -f2) <(paste <(grep '4-[A-B]_HA' ${in}/tables/READ_COUNTS.txt | cut -f1 | cut -d '_' -f1,3 | cut -d '-' -f2) <(grep '4-[A-B]_NA' ${in}/tables/READ_COUNTS.txt | cut -f1 | cut -d '_' -f3) | tr '\t' '_') <(grep '4-[A-B]_HA' ${in}/tables/READ_COUNTS.txt | cut -f2) <(grep '4-[A-B]_MP' ${in}/tables/READ_COUNTS.txt | cut -f2) <(grep '4-[A-B]_NA' ${in}/tables/READ_COUNTS.txt | cut -f2) <(grep '4-[A-B]_NP' ${in}/tables/READ_COUNTS.txt | cut -f2) <(grep '4-[A-B]_NS' ${in}/tables/READ_COUNTS.txt | cut -f2) <(grep '4-[A-B]_PA' ${in}/tables/READ_COUNTS.txt | cut -f2) <(grep '4-[A-B]_PB1' ${in}/tables/READ_COUNTS.txt | cut -f2) <(grep '4-[A-B]_PB2' ${in}/tables/READ_COUNTS.txt | cut -f2); done >> irma_stats.txt
+
+echo -e "sample_ID\tTotalReads\tMappedReads\tFlu_type\tReads_HA\tReads_MP\tReads_NA\tReads_NP\tReads_NS\tReads_PA\tReads_PB1\tReads_PB2" > irma_stats.txt
+
+cat ../samples_id.txt | while read in 
+do 
+SAMPLE_ID=$(echo ${in})
+TOTAL_READS=$(grep '1-initial' ${in}/tables/READ_COUNTS.txt | cut -f2)
+MAPPEDREADS=$(grep '3-match' ${in}/tables/READ_COUNTS.txt | cut -f2)
+FLU_TYPE=$(paste <(grep '4-[A-C]_MP' ${in}/tables/READ_COUNTS.txt | cut -f1 | cut -d '_' -f1 | cut -d '-' -f2) <(grep '4-[A-B]_HA' ${in}/tables/READ_COUNTS.txt | cut -f1 | cut -d '_' -f3 | cut -d '-' -f2) <(grep '4-[A-B]_NA' ${in}/tables/READ_COUNTS.txt | cut -f1 | cut -d '_' -f3) | tr '\t' '_')
+HA=$(grep '4-[A-C]_HA' ${in}/tables/READ_COUNTS.txt | cut -f2)
+MP=$(grep '4-[A-C]_MP' ${in}/tables/READ_COUNTS.txt | cut -f2)
+NA=$(grep '4-[A-C]_NA' ${in}/tables/READ_COUNTS.txt | cut -f2)
+NP=$(grep '4-[A-C]_NP' ${in}/tables/READ_COUNTS.txt | cut -f2)
+NS=$(grep '4-[A-C]_NS' ${in}/tables/READ_COUNTS.txt | cut -f2)
+PA=$(grep '4-[A-C]_PA' ${in}/tables/READ_COUNTS.txt | cut -f2)
+PB1=$(grep '4-[A-C]_PB1' ${in}/tables/READ_COUNTS.txt | cut -f2)
+PB2=$(grep '4-[A-C]_PB2' ${in}/tables/READ_COUNTS.txt | cut -f2)
+#In case of Influenza C in samples:
+HE=$(grep '4-C_HE' ${in}/tables/READ_COUNTS.txt | cut -f2)
+if [[ -n "$HE" ]]; then
+    LINE=$(paste <(echo $SAMPLE_ID) <(echo $TOTAL_READS) <(echo $MAPPEDREADS) <(echo $FLU_TYPE) <(echo $HA) <(echo $MP) <(echo $NA) <(echo $NP) <(echo $NS) <(echo $PA) <(echo $PB1) <(echo $PB2) <(echo $HE))
+else
+    LINE=$(paste <(echo $SAMPLE_ID) <(echo $TOTAL_READS) <(echo $MAPPEDREADS) <(echo $FLU_TYPE) <(echo $HA) <(echo $MP) <(echo $NA) <(echo $NP) <(echo $NS) <(echo $PA) <(echo $PB1) <(echo $PB2))
+fi
+
+echo "$LINE" >> irma_stats.txt
+
+done
+
+ANY_C=$(grep "C_" irma_stats.txt)
+if [[ -n "$ANY_C" ]]; then
+    sed -i 's/Reads_PB2/Reads_PB2\tReads_HE/g' irma_stats.txt
+fi
diff --git a/bu_isciii/templates/IRMA/ANALYSIS/ANALYSIS01_FLU_IRMA/04-irma/lablog b/bu_isciii/templates/IRMA/ANALYSIS/ANALYSIS01_FLU_IRMA/04-irma/lablog
@@ -15,12 +15,21 @@ echo "cat HA_types.txt | while read in; do mkdir \${in}; done" >> _03_post_proce
 
 echo "mkdir B" >> _03_post_processing.sh
 
+echo "mkdir C" >> _03_post_processing.sh
+
 echo "ls */*.fasta | cut -d '/' -f2 | cut -d '.' -f1 | cut -d '_' -f1,2 | sort -u | grep 'A_' > A_fragment_list.txt" >> _03_post_processing.sh
 
 echo "ls */*.fasta | cut -d '/' -f2 | cut -d '.' -f1 | cut -d '_' -f1,2 | sort -u | grep 'B_' > B_fragment_list.txt" >> _03_post_processing.sh
 
-echo 'cat HA_types.txt | while read type; do grep ${type} irma_stats.txt | cut -f1 | while read sample; do cat A_fragment_list.txt | while read fragment; do if test -f ${sample}/${fragment}*.fasta; then cat ${sample}/${fragment}*.fasta | sed "s/^>/\>${sample}_/g" | sed 's/_H1//g' | sed 's/_H3//g' | sed 's/_N1//g' | sed 's/_N2//g'; fi >> ${type}/${fragment}.txt; done; done; done' >> _03_post_processing.sh
+echo "ls */*.fasta | cut -d '/' -f2 | cut -d '.' -f1 | cut -d '_' -f1,2 | sort -u | grep 'C_' > C_fragment_list.txt" >> _03_post_processing.sh
+
+echo 'cat HA_types.txt | while read type; do grep ${type} irma_stats.txt | cut -f1 | while read sample; do cat A_fragment_list.txt | while read fragment; do if test -f ${sample}/${fragment}*.fasta; then cat ${sample}/${fragment}*.fasta | sed "s/^>/\>${sample}_/g" | sed 's/_H1//g' | sed 's/_H3//g' | sed 's/_N1//g' | sed 's/_N2//g' | sed s@-@/@g | sed s/_A_/_/g ; fi >> ${type}/${fragment}.txt; done; done; done' >> _03_post_processing.sh
+
+echo 'grep -w 'B__' irma_stats.txt | cut -f1 | while read sample; do cat B_fragment_list.txt | while read fragment; do if test -f ${sample}/${fragment}*.fasta; then cat ${sample}/${fragment}*.fasta | sed "s/^>/\>${sample}_/g" | sed s/_H1//g | sed s/_H3//g | sed s/_N1//g | sed s/_N2//g | sed s@-@/@g | sed s/_B_/_/g ; fi >> B/${fragment}.txt; done; done' >> _03_post_processing.sh
+
+echo 'grep -w 'C__' irma_stats.txt | cut -f1 | while read sample; do cat C_fragment_list.txt | while read fragment; do if test -f ${sample}/${fragment}*.fasta; then cat ${sample}/${fragment}*.fasta | sed "s/^>/\>${sample}_/g" | sed s/_H1//g | sed s/_H3//g | sed s/_N1//g | sed s/_N2//g | sed s@-@/@g | sed s/_C_/_/g ; fi >> C/${fragment}.txt; done; done' >> _03_post_processing.sh
 
-echo 'grep -w 'B_' irma_stats.txt | cut -f1 | while read sample; do cat B_fragment_list.txt | while read fragment; do if test -f ${sample}/${fragment}*.fasta; then cat ${sample}/${fragment}*.fasta | sed "s/^>/\>${sample}_/g" | sed s/_H1//g | sed s/_H3//g | sed s/_N1//g | sed s/_N2//g; fi >> B/${fragment}.txt; done; done' >> _03_post_processing.sh
+echo 'cat ../samples_id.txt | while read in; do cat ${in}/*.fasta | sed "s/^>/\>${in}_/g" | sed 's/_H1//g' | sed 's/_H3//g' | sed 's/_N1//g' | sed 's/_N2//g' | sed 's@-@/@g' | 's/_A_/_/g' | sed 's/_B_/_/g' | sed 's/_C_/_/g' >> all_samples_completo.txt; done' >> _03_post_processing.sh
 
-echo 'cat ../samples_id.txt | while read in; do cat ${in}/*.fasta | sed "s/^>/\>${in}_/g" | sed 's/_H1//g' | sed 's/_H3//g' | sed 's/_N1//g' | sed 's/_N2//g' >> all_samples_completo.txt; done' >> _03_post_processing.sh
+echo 'sed -i "s/__//g" irma_stats.txt' >> _03_post_processing.sh
+echo 'sed -i "s/_\t/\t/g" irma_stats.txt' >> _03_post_processing.sh
diff --git a/bu_isciii/templates/IRMA/RESULTS/irma_results b/bu_isciii/templates/IRMA/RESULTS/irma_results
@@ -7,3 +7,4 @@ ln -s ../../ANALYSIS/*_MET/99-stats/multiqc_report.html ./krona_results.html
 ln -s ../../ANALYSIS/*FLU_IRMA/04-irma/all_samples_completo.txt .
 ln -s ../../ANALYSIS/*FLU_IRMA/04-irma/A_H* .
 ln -s ../../ANALYSIS/*FLU_IRMA/04-irma/B .
+ln -s ../../ANALYSIS/*FLU_IRMA/04-irma/C .