From 2e7b0cd98177162f0ec6df6b7f6a674eb6a70ba4 Mon Sep 17 00:00:00 2001 From: OPSergio Date: Thu, 26 Sep 2024 16:47:46 +0200 Subject: [PATCH 1/7] Corregidos los enlaces y archivos de redireccionamiento --- docs/markdown_files/configuration.md | 2 +- docs/markdown_files/description.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/markdown_files/configuration.md b/docs/markdown_files/configuration.md index 6c42e622..9d948c2d 100644 --- a/docs/markdown_files/configuration.md +++ b/docs/markdown_files/configuration.md @@ -163,4 +163,4 @@ displaying is limited to the samples that the laboratory that user belongs to. As the process of user creation is part of the maintenance activity we have create a separate chapter inside FAQ. To check the documentation for user creation -click on a new user now [How to create new user](createNewUserAccount.md) +click on a new user now [How to create new user](create_new_user.md) diff --git a/docs/markdown_files/description.md b/docs/markdown_files/description.md index 611323b7..3b664cc5 100644 --- a/docs/markdown_files/description.md +++ b/docs/markdown_files/description.md @@ -28,5 +28,5 @@ The website stores all the processed data and metadata, and allows the visualiza ## Relecov-tools -relecov tools is a python package with helper functions for data management in the project. Read more [here](relecov_tools.md) +relecov tools is a python package with helper functions for data management in the project. Read more [here](https://github.com/BU-ISCIII/buisciii-tools) From cab37a1597d6d4dddb329477010a03a497129aed Mon Sep 17 00:00:00 2001 From: OPSergio Date: Fri, 27 Sep 2024 15:51:01 +0200 Subject: [PATCH 2/7] Aplicados cambios asociados a las urls. modificadas en urls.py, documentationSideBar.html y en los distintos md --- docs/markdown_files/configuration.md | 2 +- docs/markdown_files/index.md | 8 ++-- docs/markdown_files/intranet_overview.md | 2 +- .../intranet_upload_metadata.md | 4 +- docs/markdown_files/metadata.md | 2 +- docs/markdown_files/metadata_lab_excel.md | 4 +- docs/markdown_files/upload_to_ena.md | 3 ++ docs/markdown_files/upload_to_gisaid.md | 3 ++ docs/markdown_files/variant_dashboard.md | 3 ++ docs/templates/docs/documentationSideBar.html | 44 +++++++++---------- docs/urls.py | 37 ++++++++-------- docs/views.py | 18 +++----- 12 files changed, 67 insertions(+), 63 deletions(-) diff --git a/docs/markdown_files/configuration.md b/docs/markdown_files/configuration.md index 9d948c2d..896c9428 100644 --- a/docs/markdown_files/configuration.md +++ b/docs/markdown_files/configuration.md @@ -163,4 +163,4 @@ displaying is limited to the samples that the laboratory that user belongs to. As the process of user creation is part of the maintenance activity we have create a separate chapter inside FAQ. To check the documentation for user creation -click on a new user now [How to create new user](create_new_user.md) +click on a new user now [How to create new user](../create_new_user.md) diff --git a/docs/markdown_files/index.md b/docs/markdown_files/index.md index d7d06155..b213563b 100644 --- a/docs/markdown_files/index.md +++ b/docs/markdown_files/index.md @@ -15,10 +15,10 @@ - [relecov-tools](relecov_tools.md) - Intranet - - [Overview](intranetOverview.md) - - [Upload Metadata](intranetUploadMetadata.md) - - [Contact data](intranetContactData.md) - - [Sample Processing](sampleProcessing.md) + - [Overview](intranet_overview.md) + - [Upload Metadata](intranet_upload_metadata.md) + - [Contact data](intranet_contact_data.md) + - [Sample Processing](intranet_received_samples.md) - Dashboards - [Variant dashboard](variant_dashboard.md) diff --git a/docs/markdown_files/intranet_overview.md b/docs/markdown_files/intranet_overview.md index 767b6204..f89f3a48 100644 --- a/docs/markdown_files/intranet_overview.md +++ b/docs/markdown_files/intranet_overview.md @@ -19,7 +19,7 @@ To access Intranet click on the "Intranet" button. If you do not have login credential yet, contact with your manager to request them for you. -For **admin** user follow the instructions provide in [Create new user](/documentation/createNewUSer) to add new user. +For **admin** user follow the instructions provide in [Create new user](../create_new_user.md) to add new user. --- diff --git a/docs/markdown_files/intranet_upload_metadata.md b/docs/markdown_files/intranet_upload_metadata.md index c562cdc3..fdb07841 100644 --- a/docs/markdown_files/intranet_upload_metadata.md +++ b/docs/markdown_files/intranet_upload_metadata.md @@ -12,7 +12,7 @@ The option 1 (Filling the information in metadata lab form) you do not need to p Option 2 (Upload Metadata Excel file). You need to fill in advance the Metadata Lab excel file -The last alternative (Using SFTP server) is not available from relecov. You need to upload the Excel metadata file to the sftp server. Detailed information how to do upload this file is described in [How to fill metadata lab excel](/documentation/metadataLabExcel) +The last alternative (Using SFTP server) is not available from relecov. You need to upload the Excel metadata file to the sftp server. Detailed information how to do upload this file is described in [How to fill metadata lab excel](../metadata_lab_excel.md) If you decide to use the alternative 1 or 2, click on the left side menu the "Upload Metadata" link, to display the page to upload your metadata. @@ -46,7 +46,7 @@ As you will see in this form there are 3 types of data: - Select. Press on the cell to display and choose the option that fit in your case. --- -If you have any doubt for any field you can have a look to [How to fill metadata lab excel](/documentation/metadataLabExcel) +If you have any doubt for any field you can have a look to [How to fill metadata lab excel](../metadata_lab_excel.md) --- diff --git a/docs/markdown_files/metadata.md b/docs/markdown_files/metadata.md index 732d394e..aded1ce7 100644 --- a/docs/markdown_files/metadata.md +++ b/docs/markdown_files/metadata.md @@ -26,4 +26,4 @@ Ph4age classification fields have been extended according to their nature: 8. Pathogen diagnostic testing 9. Public databases -An excel file has been generated including the fields belonging to the labs to facilitate its collection and upload to the platform. [See next section](metadata_lab_excel.md) +An excel file has been generated including the fields belonging to the labs to facilitate its collection and upload to the platform. [See next section](../metadata_lab_excel.md) diff --git a/docs/markdown_files/metadata_lab_excel.md b/docs/markdown_files/metadata_lab_excel.md index c7600560..b952e250 100644 --- a/docs/markdown_files/metadata_lab_excel.md +++ b/docs/markdown_files/metadata_lab_excel.md @@ -1,13 +1,13 @@ # Excel metadata: data provided by the microbiology labs ## Excel description -We provide an excel that gathers all the fields that need to be provided by the labs in the network. The excel can be downloaded from the platform intranet in the [Upload section](upload_metadata_lab.md). +We provide an excel that gathers all the fields that need to be provided by the labs in the network. The excel can be downloaded from the platform intranet in the [Upload section](../intranet_upload_metadata.md). - First sheet (Overview) shows the description of each field with an example of how it has to be filled, also indicating which fields are mandatory. - Second sheet (METADATA_LAB) needs is the sheet you need to modify. First line includes notes for clarification, second line includes an example, third line a description of the field, and forth line is the field name. You must not modify or delete any of this four lines, you only have to start filling the information of the first sample in the fifth line, creating one line for each sample. Fields with limited options have a dropdown list where you can select the appropiate value for your sample. Please use the provided options or ask for adding a new one. - Third sheet (DATA VALIDATION) includes the options available for each field. If you need additional options please contact us (bioinfomatica@isciii.es) -Your lab can filled this excel manually, most of the fields belong the same for most of your samples, as most of the fields describes the sequencing protocol. But, if your lab have programming skills and you want this excel to be filled automatically from other sources (other excel files, a database, etc) we provide a python module (*homogeneizer*) enclosed in [relecov-tools package](relecov_tools.md). +Your lab can filled this excel manually, most of the fields belong the same for most of your samples, as most of the fields describes the sequencing protocol. But, if your lab have programming skills and you want this excel to be filled automatically from other sources (other excel files, a database, etc) we provide a python module (*homogeneizer*) enclosed in [relecov-tools package](../relecov_tools.md). **Table 1.** Overview of fields that must be filled by the microbiology labs. diff --git a/docs/markdown_files/upload_to_ena.md b/docs/markdown_files/upload_to_ena.md index e69de29b..c30d7245 100644 --- a/docs/markdown_files/upload_to_ena.md +++ b/docs/markdown_files/upload_to_ena.md @@ -0,0 +1,3 @@ +# Under Development + +This page is currently under development. Please check back later. \ No newline at end of file diff --git a/docs/markdown_files/upload_to_gisaid.md b/docs/markdown_files/upload_to_gisaid.md index e69de29b..c30d7245 100644 --- a/docs/markdown_files/upload_to_gisaid.md +++ b/docs/markdown_files/upload_to_gisaid.md @@ -0,0 +1,3 @@ +# Under Development + +This page is currently under development. Please check back later. \ No newline at end of file diff --git a/docs/markdown_files/variant_dashboard.md b/docs/markdown_files/variant_dashboard.md index e69de29b..c30d7245 100644 --- a/docs/markdown_files/variant_dashboard.md +++ b/docs/markdown_files/variant_dashboard.md @@ -0,0 +1,3 @@ +# Under Development + +This page is currently under development. Please check back later. \ No newline at end of file diff --git a/docs/templates/docs/documentationSideBar.html b/docs/templates/docs/documentationSideBar.html index f0f20d90..42ead4b4 100644 --- a/docs/templates/docs/documentationSideBar.html +++ b/docs/templates/docs/documentationSideBar.html @@ -27,7 +27,7 @@
  • - + Relecov platform installation
    • @@ -50,13 +50,13 @@
  • - + How to fill metadata lab excel
  • - + relecov-tools
    • @@ -68,48 +68,48 @@
  • - + Overview
  • - + Upload Metadata
  • - + Contact data
  • - + Sample search
  • - + Received Samples

    • - +
    - Dashboards + Dashboards
  • - + Variant dashboard
  • - + Methodology dashboard
    • @@ -121,15 +121,15 @@
  • - + - Nexstrain installation + Nextstrain installation
  • - + - How to use RELECOV Nexstrain + How to use RELECOV Nextstrain

    • @@ -139,13 +139,13 @@
  • - + Upload to ENA
  • - + Upload to GISAID
    • @@ -157,14 +157,14 @@
  • - + API schema
  • - + How to use platform API @@ -173,11 +173,11 @@
    - FAQs + FAQs
  • - + Create New User diff --git a/docs/urls.py b/docs/urls.py index fb0bc22f..50f5e617 100644 --- a/docs/urls.py +++ b/docs/urls.py @@ -8,9 +8,9 @@ urlpatterns = [ path("", docs.views.index, name="index"), - path("description", docs.views.description, name="description"), + path("description/", docs.views.description, name="description"), path( - "relecovInstall/", + "relecov_install/", docs.views.relecov_install, name="relecov_install", ), @@ -21,69 +21,70 @@ ), path("metadata/", docs.views.metadata, name="metadata"), path( - "metadataLabExcel/", + "metadata_lab_excel/", docs.views.metadata_lab_excel, name="metadata_lab_excel", ), - path("relecovTools/", docs.views.relecov_tools, name="relecov_tools"), + path("relecov_tools/", docs.views.relecov_tools, name="relecov_tools"), path( - "intranetOverview/", + "intranet_overview/", docs.views.intranet_overview, name="intranet_overview", ), path( - "intranetContactData/", + "intranet_contact_data/", docs.views.intranet_contact_data, name="intranet_contact_data", ), path( - "intranetSampleSearch/", + "intranet_sample_search/", docs.views.intranet_sample_search, name="intranet_sample_search", ), path( - "intranetReceivedSamples/", + "intranet_received_samples/", docs.views.intranet_received_samples, name="intranet_received_samples", ), path( - "intranetUploadMetadata/", + "intranet_upload_metadata/", docs.views.intranet_upload_metadata, name="intranet_upload_metadata", ), path( - "variantDashboard/", + "variant_dashboard/", docs.views.variant_dashboard, name="variant_dashboard", ), path( - "methodologyDashboard/", + "methodology_dashboard/", docs.views.methodology_dashboard, name="methodology_dashboard", ), path( - "nextstrainInstall/", + "nextstrain_install/", docs.views.nextstrain_install, name="nextstrain_install", ), path( - "howtoNextstrain/", + "howto_nextstrain/", docs.views.howto_nextstrain, name="howto_nextstrain", ), - path("uploadToEna/", docs.views.upload_to_ena, name="upload_to_ena"), + path("upload_to_ena/", docs.views.upload_to_ena, name="upload_to_ena"), path( - "uploadToGisaid/", + "upload_to_gisaid/", docs.views.upload_to_gisaid, name="upload_to_gisaid", ), - path("apiSchema/", docs.views.api_schema, name="api_schema"), - path("howtoApi/", docs.views.howto_api, name="howto_api"), + path("api_schema/", docs.views.api_schema, name="api_schema"), + path("howto_api/", docs.views.howto_api, name="howto_api"), path( - "createNewUser/", + "create_new_user/", docs.views.create_new_user, name="create_new_user", ), ] + if settings.DEBUG: urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT) diff --git a/docs/views.py b/docs/views.py index 1ede5439..42801309 100644 --- a/docs/views.py +++ b/docs/views.py @@ -125,9 +125,7 @@ def intranet_contact_data(request): @login_required def intranet_sample_search(request): - converted_to_html = docs.utils.markdown.markdown_to_html( - "intranet_sample_search.md" - ) + converted_to_html = docs.utils.markdown.markdown_to_html("intranet_sample_search.md") if isinstance(converted_to_html, dict): return render(request, "docs/error_404.html") converted_to_html = docs.utils.markdown.fix_img_folder(converted_to_html) @@ -140,9 +138,7 @@ def intranet_sample_search(request): @login_required def intranet_received_samples(request): - converted_to_html = docs.utils.markdown.markdown_to_html( - "intranet_received_samples.md" - ) + converted_to_html = docs.utils.markdown.markdown_to_html("intranet_received_samples.md") if isinstance(converted_to_html, dict): return render(request, "docs/error_404.html") converted_to_html = docs.utils.markdown.fix_img_folder(converted_to_html) @@ -155,9 +151,7 @@ def intranet_received_samples(request): @login_required def intranet_upload_metadata(request): - converted_to_html = docs.utils.markdown.markdown_to_html( - "intranet_upload_metadata.md" - ) + converted_to_html = docs.utils.markdown.markdown_to_html("intranet_upload_metadata.md") if isinstance(converted_to_html, dict): return render(request, "docs/error_404.html") converted_to_html = docs.utils.markdown.fix_img_folder(converted_to_html) @@ -248,7 +242,7 @@ def upload_to_gisaid(request): @login_required def create_new_user_account(request): - converted_to_html = docs.utils.markdown.markdown_to_html("createNewUserAccount.md") + converted_to_html = docs.utils.markdown.markdown_to_html("create_new_user_account.md") if isinstance(converted_to_html, dict): return render(request, "docs/error_404.html") converted_to_html = docs.utils.markdown.fix_img_folder(converted_to_html) @@ -261,7 +255,7 @@ def create_new_user_account(request): @login_required def upload_metadata(request): - converted_to_html = docs.utils.markdown.markdown_to_html("uploadMetadata.md") + converted_to_html = docs.utils.markdown.markdown_to_html("upload_metadata.md") if isinstance(converted_to_html, dict): return render(request, "docs/error_404.html") converted_to_html = docs.utils.markdown.fix_img_folder(converted_to_html) @@ -300,7 +294,7 @@ def howto_api(request): @login_required def create_new_user(request): - converted_to_html = docs.utils.markdown.markdown_to_html("createNewUserAccount.md") + converted_to_html = docs.utils.markdown.markdown_to_html("create_new_user.md") if isinstance(converted_to_html, dict): return render(request, "docs/error_404.html") converted_to_html = docs.utils.markdown.fix_img_folder(converted_to_html) From f77f059b3d9fae461cda273b53e6c90f82042562 Mon Sep 17 00:00:00 2001 From: OPSergio Date: Fri, 27 Sep 2024 16:05:32 +0200 Subject: [PATCH 3/7] Switching to black format --- docs/views.py | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/docs/views.py b/docs/views.py index 42801309..c48fcbfb 100644 --- a/docs/views.py +++ b/docs/views.py @@ -125,7 +125,9 @@ def intranet_contact_data(request): @login_required def intranet_sample_search(request): - converted_to_html = docs.utils.markdown.markdown_to_html("intranet_sample_search.md") + converted_to_html = docs.utils.markdown.markdown_to_html( + "intranet_sample_search.md" + ) if isinstance(converted_to_html, dict): return render(request, "docs/error_404.html") converted_to_html = docs.utils.markdown.fix_img_folder(converted_to_html) @@ -138,7 +140,9 @@ def intranet_sample_search(request): @login_required def intranet_received_samples(request): - converted_to_html = docs.utils.markdown.markdown_to_html("intranet_received_samples.md") + converted_to_html = docs.utils.markdown.markdown_to_html( + "intranet_received_samples.md" + ) if isinstance(converted_to_html, dict): return render(request, "docs/error_404.html") converted_to_html = docs.utils.markdown.fix_img_folder(converted_to_html) @@ -151,7 +155,9 @@ def intranet_received_samples(request): @login_required def intranet_upload_metadata(request): - converted_to_html = docs.utils.markdown.markdown_to_html("intranet_upload_metadata.md") + converted_to_html = docs.utils.markdown.markdown_to_html( + "intranet_upload_metadata.md" + ) if isinstance(converted_to_html, dict): return render(request, "docs/error_404.html") converted_to_html = docs.utils.markdown.fix_img_folder(converted_to_html) @@ -242,7 +248,9 @@ def upload_to_gisaid(request): @login_required def create_new_user_account(request): - converted_to_html = docs.utils.markdown.markdown_to_html("create_new_user_account.md") + converted_to_html = docs.utils.markdown.markdown_to_html( + "create_new_user_account.md" + ) if isinstance(converted_to_html, dict): return render(request, "docs/error_404.html") converted_to_html = docs.utils.markdown.fix_img_folder(converted_to_html) From 5773cb4eae0efdf7b0dc32642512f0b4bdb7979a Mon Sep 17 00:00:00 2001 From: OPSergio Date: Mon, 30 Sep 2024 10:36:01 +0200 Subject: [PATCH 4/7] Remove duplicate code --- docs/views.py | 15 --------------- 1 file changed, 15 deletions(-) diff --git a/docs/views.py b/docs/views.py index c48fcbfb..db25549c 100644 --- a/docs/views.py +++ b/docs/views.py @@ -246,21 +246,6 @@ def upload_to_gisaid(request): ) -@login_required -def create_new_user_account(request): - converted_to_html = docs.utils.markdown.markdown_to_html( - "create_new_user_account.md" - ) - if isinstance(converted_to_html, dict): - return render(request, "docs/error_404.html") - converted_to_html = docs.utils.markdown.fix_img_folder(converted_to_html) - return render( - request, - "docs/base.html", - {"html": converted_to_html}, - ) - - @login_required def upload_metadata(request): converted_to_html = docs.utils.markdown.markdown_to_html("upload_metadata.md") From d7b9566f78ac12b00d1d8c7746a6a7199fbc2408 Mon Sep 17 00:00:00 2001 From: OPSergio Date: Mon, 30 Sep 2024 10:39:55 +0200 Subject: [PATCH 5/7] Change link --- docs/markdown_files/description.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/markdown_files/description.md b/docs/markdown_files/description.md index 3b664cc5..abc8845f 100644 --- a/docs/markdown_files/description.md +++ b/docs/markdown_files/description.md @@ -28,5 +28,5 @@ The website stores all the processed data and metadata, and allows the visualiza ## Relecov-tools -relecov tools is a python package with helper functions for data management in the project. Read more [here](https://github.com/BU-ISCIII/buisciii-tools) +relecov tools is a python package with helper functions for data management in the project. Read more [here](../relecov_tools.md) From 7aa240265afb37c6a2fe01962a17d4cb6184988f Mon Sep 17 00:00:00 2001 From: OPSergio Date: Mon, 30 Sep 2024 11:31:13 +0200 Subject: [PATCH 6/7] Update relecov_tools.md --- docs/markdown_files/relecov_tools.md | 479 +++++++++++++++------------ 1 file changed, 261 insertions(+), 218 deletions(-) diff --git a/docs/markdown_files/relecov_tools.md b/docs/markdown_files/relecov_tools.md index 740b871e..4f40ea2c 100644 --- a/docs/markdown_files/relecov_tools.md +++ b/docs/markdown_files/relecov_tools.md @@ -1,186 +1,90 @@ # relecov-tools +[![python_lint](https://github.com/BU-ISCIII/relecov-tools/actions/workflows/python_lint.yml/badge.svg)](https://github.com/BU-ISCIII/relecov-tools/actions/workflows/python_lint.yml) +[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black) + +> THIS REPO IS UNDER ACTIVE DEVELOPMENT. + +relecov-tools is a set of helper tools for the assembly of the different elements in the RELECOV platform (Spanish Network for genomic surveillance of SARS-Cov-2) as data download, processing, validation and upload to public databases, as well as analysis runs and database storage. + ## Table of contents -1. [Overview](#overview) - 1. [command-line](#command-line) - 2. [package import](#package-import) -2. [Modules](#modules) - 2. [homogeneizer](#homogeneizer) - 3. [download](#download) - 4. [read-metadata](#read-metadata) - 5. [validate](#validate) - 6. [map](#map) - 7. [upload-to-ena](#upload-to-ena) - 8. [upload-to-gisaid](#upload-to-gisaid) - 9. [update-db](#update-db) - 10. [launch](#launch) +- [relecov-tools](#relecov-tools) + - [Table of contents](#table-of-contents) + - [Installation](#installation) + - [Bioconda](#bioconda) + - [Pip](#pip) + - [Development version](#development-version) + - [Usage](#usage) + - [Command-line](#command-line) + - [download](#download) + - [read-lab-metadata](#read-lab-metadata) + - [read-bioinfo-metadata](#read-bioinfo-metadata) + - [validate](#validate) + - [map](#map) + - [upload-to-ena](#upload-to-ena) + - [upload-to-gisaid](#upload-to-gisaid) + - [update-db](#update-db) + - [launch-pipeline](#launch-pipeline) + - [logs-to-excel](#logs-to-excel) + - [build-schema](#build-schema) + - [Mandatory Fields](#mandatory-fields) + - [custom logs](#custom-logs) + - [Python package mode](#python-package-mode) + - [Acknowledgements](#acknowledgements) + +## Installation + +### Bioconda +Soon + +### Pip +relecov-tools is available in Pypi and can be installed via pip: +``` +pip install relecov-tools +``` +### Development version +If you want to install the latest code in the repository: -## Overview -[relecov-tools](https://github.com/BU-ISCIII/relecov-tools) is a set of helper tools for the assembly of the different elements in the RELECOV platform (Spanish Network for genomic surveillance of SARS-Cov-2) as data download, processing, validation and upload to public databases, as well as analysis runs and database storage. +``` +conda create -n relecov_dev pip +pip install --force-reinstall --upgrade git+https://github.com/bu-isciii/relecov-tools.git@develop +``` -relecov-tools is designed as a python package, it can be installed both via conda or pip, and it can be used as a command line tool or importing it in any python program as a normal package. +## Usage ### Command-line relecov-tools provides a command-line version with help descriptions and params prompt if needed. ``` $ relecov-tools --help - ___ ___ ___ ___ ___ - \ |--| | \ | | | | | | \ / - \ \ / |__ / |__ | |___ | | | \ / - / / \ | \ | | | | | | \ / - / |--| | \ |___ |___ |___ |___ |___| \/ - RELECOV-tools version 0.0.1 - Usage: relecov-tools [OPTIONS] COMMAND [ARGS]... - - Options: - --version Show the version and exit. - -v, --verbose Print verbose output to the console. - -l, --log-file Save a verbose log to a file. - --help Show this message and exit. - - Commands: - download Download files located in sftp server. - read-metadata Create the json compliant to the relecov schema from... - validate Validate json file against schema. - map Convert data between phage plus schema to ENA,... - upload-to-ena parsed data to create xml files to upload to ena - upload-to-gisaid parsed data to create files to upload to gisaid - launch launch viralrecon in hpc - update-db feed database with metadata jsons -``` - -### Package import -relecov-tools is designed in a way that you can use import the different modules and use them in your own scripts, for example: - + ___ ___ ___ ___ ___ +\ |--| | \ | | | | | | \ / +\ \ / |__ / |__ | |___ | | | \ / +/ / \ | \ | | | | | | \ / +/ |--| | \ |___ |___ |___ |___ |___| \/ +RELECOV-tools version 1.1.0 +Usage: relecov-tools [OPTIONS] COMMAND [ARGS]... + +Options: +--version Show the version and exit. +-v, --verbose Print verbose output to the console. +-l, --log-file Save a verbose log to a file. +--help Show this message and exit. + +Commands: + download Download files located in sftp server. + read-lab-metadata Create the json compliant to the relecov schema from... + read-bioinfo-metadata Create the json compliant to the relecov schema with Bioinfo Metadata. + validate Validate json file against schema. + map Convert data between phage plus schema to ENA,... + upload-to-ena parsed data to create xml files to upload to ena + upload-to-gisaid parsed data to create files to upload to gisaid + update-db feed database with metadata jsons + build-schema Generates and updates JSON Schema files from... + launch-pipeline Create the symbolic links for the samples which... ``` -import relecov_tools.sftp_handle -user="admin" -passwd="1234" -conf_file="/path/to/conf" - -sftp_connection = relecov_tools.sftp_handle.SftpHandle( - user, password, conf_file - ) - sftp_connection.download() -``` - -We'll provide more docs as soon as posible. - -## Modules -It consists in a series of modules with different purposes as described below. - -![overview-tools](img/overview_tools.png) - -#### homogeneizer -The command `metadata-homogeneizer` allows parsing one or multiple files that containing required information to generate the metadata excel file required to upload data to the platform. - -In order to use the module you have to prepare two files: - -1. First a config file indicating the files, mapping keys and require procesing. - -``` -{ - ## Python script with processing functions - "python_file": "ISCIII.py", - "required_files": { - # N required files. - # Each file has several fields: file_name, mapped_fields, - # and function if any additional processing is needed. - "metadata_file": { - # file name - "file_name": "new_lab_metadata.xlsx", - # how the columns in the metadata_lab_excel map against YOUR excel. - "mapped_fields": { - # column name in metadat excel lab : column name in YOUR excel/file. - "Public Health sample id (SIVIES)": "Código SiViEs", - "Sample ID given by originating laboratory": "Ref Hospital", - "Specimen source": "Muestra", - }, - # Name of the function in ISCIII.py that is needed for fields processing. - "function": "None" - }, - # Example for adding information in other file. - # Here is a json that has the fastq filenames info that are going to be added to the excel. - "sample_file": { - "file_name": "sample_data.json", - "mapped_fields": { - "Sequence file R1 fastq": "sequence_file_R1_fastq", - "Sequence file R2 fastq": "sequence_file_R2_fastq" - }, - "function": "None", - # Column used for adding the new two fields. - "mapped_key": "Sample ID given for sequencing" - }, - # Example of field fixing based on another file. - # The names of the originating labs does not match the options - # available in the metadata lab excel so we have a file to correct that. - "org_lab_file": { - "file_name": "laboratory_mapping_list.tsv", - "mapped_fields": { - "Originating Laboratory": "mapped_org_lab" - }, - "mapped_key": "", - "function": "replace_originating_lab" - }, - ## Example: just translating a field to english. - "host_gender_file": { - "file_name": "", - "mapped_fields": { - "Host Gender": "" - }, - "mapped_key": "", - "function": "translate_gender_to_english" - }, - }, - # Finally these are fields that are fixed/same for all samples. - "fixed_fields": { - "Submitting Institution": "Instituto de Salud Carlos III", - "Biological Sample Storage Condition": "-80 C", - "Purpose of sampling": "Surveillance", - "Host": "Human", - "Library Preparation Kit": "Illumina DNA Prep", - "Enrichment Protocol": "Amplicon", - } -} -``` - -2. A python script with your functions for additional processing that should look like this:i - -``` -def translate_gender_to_english(metadata, f_data, mapped_fields, heading): - """Translate into english the host gender that is written in spanish""" - map_dict = { - "hombre": "Male", - "mujer": "Female", - "genero no-binario": "Non-binary Gender", - "genero no-binario": "Non-binary Gender", - "desconocido": "Not Provided", - "unknown": "Not Provided", - } - for row in metadata[1:]: - for key, val in mapped_fields.items(): - m_idx = heading.index(key) - if row[m_idx] is None or row[m_idx] == "": - row[m_idx] = "Not Provided" - continue - item = row[m_idx].lower() - if item in map_dict: - row[m_idx] = map_dict[item] - else: - log.error("The '%s' is not a valid data for translation", row[m_idx]) - stderr.print( - "f[red] The '{row[m_idx]}' is not a valid data for translation" - ) - sys.exit(1) - return metadata -``` - -A complete example of the config file can be found [here](https://github.com/BU-ISCIII/relecov-tools/blob/develop/relecov_tools/schema/institution_schemas/ISCIII.json) and a functioning script file for an instution can be found [here](https://github.com/BU-ISCIII/relecov-tools/blob/develop/relecov_tools/institution_scripts/ISCIII.py) - - #### download The command `download` connects to a transfer protocol (currently sftp) and downloads all files in the different available folders in the passed credentials. In addition, it checks if the files in the current folder match the files in the metadata file and also checks if there are md5sum for each file. Else, it creates one before storing in the final repository. @@ -190,19 +94,24 @@ Usage: relecov-tools download [OPTIONS] Download files located in sftp server. - Options: - -u, --user TEXT User name for login to sftp server - -p, --password TEXT password for the user to login - -f, --conf_file TEXT Configuration file in yaml format (no params file) - --help Show this message and exit. + Options: + -u, --user User name for login to sftp server + -p, --password Password for the user to login + -d, --download_option Select the download option: [download_only, download_clean, delete_only]. + download_only will only download the files + download_clean will remove files from sftp after download + delete_only will only delete the files + -o, --output_location Flag: Select location for downloaded files, overrides config file location + -t, --target_folders Flag: Select which sftp folders will be targeted giving [paths] or via prompt + -f, --conf_file Configuration file in yaml format (no params file) + --help Show this message and exit. ``` Configuration can be passed in several formats: - - if no config_file is passed, default values are fetched from conf/configuration.json, and user and password are asked in prompt. - - Default values can be overwritten using a yml config file, so you can input user, password, sftp_server, etc. +- if no config_file is passed, default values are fetched from conf/configuration.json, and user and password are asked in prompt. +- Default values can be overwritten using a yml config file, so you can input user, password, sftp_server, etc. Config file example with all available options: - ``` sftp_server: "sftprelecov.isciii.es" sftp_port: "22" @@ -215,40 +124,58 @@ allowed_sample_extensions: - .fasta ``` -#### read-metadata -`read-metadata` command reads the excel file with laboratory metadata and processes it adding additional needed fields. +#### read-lab-metadata +`read-lab-metadata` command reads the excel file with laboratory metadata and processes it adding additional needed fields. ``` -$ relecov-tools read-metadata --help +$ relecov-tools read-lab-metadata --help Usage: relecov-tools read-metadata [OPTIONS] - Create the json compliant to the relecov schema from the Metadata file. + Create the json compliant to the relecov schema from the Metadata file. - Options: - -m, --metadata_file PATH file containing metadata in xlsx format. - -s, --sample_list_file PATH Json with the additional metadata to add to the - received user metadata. - -o, --metadata-out PATH Path to save output metadata file in json format. - --help Show this message and exit. + Options: + -m, --metadata_file PATH file containing metadata in xlsx format. + -s, --sample_list_file PATH Json with the additional metadata to add to the + received user metadata. + -o, --metadata-out PATH Path to save output metadata file in json format. + --help Show this message and exit. ``` + An example for the metadata excel file can be found [here](./relecov_tools/example_data/METADATA_LAB_TEST.xlsx) +#### read-bioinfo-metadata +`read-bioinfo-metadata` Include the results from the Bioinformatics analysis into the Json previously created with read-lab-metadata module. + +``` +$ relecov-tools read-bioinfo-metadata --help +Usage: relecov-tools read-bioinfo-metadata [OPTIONS] + + Create the json compliant to the relecov schema with Bioinfo Metadata. + + Options: + -j, --json_file Json file containing lab metadata + -i, --input_folder Path to folder containing analysis results + -s, --software_name Name of the software employed in the bioinformatics analysis (default: viralrecon). + -o, --out_dir Path to save output file" +``` +- Note: Software-specific configurations are available in [bioinfo_config.json](./relecov_tools/conf/bioinfo_config.json). + #### validate -`validate` commands validate the data in json format outputted by `read-metadata` command against a json schema, in this case the relecov [schema specification](./relecov_tools/schema/relecov_schema.json). +`validate` commands validate the data in json format outputted by `read-metadata` command against a json schema, in this case the relecov [schema specification](./relecov_tools/schema/relecov_schema.json). It also creates a summary of the errors and warnings found in excel format as a report to the users. ``` $ relecov-tools validate --help Usage: relecov-tools validate [OPTIONS] - Validate json file against schema. + Validate json file against schema. - Options: - -j, --json_file TEXT Json file to validate - -s, --json_schema TEXT Json schema - -m, --metadata PATH Origin file containing metadata - -o, --out_folder TEXT Path to save validate json file - --help Show this message and exit. + Options: + -j, --json_file TEXT Json file to validate + -s, --json_schema TEXT Json schema (default: relecov-schema) + -m, --metadata PATH Origin file containing metadata + -o, --out_folder TEXT Path to save validate json file + --help Show this message and exit. ``` @@ -259,44 +186,160 @@ The command `map` converts a data in json format from relecov data model to ena $ relecov-tools map --help Usage: relecov-tools map [OPTIONS] -Convert data between phage plus schema to ENA, GISAID, or any other schema + Convert data between phage plus schema to ENA, GISAID, or any other schema - Options: - -p, --origin_schema TEXT File with the origin (relecov) schema - -j, --json_data TEXT File with the json data to convert - -d, --destination_schema [ENA|GSAID|other] - schema to be mapped - -f, --schema_file TEXT file with the custom schema - -o, --output TEXT File name and path to store the mapped json - --help Show this message and exit. + Options: + -p, --origin_schema TEXT File with the origin (relecov) schema + -j, --json_data TEXT File with the json data to convert + -d, --destination_schema [ENA|GSAID|other] + schema to be mapped + -f, --schema_file TEXT file with the custom schema + -o, --output TEXT File name and path to store the mapped json + --help Show this message and exit. ``` #### upload-to-ena `upload-to-ena` command uses json data mapped to ena schema to use the [ena_upload_cli](https://github.com/usegalaxy-eu/ena-upload-cli) package to upload raw data and metadata to ENA db. - ``` Usage: relecov-tools upload-to-ena [OPTIONS] -Parse metadata to create xml files to upload to ena + parsed data to create xml files to upload to ena + + Options: + -u, --user user name for login to ena + -p, --password password for the user to login + -c, --center center name + -e, --ena_json where the validated json is + -t, --template_path path to folder containing ENA xml templates + -a, --action select one of the available options: [add|modify|cancel|release] + --dev Flag: Test submission + --upload_fastq Flag: Upload fastq files. Mandatory for "add" action + -m", --metadata_types List of metadata xml types to submit [study,experiment,run,sample] + -o, --output_path TEXT output folder for the xml generated files + --help Show this message and exit. - Options: - -u, --user TEXT user name for login to ena - -p, --password TEXT password for the user to login - -e, --ena_json TEXT where the validated json is - -s, --study TEXT study/project name to include in xml files - -a, --action [add|modify|cancel|release] select one of the available options - --dev / --production - -o, --output_path TEXT output folder for the xml generated files - --help Show this message and exit. ``` #### upload-to-gisaid -SOON! +`upload-to-gisaid` uses the json mapped to gisaid schema to upload raw data and metadata to GISAID db + +``` +Usage: relecov-tools upload-to-gisaid [OPTIONS] + + parsed data to create xml files to upload to ena + + Options: + -u, --user user name for login + -p, --password password for the user to login + -c, --client_id client-ID provided by clisupport@gisaid.org + -t, --token path to athentication token + -e, --gisaid_json path to validated json mapped to GISAID + -i, --input_path path to fastas folder or multifasta file + -f, --frameshift frameshift notification: ["catch_all", "catch_none", "catch_novel"] + -x, --proxy_config introduce your proxy credentials as: username:password@proxy:port + --single Flag: input is a folder with several fasta files. + --gzip Flag: input fasta is gziped. +``` #### update-db + -u, --user user name for login + -p, --password password for the user to login + -t, --type Select the type of information to upload to database [sample,bioinfodata,variantdata] + -d, --databaseServer Name of the database server receiving the data [iskylims,relecov] + +#### launch-pipeline +Create the folder structure to execute the given pipeline for the latest sample batches after executing download, read-lab-metadata and validate modules. This module will create symbolic links for each sample and generate the necessary files for pipeline execution using the information from validated_BATCH-NAME_DATE.json. +``` +Usage: relecov-tools launch-pipeline [OPTIONS] + + Create the symbolic links for the samples which are validated to prepare for + bioinformatics pipeline execution. + +Options: + -i, --input PATH Path to the input folder where sample files are located + -t, --template PATH Path to the pipeline template folder to be copied in the output folder + -c, --config PATH Path to the the template config file + -o, --out_dir PATH Path to output folder + --help Show this message and exit. +``` + +#### logs-to-excel +Creates an xlsx file with all the entries found for a specified laboratory in a given set of log_summary.json files (from log-summary module). The laboratory name must match the name of one of the keys in the provided logs to work. +``` +Usage: relecov-tools logs-to-excel [OPTIONS] + + Creates a merged xlsx report from all the log summary jsons given as input + +Options: + -l, --lab_name Name for target laboratory in log-summary.json files + -o, --output_folder Path to output folder where xlsx file is saved + -f, --files Paths to log_summary.json files to merge into xlsx file, called once per file +``` + +### build-schema +The `build-schema` module provides functionality to generate and manage JSON Schema files based on database definitions from Excel spreadsheets. It automates the creation of JSON Schemas, including validation, drafting, and comparison with existing schemas. + +``` +Usage: relecov-tools build-schema [OPTIONS] + + Generates and updates JSON Schema files from Excel-based database + definitions. + +Options: + -i, --input_file PATH Path to the Excel document containing the database + definition. This file must have a .xlsx extension. + [required] + -s, --schema_base PATH Path to the base schema file. This file is used as + a reference to compare it with the schema + generated using this module. (Default: installed + schema in 'relecov- + tools/relecov_tools/schema/relecov_schema.json') + -v, --draft_version TEXT Version of the JSON schema specification to be + used. Example: '2020-12'. See: https://json- + schema.org/specification-links + -d, --diff BOOLEAN Prints a changelog/diff between the base and + incoming versions of the schema. + -o, --out_dir PATH Path to save output file/s + --help Show this message and exit. +``` + +#### Mandatory Fields +Ensure that the fields below are properly defined as headers in your Excel sheet (database definition): + +``` +enum: List of possible values for enumeration. +examples: Example values for the property. +ontology_id: Identifier for ontology. +type: Data type of the property (e.g., string, integer). +description: Description of the property. +classification: Classification or category of the property. +label_name: Label or name for the property. +fill_mode: Mode for filling in the property (e.g., required, optional). +required (Y/N): Indicates if the property is required (Y) or optional (N). +complex_field (Y/N): Indicates if the property is a complex (nested) field (Y) or a standard field (N). +``` + +#### custom logs +After executing each of these modules, you may find a custom log report in json format named "DATE_EXECUTED-MODULE_log_summary.json. These custom log summaries can be useful to detect errors in metadata in order to fix them and/or notify the users. + +### Python package mode +relecov-tools is designed in a way that you can use import the different modules and use them in your own scripts, for example: + +``` +import relecov_tools.sftp_handle +user="admin" +passwd="1234" +conf_file="/path/to/conf" + +sftp_connection = relecov_tools.sftp_handle.SftpHandle( + user, password, conf_file +) +sftp_connection.download() +``` +DOCs soon!! -#### Launc -SOON! +## Acknowledgements +Python package idea and design is really inspired in [nf-core/tools](https://github.com/nf-core/tools). \ No newline at end of file From a6002b24782b6c8cb5b3385023aac221c10cbf71 Mon Sep 17 00:00:00 2001 From: OPSergio Date: Mon, 30 Sep 2024 11:52:08 +0200 Subject: [PATCH 7/7] Replaced the first lines with the link to the github repository. --- docs/markdown_files/relecov_tools.md | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/docs/markdown_files/relecov_tools.md b/docs/markdown_files/relecov_tools.md index 4f40ea2c..5bb77a2b 100644 --- a/docs/markdown_files/relecov_tools.md +++ b/docs/markdown_files/relecov_tools.md @@ -1,9 +1,6 @@ # relecov-tools -[![python_lint](https://github.com/BU-ISCIII/relecov-tools/actions/workflows/python_lint.yml/badge.svg)](https://github.com/BU-ISCIII/relecov-tools/actions/workflows/python_lint.yml) -[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black) - -> THIS REPO IS UNDER ACTIVE DEVELOPMENT. +Find more information about the tools here: https://github.com/BU-ISCIII/relecov-tools relecov-tools is a set of helper tools for the assembly of the different elements in the RELECOV platform (Spanish Network for genomic surveillance of SARS-Cov-2) as data download, processing, validation and upload to public databases, as well as analysis runs and database storage.