diff --git a/docs/architecture/ara/index.md b/docs/architecture/ara/index.md index b75032b..0f6d10b 100644 --- a/docs/architecture/ara/index.md +++ b/docs/architecture/ara/index.md @@ -8,13 +8,13 @@ Translator "Autonomous Relay Agents" (ARAs) build build upon the knowledge contr - [Autonomous Relay Agent usage tutorials](../../development-guide/tutorials/index.md) -## Existing Translator ARA's +## Existing Translator ARAs | Team | Name | Documentation | Github Repository | | ------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------- | | [Ranking Agent](../../teams/ranking-agent.md) | Aragorn | [Aragorn Page](aragorn.md) | [ranking-agent/aragorn](https://github.com/ranking-agent/aragorn) | -| [Expander Agent](../../teams/expander-agent.md) | ARAX | [ARAX Page](arax.md)
[ARAX Wiki](https://github.com/NCATSTranslator/Translator-All/wiki/ARAX) | [RTXteam/RTX](https://github.com/RTXteam/RTX) | +| [Expander Agent](../../teams/expander-agent.md) | ARAX | [ARAX Wiki](https://github.com/NCATSTranslator/Translator-All/wiki/ARAX) | [RTXteam/RTX](https://github.com/RTXteam/RTX) | | (im)Proving Agent | Scalable Precision Medicine
Oriented Knowledge Engine ('SPOKE') | [SPOKE Wiki](https://github.com/NCATSTranslator/Translator-All/wiki/imProving-Agent) | [BaranziniLab/PSEV](https://github.com/BaranziniLab/PSEV) | -| [Exploring Agent](../../teams/exploring-agent.md) | [BioThings Explorer](https://explorer.biothings.io/) | [BioThings Explorer Page](bte.md) [BioThings Explorer Wiki]() | [biothings/biothings_explorer](https://github.com/biothings/biothings_explorer) | +| [Exploring Agent](../../teams/exploring-agent.md) | [BioThings Explorer](https://explorer.biothings.io/) | [BioThings Explorer Wiki]() | [biothings/biothings_explorer](https://github.com/biothings/biothings_explorer) | | Explanatory Agent | xARA | [xARA Wiki](https://github.com/NCATSTranslator/Translator-All/wiki/Explanatory-Agent) | | | Unsecret Agent | Unsecret Agent | [Unsecret Wiki](https://github.com/NCATSTranslator/Translator-All/wiki/UnSecret-Agent) | [webyrd/mediKanren](https://github.com/webyrd/mediKanren) | diff --git a/docs/architecture/index.md b/docs/architecture/index.md index 5a63b63..cdd18fe 100644 --- a/docs/architecture/index.md +++ b/docs/architecture/index.md @@ -5,8 +5,7 @@ The Translator platform consists of five main components, shown in the diagram b information abstracted from one or more underlying ‘knowledge sources’. Autonomous Relay Agents (ARAs) build upon the knowledge contributed by KPs by way of reasoning and inference across KPs. The Autonomous Relay System (ARS) functions as a central relay station and broadcasts user queries to the broader Translator -ecosystem and, in turn, compiles results. A user interface (UI) is under development and intended to serve as a public UI to the Translator system. -Finally, a Standards and Reference Implementation (SRI) Component provides services and community-based collaboration +ecosystem and, in turn, compiles results. A [Translator UI](https://ui.transltr.io/) is under development and intended to serve as the public interface to the Translator system. Finally, a Standards and Reference Implementation (SRI) Component, while not directly contributing to the Translator architecture, provides services and community-based collaboration guidance related to the development, adoption, and implementation of the standards needed to achieve the overall implementation goals of the Translator system. @@ -34,7 +33,7 @@ ibuprofen. Translator also provided answers that are correct but not terribly in user intent such as caffeine (an adjuvant included in certain pain medication). Also included among Translator answers to the example query are answers such as naltrexone, an opioid antagonist, which may not be expected by users. In support of naltrexone, Translator provided evidence and provenance indicating that naltrexone indeed may be used to treat chronic pain, as highlighted -below. Translator evidence and provenance included ranked answers with scores, primary and secondary knowledge sources +below in a screenshot from an experimental UI. Translator evidence and provenance included ranked answers with scores, primary and secondary knowledge sources behind any assertions, PubMedCentral or PubMed identifiers, and links to abstracts, etc. This sort of serendipitous discovery, or unexpected insight, represents the type of scientific discovery that @@ -51,4 +50,4 @@ In terms of impact, Translator is currently being used to promote serendipitous in a variety of disease spaces, including Fanconi anemia, systemic sclerosis, cystic fibrosis, Parkinson’s disease, and drug-induced liver injury. -Further technical details about the components of the Translator architecture are provided within the [**Development Guide**](../development-guide/index.md). \ No newline at end of file +Further technical details about the components of the Translator architecture are provided within the [**Development Guide**](../development-guide/index.md). diff --git a/docs/architecture/kp/index.md b/docs/architecture/kp/index.md index 431bc93..e124bb8 100644 --- a/docs/architecture/kp/index.md +++ b/docs/architecture/kp/index.md @@ -13,7 +13,7 @@ Knowledge Providers (KPs) contribute domain-specific, high-value information abs | Team | Name | Documentation | Github Repository | | ----------------------- | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------- | | Molecular Data Provider | Molecular Data Provider ('MolePro') | [Molecular Data Provider Wiki](https://github.com/NCATSTranslator/Translator-All/wiki/Molecular-Data-Provider) | [broadinstitute/molecular-data-provider](https://github.com/broadinstitute/molecular-data-provider) | -| [Service Provider](../../teams/service-provider.md) | BioThings | [Service Provider Page](service-provider.md) | [biothings/BioThings_Explorer_TRAPI](https://github.com/biothings/BioThings_Explorer_TRAPI) | +| [Service Provider](../../teams/service-provider.md) | BioThings | [Service Provider Wiki](https://github.com/NCATSTranslator/Translator-All/wiki/Service-Provider) | [biothings/BioThings_Explorer_TRAPI](https://github.com/biothings/BioThings_Explorer_TRAPI) | | Text Mining Provider | Text Mining Provider | [Targeted Text-Mined Association KG Wiki](https://github.com/NCATSTranslator/Translator-All/wiki/Targeted-Text-Mined-Association-KG) | [NCATSTranslator/Text-Mining-Provider-Roadmap](https://github.com/NCATSTranslator/Text-Mining-Provider-Roadmap) | | Clinical Data Provider | Columbia Open Health Data ('COHD') | [COHD KP Wiki](https://github.com/NCATSTranslator/Translator-All/wiki/COHD-KP) | | | Clinical Data Provider | OpenPredict | [OpenPredict KP Wiki](https://github.com/NCATSTranslator/Translator-All/wiki/OpenPredict-KP) | | @@ -21,9 +21,9 @@ Knowledge Providers (KPs) contribute domain-specific, high-value information abs | Multiomics Provider | BigGIM II-DrugResponse | [BigGIM II-DrugResponse Wiki](https://github.com/NCATSTranslator/Translator-All/wiki/Big-GIM-II:-Drug-Response-KP) | | | Multiomics Provider | Multiomics Wellness | [Multiomics Wellness Wiki](https://github.com/NCATSTranslator/Translator-All/wiki/Wellness-KP) | | | Multiomics Provider | EHR Clinical Risk | [EHR Clinical Risk Wiki](https://github.com/NCATSTranslator/Translator-All/wiki/EHR-Risk-KP) | | -| Multiomics Provider | Clinical Trials | [ClinicalTrials KP](https://github.com/NCATSTranslator/Translator-All/wiki/Clinical-Trials-KP) | [ClinicalTrials_KP](https://github.com/multiomicsKP/clinical_trials_kp) | -| Multiomics Provider | Drug Approvals | [Drug Approvals KP](https://github.com/NCATSTranslator/Translator-All/wiki/Multiomics-Drug-Approvals-KP) | [Drug Approvals KP](https://github.com/multiomicsKP/drug_approvals_kp) | -| Expander Agent | RTX-KG2 | [RTX-KG2 Page](rtx-kg2.md)
[RTX-KG2 Wiki](https://github.com/NCATSTranslator/Translator-All/wiki/KG2) | [RTXteam/RTX-KG2](https://github.com/RTXteam/RTX-KG2) | +| Multiomics Provider | Clinical Trials | [ClinicalTrials Wiki](https://github.com/NCATSTranslator/Translator-All/wiki/Clinical-Trials-KP) | | +| Multiomics Provider | Drug Approvals | [Drug Approvals Wiki](https://github.com/NCATSTranslator/Translator-All/wiki/Multiomics-Drug-Approvals-KP) | | +| Expander Agent | RTX-KG2 |[RTX-KG2 Wiki](https://github.com/NCATSTranslator/Translator-All/wiki/KG2) | [RTXteam/RTX-KG2](https://github.com/RTXteam/RTX-KG2) | | [Service Provider](../../teams/service-provider.md) | Pathway Figure OCR | [Pathway Figure OCR Wiki]() | [wikipathways/pathway-figure-ocr](https://github.com/wikipathways/pathway-figure-ocr) | | Exposures Provider | ICEES | [ICEES Wiki](https://github.com/NCATSTranslator/Translator-All/wiki/Exposures-Provider-ICEES) | [NCATS-Tangerine/icees-api](https://github.com/NCATS-Tangerine/icees-api) | | Exposures Provider | Causal Activity Model (CAM) KP | [CAM-KP Wiki](https://github.com/NCATSTranslator/Translator-All/wiki/CAM-KP) | [NCATS-Tangerine/cam-kp-api](https://github.com/NCATS-Tangerine/cam-kp-api) | diff --git a/docs/architecture/ui.md b/docs/architecture/ui.md index 31ef72b..04cc129 100644 --- a/docs/architecture/ui.md +++ b/docs/architecture/ui.md @@ -1,3 +1,3 @@ # Translator User Interface -The primary Translator User Interface (UI) is under development as a [web portal](https://ui.transltr.io/) to the Translator system that receives scored results and evidence, confidence, and provenance from the ARS and exposes results to users. This section doesn't cover the use of the UI, but rather, technical details facilitating its further development and maintenance. +The [Translator User Interface (UI)](https://ui.transltr.io/) is under development as the primary access point to the Translator system. The UI receives scored results and evidence, confidence, and provenance from the ARS and exposes results to users. This section isn't intended to cover the use of the UI, but rather, technical details facilitating its further development and maintenance. diff --git a/docs/deployment-guide/ci.md b/docs/deployment-guide/ci.md index 0ec5c25..8b13789 100644 --- a/docs/deployment-guide/ci.md +++ b/docs/deployment-guide/ci.md @@ -1,3 +1 @@ -# Continuous Integration (CI) Testing of the Translator System -T.B.A. (Mark Williams) diff --git a/docs/deployment-guide/deployment.md b/docs/deployment-guide/deployment.md index aa8db14..72f6b18 100644 --- a/docs/deployment-guide/deployment.md +++ b/docs/deployment-guide/deployment.md @@ -1,3 +1,3 @@ # Deployment -T.B.A. (Mark Williams) +T.B.A. (Sarah Stemann) diff --git a/docs/deployment-guide/index.md b/docs/deployment-guide/index.md index c9805b2..bca14ab 100644 --- a/docs/deployment-guide/index.md +++ b/docs/deployment-guide/index.md @@ -2,6 +2,5 @@ This section of the Translator technical documentation provides details mainly of interest to system administrators and development operations teams deploying, and monitoring the ongoing performance of, the Translator system in a production environment. -* [Continuous Integration Processes for Translator](ci.md) * [Deploying Components to Production](deployment.md) * [Monitoring the Translator System](monitoring.md) diff --git a/docs/development-guide/tutorials/consortium_development_guidelines.md b/docs/development-guide/tutorials/consortium_development_guidelines.md index ba913a2..8b13789 100644 --- a/docs/development-guide/tutorials/consortium_development_guidelines.md +++ b/docs/development-guide/tutorials/consortium_development_guidelines.md @@ -1,30 +1 @@ -# Consortium Development Guidelines -1. Project Management - 1. Project Management in Translator - 1. Project milestones - 2. Github in Translator - 1. Setting up repositories - 1. Role of NCATSTranslator, and other related specialized Git Organizations - 2. User Groups - 3. Access Policies - 2. Forking code - 3. Issue tracking - 4. How to submit a project GitHub Pull Request (PR) - 5. Publishing releases - 6. Publishing modules (pypi) -2. Coding process - Tim, Chris B - 1. Test Driven development - 2. Documentation Requirements - 1. Licensing -3. Describe how to test, deploy & version artifacts - 1. Versioning artifacts - 2. Component registration: Translator Smart API Registry - 1. SmartAPI translator extension metadata - 3. Deployment environments: - 1. x-maturity: dev, prod, etc. - 4. Continuous Integration - 5. Packaging - 1. Docker - 6. Deployment Process (DevOps) - 1. ITRB diff --git a/docs/faq.md b/docs/faq.md index f45e059..3f10844 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -4,7 +4,7 @@ Check out our [introduction to knowledge graphs](architecture/biolink/knowledge_graphs.md)? ### How can I start playing with Translator? -Translator is generally accessible to researchers via the [Translator Web portal](https://ui.transltr.io/). Python-savvy data scientists can try out the ["Hello Translator" Jupyter Notebook](development-guide/HelloTranslator.ipynb). +Translator is accessible to translational science end users via the [Translator user interface](https://ui.transltr.io/). Python-savvy data scientists can try out the [“Hello Translator” Jupyter Notebook](development-guide/HelloTranslator.ipynb). ### How can I develop software to use and/or contribute to Translator? We composed a [set of tutorials](development-guide/tutorials/index.md) for you. diff --git a/docs/index.md b/docs/index.md index c9c4de0..3a68718 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,25 +1,44 @@ ![image](img/translator-banner.jpg) # Welcome to the Biomedical Data Translator - Technical Documentation Site -The vision of the Biomedical Data Translator ("Translator") program is to accelerate translational science "_through an informatics platform that + +This site hosts the official technical documentation for the Biomedical Data Translator ("Translator") program. Key sections of the documentation are: + +- This overview page (and related links), presenting an overview of the Biomedical Data Translator consortium and its platform. +- [Knowledge Graphs](architecture/biolink/knowledge_graphs.md): Knowledge Graphs - the core scientific principle behind Translator. +- [Translator Architecture](architecture/index.md): overview of the Translator knowledge integration platform. +- [Software Development Guide](development-guide/index.md): guidelines for Translator software development, including tutorials. +- [System Deployment Guide](deployment-guide/index.md): guidelines for Translator systems administration, including continuous integration testing, production deployment and monitoring. + +## Vision for the Translator Program + +The vision for the Translator program is to accelerate translational science "_through an informatics platform that enables interrogation of relationships across the full spectrum of data types_" ([Austin _et al._ 2019](https://doi.org/10.1111/cts.12595), [BDTC 2019a](https://doi.org/10.1111/cts.12591), [BDTC 2019b](https://doi.org/10.1111/cts.12592), [Fecho _et al._ 2022](https://doi.org/10.1111/cts.13301), [other references](#references)). The goal is to build the infrastructure required to support and facilitate data-driven translational research on a large scale. The fundamental aim is to integrate as many datasets as possible, using a ‘knowledge graph’–based architecture, and allow them to be cross-queried and reasoned over by -translational researchers. A fundamental tenet of the Translator program is open data, including open (de-identified) +translational scientists in an effort to derive new insights and knowledge. A fundamental tenet of the Translator program is open data, including open (de-identified) patient data, and open team science. -This site hosts the official technical documentation for Translator. Key sections of the documentation are: +## Overview of the Translator System -- This overview page (and related links), presenting an overview of the Biomedical Data Translator consortium and its platform. -- [Knowledge Graphs](architecture/biolink/knowledge_graphs.md): Knowledge Graphs - the core scientific principle behind Translator. -- [Translator Architecture](architecture/index.md): overview of the Translator knowledge integration platform. -- [Software Development Guide](development-guide/index.md): guidelines for Translator software development, including tutorials. -- [System Deployment Guide](deployment-guide/index.md): guidelines for Translator systems administration, including continuous integration testing, production deployment and monitoring. +The Translator System is both federated and hierarchical. A schematic is provided on the architecture page. In brief, a user query's to the Translator user interface (UI) gets transmitted to the Translator "Autonomous Relay Service (ARS)", which then transmits the query in machine language to several Translor "Autonomous Relay Agents (ARAs)", who then transmit the machine-language query to numerous Translator "Knowledge Providers (KPs)", who then return their answers to the original query, in whole or in part, back to the ARAs, who then compile results and reason over them to derive answers to the user's original query, and then send them back to the ARS, which then merges for results from all ARAs, scores and ranks them, and adds annotations before returning the full result set back to the UI for user display and interrogation of the supporting evidence, confidence in results, and related provenance. + +Translator currently supports four main types of queries, with full evidence, providence, and confidence returned with query results. Here, we provide a brief high-level overview of each query type, with details included in other sections. + +1. "Lookup" queries refer to queries that ask Translator to essentially find "facts" or highly curated knowledge that typically take the form of a simple "one-hop" answer path. For instance, in response to a natural language query that asks "what drugs may treat asthma?", Translator may return an anser that states "albuterol treats asthma". +2. "Inferred" queries ask Translator to suggest an answer to a question that has varying degrees of confidence in the results. These inferences are derived from multiple reasoning algorithms and chains, including rule-based approaches, probabilistic models, and curated workflow paths that subject matter experts hypothesize as having the potential to derive novel results. "Inferred" queries can yield one-hop answer paths, but they are typically multi-hop answer paths. For example, in response to a natural language query that asks "what chemicals may increase the activity of the gene/protein SCN1A?", Translator might yield an inferred answer that suggests ranolazine, but in the form of an inferred answer path that states "ranolazine causes an increased activity or abundance of the gene/protein MTOR, which causes an increased activity or abundance of the gene/protein SCN1A". +3. "Pathfinder" queries ask Translator to find paths that connect two biomedical entities, for instance, a chemical exposure and an adverse outcome. Translator uses a variety of approaches, including combinations of "lookup" and "inferred" queries to derive answers to "pathfinder" queries. As such, these queries typically yield complex multi-hop answer paths. For instance, in natural language, a user might ask "what biological mechanisms might explain an observed relationship between exposure to polybrominated diphenyl ethers and cardiovascular disease?" +4. "Input_set" queries differ from the other types of queries in that users ask Translator to find a shared relationship between multiple user-contributed input entities (e.g., phenotypes) and another entity type (e.g., gene). This query functionality differs from a BATCH query, in which multiple input entities are entered by users, with Translator returning independent results for each input entity. The functionality also differs from an AND query, in which multiple input entities are entered by users, with Translator returning results for only those entities that are shared by all of the input entities. Rather, the "input_set" functionality operates more as an OR query, in which multiple input entities are entered by users, with Translator returning results for entities that are shared by some, but not all, of the input entities. For instance, a user may ask "what genes are related to one or more of these unusual and presumably unrelated phenotypes?" + +Translator is currently being used to promote serendipitous discovery and augment human reasoning in a variety of +disease spaces, including Fanconi anemia, systemic sclerosis, cystic fibrosis, Parkinson’s disease, drug-induced liver injury, primary ciliary dyskinesia, cyclic vomiting syndrome, and many others. (Please see the abbreviated References list for additional examples and details.) + +Translational scientists can access Translator via the [Translator UI](https://ui.transltr.io/). Additionally, developers or anyone with skills in Python can try out the [“Hello Translator” Jupyter Notebook](development-guide/HelloTranslator.ipynb). -## About the Biomedical Data Translator +## About the Translator Program The [**Biomedical Data Translator**](https://ncats.nih.gov/translator) program was launched by the [National Center for Advancing Translational Sciences ("NCATS")](https://ncats.nih.gov) in Fall of 2016. ([Austin et al. 2019; BDTC 2919a/b; @@ -30,16 +49,6 @@ Phase 2 "Development" focused activities of the **Translator Consortium** engage The program is [funded primarily through a National Institutes of Health (NIH) Other Transaction Awards (OTA) mechanism and an NIH contract awarded to support development of a Translator user interface (UI)](#consortium-funding). -An overview of the Translator timeline, in the context of key milestones and award funding from program inception through September 2022, is depicted below. - -![image](https://user-images.githubusercontent.com/26254388/174347625-c20cc7b1-134b-4a19-ab21-72c4ad4d2f89.png) - -Taken from Fecho _et al._ 2022, Supplementary Figure 1 - -Translator is currently being used to promote serendipitous discovery and augment human reasoning in a variety of -disease spaces, including Fanconi anemia, systemic sclerosis, cystic fibrosis, Parkinson’s disease, -drug-induced liver injury, and many others. - ### Key Programmatic Priorities - **Quality Metrics/Benchmarking**: Identify quality metrics/benchmarking to evaluate query answers and verify that the system is robust and bug-free @@ -63,11 +72,9 @@ drug-induced liver injury, and many others. ### Licensing -Translator is intended to evolve into a global public good through open source and access licensing. - -In this spirit, **_this_** document repository is published under the [Creative Commons CC0 1.0 Universal License](license.md) +The Translator Consortium makes every effort to draw only from openly available, public datasets. The Translator program's leadership also has brokered licensing agreements with certain data owners such as DrugBank. Note, however, that the system components developed by different Translator teams may have other distinct specific licensing. -Note, however, that the system components developed by different Translator teams may have other distinct specific licensing. +**_This_** document repository is published under the [Creative Commons CC0 1.0 Universal License](license.md) ### References diff --git a/docs/license.md b/docs/license.md index 439b462..834efd5 100644 --- a/docs/license.md +++ b/docs/license.md @@ -1,6 +1,6 @@ # Creative Commons Legal Code -The License covering this documentation repository only (not all software components of the Translator system, which may each have their own applicable licensing) is the one here below. +The License covering this documentation repository only (not all software components of the Translator system, which may each have their own applicable licensing) is the one here below. The Translator Consortium makes every effort to draw only from openly available, public datasets. The Translator program's leadership also has brokered licensing agreements with certain data owners such as DrugBank. Note, however, that the system components developed by different Translator teams may have other distinct specific licensing. ## CC0 1.0 Universal