From c30ffbe6f96741b5082ec9b6048c2965f3ca01c7 Mon Sep 17 00:00:00 2001 From: "Darren A. Natale" Date: Wed, 13 Sep 2023 16:46:27 -0400 Subject: [PATCH 1/2] Create OntologiesReviewWorkflow This is to update outdated information regarding the review process, which is no longer for 'promotion' from library to foundry, but rather for new ontology submissions. This file is intended to be linked from the EWG page. --- docs/OntologiesReviewWorkflow | 42 +++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) create mode 100644 docs/OntologiesReviewWorkflow diff --git a/docs/OntologiesReviewWorkflow b/docs/OntologiesReviewWorkflow new file mode 100644 index 000000000..2ae739306 --- /dev/null +++ b/docs/OntologiesReviewWorkflow @@ -0,0 +1,42 @@ +--- +layout: doc +title: Ontology Review Process +--- + +This page deals with the process, policies, and guidelines for manually reviewing newly-submitted ontologies, with particular focus on operational aspects such as: + +- which ontologies are reviewed, and in what order? +- how are reviewers chosen? +- what is the workflow for conducting a review? +- what are the criteria used? + +Ontologies are reviewed manually after completing the automated evaluation. Though the automated evaluation checks for adherence to [OBO Foundry principles](http://www.obofoundry.org/principles/fp-000-summary.html), it cannot capture certain nuances outlined in the principles nor evaluate ontology quality. The purpose of the manual review is to check the ontology for issues that the NOR Dashboard does not cover. + +# Steps prior to manual review + +Once an ontology is submitted to the OBO Foundry, it is added to the New Ontology Dashboard for automated evaluation. The steps involved in ontology submission and subsequent dashboard evaluation are described in detail in this [FAQ answer](http://obofoundry.org/faq/how-do-i-register-my-ontology.html) and links therein. + +# Review priority + +All submitted ontologies will be manually reviewed based on the order in which the ontology becomes compliant with the automated evaluation (that is, when the [New Ontology Request (NOR) Dashboard](http://obofoundry.org/obo-nor.github.io/dashboard/index.html) status is set to 'pass'). + +# Choosing reviewers + +Reviewers are chosen on a rotating basis from the members of the [OBO Foundry Operations Committee](http://obofoundry.org/docs/Membership.html). + +# Ontology review workflow + +The manual process begins once a reviewer is assigned. The reviewer will assess key aspects of the ontology (see below) and report the findings both to the submitter (via the issue tracker ticket used for submission) and to the OBO Operations Committee. The latter will discuss the findings and make a recommendation for acceptance, revision, or rejection. + +# Manual review criteria + +Criteria for review include (but are not limited to): + +1. Ontology scope - Do the terms fall within the ontology's stated target domain of knowledge? Was the ontology developed for a very specific purpose or community? +2. Terms with the new ontology prefix - Do the terms follow the OBO identifier scheme? Are there terms with the same meaning available in another OBO Foundry ontology? Is there another OBO Foundry ontology whose scope covers any of the new terms? +3. Correct use of imported terms - Does the ontology accurately reuse terms from other OBO ontologies? Are imported terms in appropriate hierarchies, and do they preserve the term's upper-level alignment? Are any additional axioms used for these terms correct in both a technical (e.g. passes reasoning) and substantive sense? +4. Basic review of axiomatic patterns - Are axioms generally highly complex? Are existential restrictions used correctly? (Typical mistakes include “R some (A and B and C)” to mean “(R some A and R some B and R some C)”). +5. Appropriate use of object properties - Are object properties used in a manner consistent with their definitions, domain, and range? (Examples of incorrect usage include those based on some interpretation of the label of the object property but not actually fitting the property definition or domain and range.) +6. Responsiveness to suggested changes - Have the developers been willing to fix any identified issues during the review? + + From 97d08a7a15146e65165210a1cf9122a4625e5d1b Mon Sep 17 00:00:00 2001 From: "Darren A. Natale" Date: Thu, 14 Sep 2023 12:53:45 -0400 Subject: [PATCH 2/2] Update OntologiesReviewWorkflow Addresses concerns raised by Nomi --- docs/OntologiesReviewWorkflow | 32 ++++++++++++++++++++++---------- 1 file changed, 22 insertions(+), 10 deletions(-) diff --git a/docs/OntologiesReviewWorkflow b/docs/OntologiesReviewWorkflow index 2ae739306..1713bf4e1 100644 --- a/docs/OntologiesReviewWorkflow +++ b/docs/OntologiesReviewWorkflow @@ -10,7 +10,7 @@ This page deals with the process, policies, and guidelines for manually reviewin - what is the workflow for conducting a review? - what are the criteria used? -Ontologies are reviewed manually after completing the automated evaluation. Though the automated evaluation checks for adherence to [OBO Foundry principles](http://www.obofoundry.org/principles/fp-000-summary.html), it cannot capture certain nuances outlined in the principles nor evaluate ontology quality. The purpose of the manual review is to check the ontology for issues that the NOR Dashboard does not cover. +Newly-submitted ontologies are reviewed manually after passing the automated evaluation. Though the automated evaluation checks for adherence to [OBO Foundry principles](http://www.obofoundry.org/principles/fp-000-summary.html), it cannot capture certain nuances outlined in the principles nor evaluate ontology quality. The purpose of the manual review is to check the ontology for issues that the NOR Dashboard does not cover. # Steps prior to manual review @@ -18,7 +18,7 @@ Once an ontology is submitted to the OBO Foundry, it is added to the New Ontolog # Review priority -All submitted ontologies will be manually reviewed based on the order in which the ontology becomes compliant with the automated evaluation (that is, when the [New Ontology Request (NOR) Dashboard](http://obofoundry.org/obo-nor.github.io/dashboard/index.html) status is set to 'pass'). +All submitted ontologies will be manually reviewed based on the order in which the ontology becomes compliant with the automated evaluation (that is, when the [New Ontology Request (NOR) Dashboard](http://obofoundry.org/obo-nor.github.io/dashboard/index.html) status is 'pass'). # Choosing reviewers @@ -30,13 +30,25 @@ The manual process begins once a reviewer is assigned. The reviewer will assess # Manual review criteria -Criteria for review include (but are not limited to): - -1. Ontology scope - Do the terms fall within the ontology's stated target domain of knowledge? Was the ontology developed for a very specific purpose or community? -2. Terms with the new ontology prefix - Do the terms follow the OBO identifier scheme? Are there terms with the same meaning available in another OBO Foundry ontology? Is there another OBO Foundry ontology whose scope covers any of the new terms? -3. Correct use of imported terms - Does the ontology accurately reuse terms from other OBO ontologies? Are imported terms in appropriate hierarchies, and do they preserve the term's upper-level alignment? Are any additional axioms used for these terms correct in both a technical (e.g. passes reasoning) and substantive sense? -4. Basic review of axiomatic patterns - Are axioms generally highly complex? Are existential restrictions used correctly? (Typical mistakes include “R some (A and B and C)” to mean “(R some A and R some B and R some C)”). -5. Appropriate use of object properties - Are object properties used in a manner consistent with their definitions, domain, and range? (Examples of incorrect usage include those based on some interpretation of the label of the object property but not actually fitting the property definition or domain and range.) -6. Responsiveness to suggested changes - Have the developers been willing to fix any identified issues during the review? +For the list below, yes/no questions include the expected answers in square brackets after the question. Criteria for review minimally include: + +1. Ontology scope - + Do the terms fall within the ontology's stated target domain of knowledge? [yes] + Was the ontology developed for a very specific purpose or community? +2. Terms with the new ontology prefix - + Do the terms follow the OBO identifier scheme? [yes] + Are there terms with the same meaning available in another OBO Foundry ontology? [no] + Is there another OBO Foundry ontology whose scope covers any of the new terms? [no] +3. Correct use of imported terms - + If the ontology reuses terms from other OBO ontologies, are they used accurately? [yes] + Are imported terms in appropriate hierarchies, and do they preserve the term's upper-level alignment? [yes] + Are any additional axioms used for these terms correct in both a technical (e.g. passes reasoning) and substantive sense? [yes] +4. Basic review of axiomatic patterns - + Are axioms generally stated simply or are they highly complex? (Highly complex axioms will require extra scrutiny.) + Are existential restrictions used correctly? [yes] (Typical mistakes include “R some (A and B and C)” to mean “(R some A and R some B and R some C)”). +5. Appropriate use of object properties - + Are object properties used in a manner consistent with their definitions, domain, and range? [yes] (Examples of incorrect usage include those based on some interpretation of the label of the object property but not actually fitting the property definition or domain and range.) +6. Responsiveness to suggested changes - + Have the developers been willing to fix any identified issues during the review? [yes]