Document how we should organise community contributions in ontology reviews

[matentzn]

Contributions like #2406 (comment) are very valuable for our ontology reviews, but right now our SOPs have a gaping whole in them how to elicit and communicate these reviews. In my view, there should be a documentation section call "community input on ontology reviews" which contributors like Charlie can be pointed to, and a standard blurp that can be copy-pasted into the review that ensures the ontology submitters know which elements need to be fixed and which don't. The OBO Foundry assigned primary reviewer also needs a sentence in their SOP that clarifies that they need to pick and choose from community reviews which element are SHOULD, MUST and MUST NOT..

[pfabry]

In my opinion:

• We should insist on keeping the review in two clearly differentiated stages: technical and content
• There is certainly room for improvement on the technical side, but the NOR Dashboard + SOP covers most of it
• For the content review, having a separate issue/channel of discussion for the community input is indeed a good idea.
• The "official" reviewer should act as a gate between the submitters and the community, but all the community input should be freely accessible to the submitters.

[nataled]

EWG questions for @matentzn:

1. You mention the need for a new 'documentation section'. Which document?
2. How do we ensure that those who provide community input would read it?
3. Can you provide a draft standard blurb?

EWG questions for @pfabry:

1. It's unclear to me how these stages (technical, content) would be enforced.
2. You mention for content review there should be a separate channel for community input. Do you propose to have separate tracker items for each (official vs community)? If so, again, how would these be enforced?

Overall the EWG agrees with Paul's last bullet point and Nico's last sentence.

Regarding the SOP, we've identified a potential place for the new directive for the NOR reviewer (not Manager). We propose that it go between 5 and 6 below:

5. Once the new ontology passes the review, the NOR manager assigns the next available reviewer from the OBO operation members.
   NEW: The assigned reviewer is considered the final arbiter of requirements, and in some cases might have to clarify which suggestions made by other reviewers are required, which are simply good to do but not required, and which should not be done.
6. Finally, when the ontology is fully accepted, the NOR manager remove the ontology from the OBO NOR dashboard.

Note: I'd like to think about the use of SHOULD, MUST, etc notation for these suggestions. So far we use them in principles, but I found during my own NOR review that more nuance was needed. In my case I used Consider, Recommend, Strongly Recommend, and Imperative, defined below:

CONSIDER means just what it says--think about it. Deciding against such suggestions will (probably) not adversely impact acceptance, but might impact users in terms of understanding and usage. RECOMMEND and STRONGLY RECOMMEND are changes that really should be done at some point (though not necessarily before acceptance). IMPERATIVE means that the ontology cannot be accepted until the change is made.

There should also at least be an addition (italicized below) to one of the bullet points in the notice given by the NOR Manager:

• After you have successfully passed the Dashboard you will be assigned an OBO Operations committee member to review the ontology. The assigned reviewer is considered the "official" reviewer, and will act as arbiter between the ontology submitter and any suggestions made by others.

We'll likely need a bit more than that, but I just wanted to get the ball rolling there. Suggestions welcome.

[pfabry]

EWG questions for @pfabry:

1. It's unclear to me how these stages (technical, content) would be enforced.

2. You mention for content review there should be a separate channel for community input. Do you propose to have separate tracker items for each (official vs community)? If so, again, how would these be enforced?

I think there is a need to make a distinction between how these steps will be implemented and how they will be enforced.

For the implementation, a solution could be to use the NOR tracker in the first step and do all the technical back and forth in there. Then, once the technical review is completed, the NOR manager creates an issue, assigned from the beginning to the "official" reviewer, in the OBO tracker. All inputs from community and official reviewer could take place in the same issue with the condition that the submitter are notified that only the official reviewer inputs are mandatory. I think creating a second issue only for community input is not realistic.

How to enforce this falls under the more general question of how to enforce OBO Foundry rules and decisions.

I fully agree with your proposed modifications for the NOR manager SOP.

[matentzn]

You mention the need for a new 'documentation section'. Which document?

https://github.com/OBOFoundry/OBOFoundry.github.io/blob/master/docs/SOP.md#ROOM

How do we ensure that those who provide community input would read it?

We don't - we add a bullet for the ontology reviewer SOP (link above) to tell them to paste a link to the respective blurp on the site (can also be in that document above), with a standard text like "Thank you for your highly appreciated review contribution! The designated OBO Foundry reviewer will asses the content of your review and communicate to the submitter which issues MUST and SHOULD be addressed [..link to SOP explaining in one sentence why only the primary reviewer has the last word..]

Can you provide a draft standard blurb?

See above. Note there should be too bullets: 1 in the Ontology Review SOP that instructs the primary OBO reviewer to inform community reviewers of the SOP when they provide a voluntary review, and 2 that instructs the OBO reviewer to coordinate community responses into actionable items that MUST, SHOULD or MUST NOT be processed by the ontology submitter. They should inform ontology submitters that only the items provided by them in the review need to be addressed.

I think MUST SHOULD etc is good practice because there is a public RFC for this: https://datatracker.ietf.org/doc/html/rfc2119 that is used by most standard texts. We could add CONSIDER as a very weak suggestion as well? Or do you see any could reason to replace MUST by IMPERATIVE?

[pfabry]

Below a proposition for the modified SOP for the reviewer:

Reviewing Ontologies for OBO Membership

The goal of this SOP is to provide a clear set of rules of communication and criteria of review for the designated OBO foundry reviewer. It is expected that a programmatic review using the Dashboard has already been done and the submitters have addressed any problems found. The purpose of the manual review is to check the ontology for issues that the Dashboard review does not cover. A sample of terms/axioms should be checked. In order for this review to be relatively quick (~ 2 hours), the reviewer is not expected to review all the terms/axioms.

Rules of communication

1. The designated OBO Foundry reviewer acts as the interface between the ontology submitter and the OBO community. Its role is to:
   a. Communicate its review of the submitted ontology to the ontology submitter in the form of items of feedback.
   b. Coordinate the OBO community comments into items of feedback and communicate these items to the ontology submitter.
   c. Inform the ontology submitter and the community that only the items provided by the designated OBO Foundry reviewer need to be addressed.
2. All items of feedback must be formatted as actionable items that MUST, SHOULD or MUST NOT be processed by the ontology submitter. These actionable items are provided using GitHub checklist syntax (- [ ] TODO) in order to track how far along they are in being addressed. Addressable issues identified as part of the review should be added to the new ontology’s issue tracker.

Criteria of review

1. Ontology scope. The new ontology must present use cases demonstrating its relevance to the life sciences. Was the ontology developed using expert input or trusted scientific sources representative of the consensus in its target domain of knowledge? If the ontology was developed for a very specific purpose or community, representation and consensus need not be broad; however, this scope should be clearly stated.
2. Terms with the new ontology prefix. All new terms MUST follow the OBO identifier scheme (often they are accidentally written wrongly, e.g. using https instead of http). There MUST NOT be a term with the same meaning available in another OBO Foundry ontology, ie there must not be a term referring to a concept that already exists in another OBO Foundry ontology (whether or not the label is identical). There SHOULD NOT be another OBO Foundry ontology whose scope covers any of the new terms. In the event that these conditions cannot be fulfilled, justification(s) MUST be provided. Such justification(s) include:
   a. the demonstration that these terms are actually not the same (this happens when term meaning/concept is ambiguous); or
   b. the other OBO Foundry ontology (for which the terms were in scope) was contacted and rejected the request for adding new terms in scope for that ontology.
3. Correct use of imported terms. Does the ontology accurately reuse terms from other OBO ontologies? Are imported terms in appropriate hierarchies? That is, has the import of the term preserved its upper-level alignment? Are any additional axioms used for these terms correct in both a technical (e.g. passes reasoning) and substantive sense?
4. Basic review of axiomatic patterns: Are existential restrictions used correctly? Typical mistakes include “R some (A and B and C)” to mean “(R some A and R some B and R some C)” Are axioms generally highly complex? If so, we should review a handful to ensure they are as intended.
5. Appropriate use of object properties. Examples of incorrect usage include those based on some interpretation of the label of the object property but not actually fitting the property definition or domain and range.
6. Responsiveness to fixing changes. A willingness to fix any identified issues during the review must be demonstrated. Issues expected to be addressed should be added using GitHub checklist syntax (- [ ] TODO) in the GitHub issue. The time limit for addressing these is 2 months; a longer period should be requested if needed.