Adding a flag "unresponsive" to OBO metadata

[matentzn]

Problem: We (OFOC) have received several messages about ontologies that are actively maintained, but do not respond to issue tracker requests or pull requests at all over the course of multiple months. This is an important piece of information for many ontology users, especially curators, that need to ensure that issues with the ontology and new term requests can be processed at all.

Proposal: The proposal is to introduce a new flag, unresponsive: true which documents that the fact that an ontology, while perhaps active, does not respond to issues, including bug reports and new term requests. It is valuable for prospective users to understand this before using an ontology. This flag could be used to display a small icon, like a red speech bubble icon or some such, on the main obo website. This flag would also inform the "responsiveness" check of the OBO dashboard, which currently only is checked by the mere presence of an issue tracker.

[dosumis]

What would be the timeframe/metric for judging unresponsiveness? Will it be lack of response on any issue? Complete lack of response on tracker? Will this track ticket closure rate?

I think that to be fair to the individual ontologies, they should get an automated email warning if they are failing this one. It may be the nudge the need to stay engaged.

[cthoyt]

I think this is super valuable. There are lots of ontologies that are unresponsive even though they're being maintained, the ones from the RGD are examples of this (e.g., https://github.com/rat-genome-database/RS-Rat-Strain-Ontology, https://github.com/rat-genome-database/CMO-Clinical-Measurement-Ontology, https://github.com/rat-genome-database/MMO-Measurement-Method-Ontology). This is not the only group/individual doing this, another example is https://github.com/luis-gonzalez-m/Collembola, who has in early January 2023 made commits but leaves issues and PRs unanswered from 6+ months ago.

For now, an excellent way to judge which ontologies do not read feedback is to look at https://cthoyt.com/obo-community-health/ and see which ones have not introduced the obofoundry topic suggested in #1538. I automatically created an issue on every single OBO Foundry ontology's issue tracker over a year ago, so if they have not responded to this then there's a high likelihood that they do not read/engage with their issue tracker.

If you decide to annotate this in OBO Foundry metadata, I'd also suggest adding the date at which the most recent ping was done and also the date when it was decided to mark as unresponsive.

I'd be careful when writing policy about this - if a group finds out they have been relegated to "unresponsive", they might be motivated in the short term to engage in their tracker to get the flag taken away, but this might not influence their medium- or long-term behavior. How can you best write a policy that guards against this?

[nlharris]

if a group finds out they have been relegated to "unresponsive", they might be motivated in the short term to engage in their tracker to get the flag taken away, but this might not influence their medium- or long-term behavior. How can you best write a policy that guards against this?

Just thinking out loud here, but maybe there could be a yearly "responsiveness check" (it would need to be automated somehow, not sure how) so that even if an ontology just responded to our initial prod and then went unresponsive again, they'd risk getting labeled "unresponsive" at the next check. Even if that just motivated them to engage with their tracker once a year, that would be better than some!

For the automated check, maybe there could be something like "must have closed at least 2 issues or responded to at least 4 issues in the past year"? It's hard, though, because some trackers are a lot more active than others. Some don't get a lot of issues submitted.

[zhengj2007]

For the automated check, maybe there could be something like "must have closed at least 2 issues or responded to at least 4 issues in the past year"? It's hard, though, because some trackers are a lot more active than others. Some don't get a lot of issues submitted.

Not sure about the criteria. I am the contact person of OPL (Ontology for Parasite Lifecycle). It is a small ontology and mainly used by GeneDB and VEuPathDB. Current it contains all the terms needed by the database and I solved all the posted issues. So, it may not be able to meet the responsive criteria. But I do response the emails and issues of OPL if I got any.

[cthoyt]

For the automated check, maybe there could be something like "must have closed at least 2 issues or responded to at least 4 issues in the past year"? It's hard, though, because some trackers are a lot more active than others. Some don't get a lot of issues submitted.

Not sure about the criteria. I am the contact person of OPL (Ontology for Parasite Lifecycle). It is a small ontology and mainly used by GeneDB and VEuPathDB. Current it contains all the terms needed by the database and I solved all the posted issues. So, it may not be able to meet the responsive criteria. But I do response the emails and issues of OPL if I got any.

Super kudos for having Inbox 0 on your repo! For sure we should not penalize people in your situation, but this is definitely the exception to the rule

[zhengj2007]

Yes. It is a rare case :)

[lschriml]

One thing to consider, groups use the issue tracker within a project.
So they may not be responded to, as they are internal to a project.
Not sure how to be able to distinguish these cases.

[nlharris]

One thing to consider, groups use the issue tracker within a project. So they may not be responded to, as they are internal to a project. Not sure how to be able to distinguish these cases.

I'm not sure whether Lynn means that some projects use internal (non-public) issue trackers (which I think is not ok for OBO Foundry ontologies) or whether she means that some projects may not respond to issues that are posted by people external to the project (which I think is also not ok for OBO Foundry ontologies). @lschriml can you clarify?

[matentzn]

I understood Lynn like: if the issue tracker is used internally, there may be many open issues and no responses (which is totally fine), and you could not reliably discern that from the outside.. which is true! See 1000 open issues on HPO.

[nlharris]

Ok, I see. Thanks.

[cthoyt]

Like I mentioned in #2255 (comment), two years ago, I sent an automated request to all 176 active (at the time) OBO ontologies' respective repositories asking to put the obofoundry tag into the metadata. There were adequate instructions and a kind message offering help and guidance. As of today, 133 of these repositories were responsive to this request.

This leaves 43 repositories have not responded in the two years. I have just pinged all 43, tagging their responsible people (this is now possible since in the last two years, I have convinced the OBO community that having the GitHub handle easily accessible for each active ontology is valuable).

I suggest that in a few months, we review who remains unresponsive and begin marking these ontologies as unresponsive.

[matentzn]

Before we call a vote, we need some more solid criteria here on how to operationalise "unresponsive". Suggestions?

[cthoyt]

Suggestion for criteria:

An OBO Foundry ontology can be considered as unresponsive if:

1. If the person listed as the primary contact does not answer emails within X months

AND

2. If the person listed as the primary contact does not answer issues/PRs where they are tagged within X months

AND

3. Nobody else who has merge rights answers issues/PRs within X months

I suggest we set X at somewhere between 3 and 6 months. I would also suggest before setting any ontology to "unresponsive" that a final email is sent giving a 2 week.

I think these probably need a bit more careful wording, but the gist is that if there's no way to get in touch with people in the project, either the primary person or someone else who can actually get stuff done, then it leads to the "unresponsive" flag.

[mbrochhausen]

Generally in agreement, but we also need to specify ways of how the flag can be removed. Best, Mathias

…

On Wed, Oct 25, 2023 at 7:04 AM Charles Tapley Hoyt < ***@***.***> wrote: Suggestion for criteria: An OBO Foundry ontology can be considered as unresponsive if: 1. If the person listed as the primary contact does not answer emails within X months AND 1. If the person listed as the primary contact does not answer issues/PRs where they are tagged within X months AND 1. Nobody else who has merge rights answers issues/PRs within X months I suggest we set X at somewhere between 3 and 6 months. I would also suggest before setting any ontology to "unresponsive" that a final email is sent giving a 2 week. I think these probably need a bit more careful wording, but the gist is that if there's no way to get in touch with people in the project, either the primary person or someone else who can actually get stuff done, then it leads to the "unresponsive" flag. — Reply to this email directly, view it on GitHub <#2255 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACF6DLTZDV6SQ657IZSCE3DYBD55FAVCNFSM6AAAAAATXZT5BKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZZGEZDAMRWG4> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[cthoyt]

@mbrochhausen I think it's totally fair to say that if the contact person gets in touch on the OBO Foundry and says "hey, sorry I was unresponsive. I am now back on the ontology train" and they demonstrate that they have engaged with the pending issues/pull requests, then there shouldn't be a very big hurdle to remove the "unresponsive" flag

If an ontology maintainer becomes responsive just for a short time to get the "unresponsive" tag removed, this is a bit disingenuous, but also will be obvious to everyone what happened.

To make this a bit more objective, I would suggest having a 6-month check-in attached to any ontology for which the "unresponsive" tag is removed - maybe just by sending an issue that says "hey, are you still there?" and if they say "yes!" it's all good. If they don't respond (same X length as noted above) then it goes back to "unresponsive".

[mbrochhausen]

Sounds good to me!

…

On Wed, Oct 25, 2023 at 3:36 PM Charles Tapley Hoyt < ***@***.***> wrote: @mbrochhausen <https://github.com/mbrochhausen> I think it's totally fair to say that if the contact person gets in touch on the OBO Foundry and says "hey, sorry I was unresponsive. I am now back on the ontology train" and they demonstrate that they have engaged with the pending issues/pull requests, then there shouldn't be a very big hurdle to remove the "unresponsive" flag If an ontology maintainer becomes responsive just for a short time to get the "unresponsive" tag removed, this is a bit disingenuous, but also will be obvious to everyone what happened. To make this a bit more objective, I would suggest having a 6-month check-in attached to any ontology for which the "unresponsive" tag is removed - maybe just by sending an issue that says "hey, are you still there?" and if they say "yes!" it's all good. If they don't respond (same X length as noted above) then it goes back to "unresponsive". — Reply to this email directly, view it on GitHub <#2255 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACF6DLRGFZBZ4XAFFHR6BOLYBFZ5RAVCNFSM6AAAAAATXZT5BKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBQGAYTKNZTHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[matentzn]

I think @cthoyt next step is to call a vote with "add unresponsive flag with charlies suggestion for a definition", "nope, I don't want an unresponsive flag" and "nope, I want an unresponsive flag but don't like the criteria (suggestions below)."

[cthoyt]

Sure. Let's keep in mind that the unresponsive flag can actually be a new field, which itself is optional.

Ontologies that have the unresponsive flag set to true will get an additional warning banner on their webpages on https://obofoundry.org/. They will continue to be shown on the front page ontology table like before.

I volunteer to write this up more carefully and make a PR with a SOP about this similar to https://github.com/OBOFoundry/OBOFoundry.github.io/blob/master/docs/OntologyStatus.md.

	Responsive	Unresponsive
Maintained	Active	Unresponsive
Unmaintained	Inactive	Orphaned

[nataled]

For clarity, here's how the proposed status fits with the others. I didn't include 'obsolete' because that's just a special case for other statuses.

[cthoyt]

@nlharris I think we're complete here as well.

[cthoyt]

@nlharris sorry nevermind, let me send a PR updating the Schema, then this is closed!