Project health and maturity metrics

[brian-jarvis-aws]

Greetings OQS TSC members,

With your PQCA lifecycle stage formally identified as Research::Labs, I now would like to request your attention to this portion of the lifecycle document:

• Identify a minimum set of metrics the project proposes to use to signal project health and maturity. The TAC will use these to evaluate the proposed change in stage and continued health and maturity of the project within their stage. Examples include organizational dependency, commit activity, or other code ownership metrics.

I would like to propose that you begin this identification process using the metrics that are available from LFX Insights for your repositories:
https://insights.linuxfoundation.org/project/open-quantum-safe

You are, of course, welcome to use any alternate metrics that you find more useful if you have them. If you see areas for improvement in the set of metrics provided "out of the box" from LFX Insights, I encourage you to provide them that feedback directly.

I appreciate your attention to this request. Please let me know if the TAC can be of assistance.

[baentsch]

Please let me know if the TAC can be of assistance.

It would be great if you could get LFX Insights to put in an extra effort to more accurately report reality and thus, more properly represent the contributions and thus, interests of the community doing the code that PQCA publicly claims to support.

To see what I mean, please check just these 5 quick examples of where I think it's easy to see that this "decision dashboard" goes wrong -- and massively so:

1. These "metrics" do not report (the collective impact of) Individual contributions at all. In the OQS case that means a substantial mis-representation of reality.

2. LFX "Insights" reports activities in projects that OQS forked as OQS activity, leading for example to Google reported as contributor. In reality, Google never contributed a single line of code to OQS. This completely skews the overall picture:

I happen to know all of these people and value them highly, but neither David nor Bob ever contributed a single line of code to OQS, let alone more than 10% as this "report" "documents". The "Boringssl LUCI" line gives away the mistake in the LFX logic.

3. It reports the US as "most active contribution country":

Nothing could be further from the truth.

4. Even single numbers are incorrectly reported:

When clicking on the number count (114) this is displayed:

This is the exact half of the summary statement (honi soit qui mal'y'pense :) When looking at oqsprovider, it's even more stark: Topline number reported is 48, factually defensible (click in the UI) is 16; actually active maybe 7; really knowledgeable, well, fewer.

5. When drilling down trying to understand how these top-line figures come together that to me appear so totally off the reality "on the ground", here's an example of how misleading these simple activity counts are:

One could think Cisco is running the project with the help of Microsoft. The reality is that Cisco contributed 60 lines of code to the project while a person completely "invisible" to these stats is still responsible for >25000 (~90%; git ls-files|git blame to validate).

This alone would be a reason to adopt more meaningful metrics to get a more realistic picture of a project.

If you see areas for improvement in the set of metrics provided "out of the box" from LFX Insights, I encourage you to provide them that feedback directly.

I did that a long time back, but to no avail. I concluded then that LinuxFoundation is not interested in (documenting) numbers that run counter to the marketing interests and project control goals of its paying members.

As I understand that rationale I stopped highlighting flaws; but as you bring it up and want to use it:

The TAC will use these to evaluate the proposed change in stage and continued health and maturity of the project within their stage

I have to reiterate that I'm afraid not just LF but also the TAC keeps running the risk of deciding things without a solid connection to reality.

[brian-jarvis-aws]

I don't think the TAC has any particular sway over the LFX Insights folks, but I can certainly bring up the topic in our next TAC meeting to think about how we could effectively proxy your request for improvements in the LFX data. As the directly-impacted project, though, it will be more useful to come from OQS directly. That "long time back" link goes to a PQCA/TAC issue. Did you raise these concerns to the LFX Insights project directly? Ultimately how you choose to track the metrics you want to use is up to you, I only proposed LFX Insights since it exists and could be a useful starting point. And if there are improvements to be made there, then there's a "rising tide raises all boats" effect. But if you would like to avoid LFX Insights entirely and produce your own tracking metrics, no problem.

[baentsch]

The "rising tide raises all boats" effect was what had me (try to) make LFX Insights improve. At that time I still thought LF wants to improve in the interest of all projects it serves; since then I gained the impression that the opposite is true: projects have to serve LF. This very issue, your demand to have OQS have metrics so that you can measure on them is a point in case.

Regarding LFXI, there's a) apparently no GH repo one can raise such LFXI issues (I repeatedly asked for it in the issue but never received a response) and b) no apparent interest to improve: The issue was simply closed with the remark "This is one opinion. Others find it useful.". Neither enticed me to make further suggestions.

But if you would like to avoid LFX Insights entirely and produce your own tracking metrics, no problem.

I personally don't think inventing new metrics for the purpose of having metrics is necessary: OQS still consists of so few people, spending this effort imo would only add insult to injury, i.e., already not finding time and people for contributing.

Also I think we know its state pretty well: As soon as Douglas loses interest, the project's dead. I personally would have stopped contributing for good if I were not feeling I owe it to Douglas and a few people I know using the software to develop and/or test interesting new things -- as well as providing every (unknown) researcher something to give their new PQC inventions a nice "real-world test/rubdown". But then again, I'm irrelevant (just check LFXI metrics :).

Sure, we could invent some metrics to make your paperwork happy, but from the vantage point of a person deeply involved in the project since years, there's many indicators telling me that the project is on a downward slope since it was used to establish PQCA: At the start there was a flurry of apparent activity, but nothing long-lasting, e.g., no new person developing to become a maintainer (very much to the contrary, we lost capable maintainers, both actually as well as from a timing/availability perspective). Here's some more points to consider:

• code quality falls due to contribution time constraints, exemplified by inexplicable code merges; fewer and less thorough reviews; more PRs that landed but still need fixes
• issues backlog rises; even PQC algorithms readily available for integration do not get integrated
• then there's actual feature set reductions, exemplified by/culminating in the elimination of OQS from external interop testing
• more and more sub projects getting fewer and fewer updates; even sub projects introduced by PQCA members don't get updates any more (https://github.com/open-quantum-safe/liboqs-cupqc)
• fewer and fewer "very active" (typically, also knowledgeable) GH contributors (without whom the concept of "community" cannot be sustained)
• more and more interactions pushed to weekends as more and more people apparently don't see the project as their "prime" (work time) activity any more

Again, these are just my thoughts and I'd be glad to hear concrete statements refuting these: Worthwhile adding to TSC meeting agenda, @dstebila ?

Either way, if the TAC is truly interested in advancing the state of the projects you control, can I suggest you to consider working yourself into evaluating metrics capturing all these things --some of them are in LFXI, but they need to be used (and some, fixed, as per the above)-- and then acting in practically helpful ways?

A simple and imo effective way for this might be the addition of people that consistently work on the project: Is my feeling so far off the mark, namely that TAC and all PQCA members spend much more effort "managing" and "marketing" than actually supporting the development?

What about each PQCA member pledges 50% of its monetary contributions in the form of additional (compared to pre-PQCA) developers' time -- with the stated goal for long-lasting contributions/maintainership of (sub) projects?

IMO that way project health could be impacted more (positively) than by having this project spend time on developing metrics it doesn't need to know its state.

Don't get me wrong, I (think I) know where you're coming from: Huge projects with dozens or hundreds of contributors/developers. Please just consider OQS is not one of these projects, has never been, is so niche it doesn't have to be to make a mark. What it needs is commitment and (a wider) support base.

[brian-jarvis-aws]

your demand to have OQS have metrics so that you can measure on them
…
I personally don't think inventing new metrics for the purpose of having metrics is necessary
…
we could invent some metrics to make your paperwork happy

I’m not sure what gave the impression that I am requesting this for the sake of paperwork, or metrics for metrics sake.

I see it more along the lines of your views shared on this other issue:

I'm a strong believer in "you get what you measure" -- and its corollary "if you don't set concrete goals, no-one will try to achieve them".

If my interpretation of your comment is correct, you want to see sustained human investment and output quality, and avoid vanity metrics. I agree, if that’s the case. So how can we measure these things?

Some thoughts on things to measure, considering your specifically listed concerns about bus factor, code quality, issues backlog, knowledgeable contributor base, and developer time:

• Maintainer count and contribution trends
• Review depth, post-merge fix rate
• Backlog depth
• Active contributor count

These aren't invented metrics, they're measurements of the problems you've identified. Admittedly, just a few as a starting point, but if the TSC is open to extending this exercise we could build out a more holistic set of concerns and ways to measure them empirically. Not just for measurement's sake, but to have concrete data for honest triage and sustainability planning.

Does this seem reasonable?

[baentsch]

If my interpretation of your comment is correct, you want to see sustained human investment and output quality, and avoid vanity metrics. I agree, if that’s the case.

Yes -- we're on the same page there.

So how can we measure these things?

Some thoughts on things to measure, considering your specifically listed concerns about bus factor, code quality, issues backlog, knowledgeable contributor base, and developer time:

• Maintainer count and contribution trends

This seems easier said than done:

1. Maintainer count would require agreement what is needed to count a person as a maintainer. I have one interpretation, e.g., as documented here but the general PQCA perspective seems to be somewhat different: It seems maintainers even can get "labelled" as such, independent of what or how much s/he actually does to move the code forward or help the community. A possible rationale for this may be that this is seen as beneficial, e.g., to create the external impression there's "more"/enough maintainers. In my eyes this is misleading, though. I'd even prefer projects documenting they have no maintainers if no-one is able to fulfill those tasks: Users would know about the problem and either help out (become contributors and maintainers) or shun the project. Either is sensible in my eyes (call it "FOSS Darwinism"). Then, for the "fun" of it, there's even a long-standing open TSC issue on the topic that's essentially unresolved. I personally have the hunch that this has to do with someone (at PQCA?) already counting maintainers, not time actually spent maintaining: This then is item
2. Wouldn't it be better --than just counting bodies-- to (try to) arrive at a (proxy?) measurement for time spent actually supporting the community, pushing forward the project technically (helping bring new contributors on-line by active coaching, doing thorough and thoughtful PR reviews, giving good community answers, actively participating in discussions, etc.)?
3. Last problem: For this to work --particularly if you want trends--, there needs to be a meaningful/statistically relevant number of maintainers. This does not exist in OQS. It doesn't even exist for contributors.
   All told, this smells very much like ineffective bean counting. Requires beans and counters. Not sure (interest for) either exists.

• Review depth, post-merge fix rate

Good suggestion. Hard to measure, though: It requires time, interest and knowledge to analyze problems and their fixes. Proposals welcome. However, this has the same "low numbers" problem / is hard to arrive at statistically significant numbers for OQS.

• Backlog depth

Sensible for projects with substantial activity and issue opening/closing rates. I'd think OQS is too small for that: Any measure in this space can easily be gamed -- in either direction. Glaring example: Many members of the OQS core team don't (find the time to) really "work" (contribute to/answer/discuss) issues but only find time to focus on high-level wish lists, e.g., here and here or doing meetings. That alone to me is a strong sign that the time (available and) spent maintaining (in my understanding of the word) is falling.

• Active contributor count

Analogous problem as with maintainer count: What defines an active contributor? Two extremes: Is it a person using the project, knowing its code base from experience and contributing substantially, even if rarely, in event that something's wrong or is it a person attending every weekly status call and not saying a word? I have a personal definition for active contributor: It's a person that's so motivated and interested in the project that I can reach out to --regarding any question, regardless of his/her expertise-- and know there'll be feedback on GH within a day or two letting us move on to solve a problem with the code. But that's just my definition and not quite easy to "measure".

These aren't invented metrics, they're measurements of the problems you've identified. Admittedly, just a few as a starting point, but if the TSC is open to extending this exercise we could build out a more holistic set of concerns and ways to measure them empirically. Not just for measurement's sake, but to have concrete data for honest triage and sustainability planning.

I'd be open to spend more time on this only if a) there's a solid, commonly agreed-upon methodology behind this and not some flaky stats and b) there'd be a clear up-front commitment by PQCA to then act in concrete terms on any findings, e.g., willingness to divert some of the PQCA marketing funding to development support and/or requiring PQCA members to commit a certain percentage of in-house developers to actively contribute to projects "in need". Would you agree to this @brian-jarvis-aws ?

If in turn all this exercise results in is "great advice" like "The TAC recommends project xyz to work harder to find new contributors" or (as is right now) "project xyz cannot proceed to another lifecycle state" (that's actually pretty irrelevant to anyone outside the PQCA bubble), this is work lost.

Does this seem reasonable?

Yes -- with some caveats as stated. Most importantly there needs to be agreement by PQCA that this is a two way road, not just command-and-control (like "You develop/deliver some measurements so we can claim we have done something").

[baentsch]

One more thing @brian-jarvis-aws : Wouldn't this be a perfect use case for "AI"/ML? No project-invented "metrics", just machine-learned GH event (analysis) coalescing into a "statement of contribution and maintenance health", ideally with pointers to weaknesses/areas of improvement. Hard to believe this does not yet already exist somewhere.

Of course such AI could not be trained on the few data points of any of the small PQCA projects, but surely something that could be gleaned from the gazillion other FOSS projects (ideally those that are healthy :). Something like this would also obviate LFXI and it's flawed stats. And alleviate tiny projects like PQCA of the need to spend time thinking about some metrics.

[brian-jarvis-aws]

On the specific metrics: I wasn't trying to prescribe exactly what OQS should measure. Those were conversation starters based on the concerns you raised. The actual methodology, definitions, and what's practical to measure for a project of OQS's size? That's for the TSC to work out, not me to dictate. You're right that some of these are hard to measure or may not make sense at OQS's scale.

On who this is for: I think there's a misunderstanding about the purpose here. This isn't about satisfying PQCA paperwork or lifecycle gatekeeping (I know you don't find that valuable anyway). It's about OQS transparently representing its state to its users. You mentioned "FOSS Darwinism" - users knowing the real state and deciding to help or move on. That requires honest metrics that show the reality, not vanity numbers. That's what I'm advocating for.

On PQCA commitment to act: I hear you that metrics are pointless without action. Whether PQCA would commit to diverting funds to development support or requiring member contributions, that's not my call to make. I can't promise that. What I can say is that having clear data about project health makes it much easier to make those asks.

These are good questions about what's practical and what definitions make sense. I hope this doesn't come off as personal offense, but I'd love to hear what other voices from the TSC also have to say about it (in addition, not instead of). Can you bring this to the TSC agenda to discuss as a group?

[baentsch]

What I can say is that having clear data about project health makes it much easier to make those asks.

Very well, then: Could you state which (type of) data you would accept to even consider an ask for help? Also, would you be willing to disregard the statistical insignificance of such data as pointed out above or will this be used in a next step to still not "divert funds to development support"? To be frank, I'm a bit "miffed" about this term as when IBM&LF approached OQS more than 2 years ago with the proposal to take over the project, this very promise -- for (increased) development support -- was the key argument then made that convinced at least me to give up the project's independence, even though it had served OQS very well until then.

I hope this doesn't come off as personal offense, but I'd love to hear what other voices from the TSC also have to say about it (in addition, not instead of). Can you bring this to the TSC agenda to discuss as a group?

No offense taken -- to the contrary: I'm also very eager to hear opinions of other people (inside or outside of TSC), actually anyone active on the project -- on many topics. But this is one of the issues I see: Not a lot of contributions happen anymore within GH. Everyone (at least on the TSC) seems to only have time anymore to participate in meetings and "work off an agenda" there, nothing more (in terms of development). And to my chagrin, I'm beginning to fall in line with this. Discussions such as this are my attempt at getting out of this "meeting-only-activity-trap".

Finally, apologies for apparently creating the impression that I "monopolize" this conversation: The thing is that I'm less able to make time for meetings in the US timezone (and in those, cannot always easily follow everything stated in English: It's not my mother tongue). Hence I use this medium to state my opinions -- at times that suit my life: I'm in a different time zone than 80% of the other TSC members -- and 100% of the PQCA TAC members, fwiw&afaik.....

[brian-jarvis-aws]

Very well, then: Could you state which (type of) data you would accept to even consider an ask for help? Also, would you be willing to disregard the statistical insignificance of such data as pointed out above or will this be used in a next step to still not "divert funds to development support"? To be frank, I'm a bit "miffed" about this term as when IBM&LF approached OQS more than 2 years ago with the proposal to take over the project, this very promise -- for (increased) development support -- was the key argument then made that convinced at least me to give up the project's independence, even though it had served OQS very well until then.

Yeah the optics aren't great. "Become a part of X so that you can get Y." "To get Y, please produce Z." I wish I had some silver bullet answer here, but all I can say is that something concrete is better than nothing / anecdata (which is itself also statistically insignificant). It comes down to the measurements you (OQS) think best illustrate the concerns/successes you want to improve/highlight. These things are significant, even if not to a statistician.