From e9f7587856e1c236d37e96f54a0cb4d72069776d Mon Sep 17 00:00:00 2001 From: Boxuan Li Date: Sun, 6 May 2018 12:12:19 +0800 Subject: [PATCH] cEP-0019.md: Meta-review system Describes how meta-reviews could be tracked and handled. Closes https://github.com/coala/cEPs/issues/121 --- cEP-0019.md | 484 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 484 insertions(+) create mode 100644 cEP-0019.md diff --git a/cEP-0019.md b/cEP-0019.md new file mode 100644 index 000000000..21e6cf29f --- /dev/null +++ b/cEP-0019.md @@ -0,0 +1,484 @@ +# Meta-review System + +| Metadata | | +| -------- | ------------------------------------------ | +| cEP | 0019 | +| Version | 1.0 | +| Title | Meta-review System | +| Authors | Boxuan Li | +| Status | Proposed | +| Type | Process | + +## Abstract + +This cEP describes the details of the process of implementing the meta-review +system as a part of the +[GSoC'18 project](https://summerofcode.withgoogle.com/projects/#5188493739819008). + +## Background + +People respond to review comments on pull requests by giving feedback. Emoji +make it much easier to give and receive feedback without a long comment thread. +Those emoji are called +[reactions](https://developer.github.com/v4/enum/reactioncontent/) on GitHub +or [award emoji](https://docs.gitlab.com/ee/user/award_emojis.html) on GitLab. +They are used by people to publicly express their feelings about review comments, +which provides feedback to comment authors and all other members. + +GitLab is beyond the scope of this project, as GitHub and GitLab have quite +different APIs and most core projects of coala are based on GitHub. This +proposal will only discuss reactions on GitHub, but not award emoji on GitLab. + +There are 6 types of reactions on GitHub: + +| Name | Markdown | Unicode | Emoji | +| ----------- | ----------- | ------- | ----- | +| THUMBS_UP | `:+1:` | U+1F44D | πŸ‘ | +| THUMBS_DOWN | `:-1:` | U+1F44E | πŸ‘Ž | +| LAUGH | `:smile:` | U+1F604 | πŸ˜„ | +| HOORAY | `:tada:` | U+1F389 | πŸŽ‰ | +| CONFUSED | `:confused` | U+1F615 | πŸ˜• | +| HEART | `:heart:` | U+FE0F | ❀️ | + +There are three types of reviews on a pull request: + +1. Issue Comment + [see sample](https://github.com/coala/gh-board/pull/18#issuecomment-385878735) + +2. Review Summary + [see sample](https://github.com/coala/gh-board/pull/18#pullrequestreview-116783753) + +3. Review Discussion + [see sample](https://github.com/coala/gh-board/pull/18#discussion_r186249423) + +Those reactions have been added to GitHub since +[2016 March](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/). +However, they are often used in an adhoc way. They are spread over all PRs +and are not collected and analyzed. Moreover, GitHub doesn't send +notifications to reviewers who receive reactions (see +[related discussion](https://github.com/isaacs/github/issues/631)). +By tracking reactions, a nice +feedback loop would be created for both senior and junior reviewers. + +## Introduction + +The meta-review project is about making use of reactions to build a +meta-review system, similar to a meta-moderation system. These responses +are to be collected, processed and displayed on +[gh-board](https://github.com/coala/gh-board), which is a nice kanban +board. Compared to [Huboard](https://huboard.com/) which requires server-side +support, gh-board is serverless. + +The scope of this project would exclude non-public meta-reviews. That is, +there would not be any anonymous meta-reviews as that introduces too many +additional complexities. The public only version would be used for +a long time before adding support for non-public meta-reviews. + +Note there is a technical limitation in that reactions can not be placed +onto the "review summary", which means general review comments should better +not be put into the "review summary" text box, as it will be not able to be +meta-reviewed. + +## Objectives + +One goal of this project is to encourage people to do meta-reviews, +systematically. Meta-reviews in need will be displayed on gh-board. + would be redirected to gh-board page which +shows meta-reviews in need. The implementation details are discussed in the +next section. + +Also, to encourage people to do more meta-reviews, statistics of meta-reviews +are to be collected and analyzed. People who do meta-reviews will get scores +according to some metrics. A ranking list is to be displayed on the community +site. + +Another objective is that the newcomer process is able to be changed after +this project. Newcomers will be required to finish at least one meta-review and +receive at least one positive meta-review, once this project is ready for +production. + +To conclude, this project builds a meta-review system on the community site +and gh-board to display score ranking statistics and meta-reviews in need, +respectively, and encourage people to do meta-review. + +## Meanings of reactions + +As mentioned in the [background](#background) section, GitHub provides 6 types +of reactions. They have different meanings in the context of meta-review system. + +THUMBS_UP and THUMBS_DOWN are most commonly used. They express strong feelings: +THUMBS_UP means the review comment is well-written and helpful, while +THUMBS_DOWN means the review comment is misleading, or even worse, violates +[coala Community Code of Conduct](https://github.com/coala/cEPs/blob/master/cEP-0006.md). +They will contribute to the score which is explained in the +[Ranking list](#ranking-list) section. + +HEART, HOORAY and LAUGH are reactions that express positive feelings, but not +as clear as THUMBS_UP, and consequently will be excluded from the scoring until +an analysis of how they should be used is done. + +CONFUSED expresses negative feeling. CONFUSED are used when people feel the +review comment is not descriptive enough. For example a common advice given +to a newcomer PR is to follow coala guidelines for commit and amend the commit +message. There are various ways in which reviewers can express this, some can +be descriptive enough, some can be very descriptive with links and stuff, also +some can be non descriptive comprising of our internal workflow lingo and hence +non newcomer friendly. If the review comment is confusing, but maybe correct, +then people should definitely give a CONFUSED reaction. It is not clear whether +this should be neutral or negative. As a consequence, it will be excluded from +the scoring until an analysis of how they should be used is done. + +## Proposed Approach + +### Collection of Meta-reviews + +Meta-review info, namely reactions, can be fetched via GitHub API V4 GraphQL. +A sample query is as follows: + +```GraphQL +{ + repository(owner: "coala", name: "gh-board") { + pullRequest(number: 18) { + author { + login + url + } + reviews(first: 5) { + nodes { + bodyText + comments(first: 1) { + nodes { + bodyText + author { + login + url + } + reactions(first: 100) { + totalCount + nodes { + createdAt + content + user { + login + } + } + } + } + } + } + } + comments(first: 1) { + nodes { + bodyText + author { + login + url + } + reactions(first: 100) { + totalCount + nodes { + createdAt + content + user { + login + } + } + } + } + } + } + } +} +``` + +The response is in JSON format as follows: + +```json +{ + "data": { + "repository": { + "pullRequest": { + "author": { + "login": "li-boxuan", + "url": "https://github.com/li-boxuan" + }, + "reviews": { + "nodes": [ + { + "bodyText": "This is looking quite good, but closed issues/prs is empty:\nhttps://deploy-preview-18--coala-gh-board.netlify.com/#/r/coala:coala?s=closed\nmeta-review will need to be able to see closed PRs", + "comments": { + "nodes": [] + } + }, + { + "bodyText": "", + "comments": { + "nodes": [ + { + "bodyText": "Leftover?", + "author": { + "login": "blazeu", + "url": "https://github.com/blazeu" + }, + "reactions": { + "totalCount": 0, + "nodes": [] + } + } + ] + } + }, + { + "bodyText": "", + "comments": { + "nodes": [ + { + "bodyText": "Yup, thanks.", + "author": { + "login": "li-boxuan", + "url": "https://github.com/li-boxuan" + }, + "reactions": { + "totalCount": 0, + "nodes": [] + } + } + ] + } + }, + { + "bodyText": "", + "comments": { + "nodes": [ + { + "bodyText": "You made the your own getTime function below.", + "author": { + "login": "blazeu", + "url": "https://github.com/blazeu" + }, + "reactions": { + "totalCount": 1, + "nodes": [ + { + "createdAt": "2018-05-07T03:09:08Z", + "content": "LAUGH", + "user": { + "login": "li-boxuan" + } + } + ] + } + } + ] + } + }, + { + "bodyText": "", + "comments": { + "nodes": [ + { + "bodyText": "Can be turned into arrow function and put at the very top.", + "author": { + "login": "blazeu", + "url": "https://github.com/blazeu" + }, + "reactions": { + "totalCount": 1, + "nodes": [ + { + "createdAt": "2018-05-05T02:19:08Z", + "content": "THUMBS_UP", + "user": { + "login": "li-boxuan" + } + } + ] + } + } + ] + } + } + ] + }, + "comments": { + "nodes": [ + { + "bodyText": "To be honest, I don't understand why we need fetch-issues.js. From the relevant code, I can see fetch-issues.js does some pre-fetching and initial-data.js dumps it into the database. But what's the aim of doing that? It doesn't seem to save API hits. These pre-fetched issues in database are replaced later anyway.", + "author": { + "login": "li-boxuan", + "url": "https://github.com/li-boxuan" + }, + "reactions": { + "totalCount": 0, + "nodes": [] + } + } + ] + } + } + } + } +} +``` + +When a new issue/pull request is detected, gh-board will fetch that issue and +loads it asynchronously. Meta-reviews also need to be fetched during this +procedure. + +### Analysis of Meta-reviews + +This section explains how meta-review information can be utilized to +generate a meaningful analysis. + +#### Ranking list + +A ranking list would be displayed on coala community website, according +to points people get. Some metrics are to be collected and used to generate +the ranking formula: + +P1 = total number of THUMBS_UP πŸ‘ a person gets for all reviews he did. + +P2 = total number of THUMBS_UP πŸ‘ a person gives to other people for +their reviews. + +N1 = total number of THUMBS_DOWN πŸ‘Ž a person gets for all reviews he did. + +N2 = total number of THUMBS_DOWN πŸ‘Ž a person gives to other people for +their reviews. + +Note P2 and N2 are used to encourage people to get involved in this +reactions feedback system. + +An example formula for final score S is: + +S = k * (P1 - N1) + c1 * P2 + c2 \* N2 + +c1 = 0.2, c2 = 0.4, are coefficients. c2 is larger than c1 because people +are usually reluctant to give negative reactions. + +k is the impact factor. The higher score a person has, the more impacts he has, +thus his meta-reviews are more valuable. k is based on the scores in the +previous iteration and is normalized. For example, in a previous iteration, +Alice got 2 marks, Bob got 0.8 marks and Charlie got 10 marks. The calculation +procedure would be as follows: + +```python +>>> c = [2, 0.8, 10] +>>> max_score = max(c) +>>> min_score = min(c) +>>> result = [(i - min_score) / (max_score - min_score) for i in c] +>>> print(result) +[0.13043478260869565, 0.0, 1.0] +>>> result_adjust = [i + 0.1 for i in result] # add bias +>>> print(result_adjust) +[0.23043478260869565, 0.1, 1.1] +>>> result_rounded = [round(i, 3) for i in result_adjust] +>>> print(result_rounded) +[0.23, 0.1, 1.1] +``` + +Therefore, the k factor for Alice, Bob and Charlie in the next iteration would +be: 0.23, 0.1 and 1.1, respectively. + +If a person has zeros for all factors, then he is regarded as an inactive +reviewer and would not be displayed. If a person has negative score, he will +still be displayed, but the meta-reviews he gives to other people will not +count. + +The ranking formula will reside in the community repo and can be modified over +time in order to achieve different results. + +#### Meta-reviews in Need + +##### Definition + +When does a review need meta-review? There are two perspectives: + +1. Meta-review before merge/close. Those review comments need meta-reviews by + all people. They are sorted according to update time on gh-board. In + particular, one will always see the review comment that is under his PR + first. + +2. Meta-review after merge/close. Review comments after merge/close need + meta-reviews by maintainers. Those meta-reviews will reflect the + accuracy of the review comments. Meta-reviews done by other developers after + merge/close will not be counted. + +Note that review comments by gitmate-bot are not to be meta-reviewed and would +not be displayed on gh-board. + +There is also concern for edited comments and deleted comments. Users are able +to edit/delete their review comments on GitHub. If someone receives negative +reactions, they might edit/delete their review comment. To ensure that +meta-review data is not discarded even if it is edited/deleted on GitHub, raw +information of comment contents and reactions are to be stored in community +repo. If the author edits/deletes their comment after it has been +meta-reviewed, they are destroying feedback from the community. This improper +behaviour will lead to a warning and a negative score given to them. + +##### Display + +Review comments that need meta-review will be displayed on a separate column +on gh-board. There are two approaches: + +1. A basic version is to store the identifier of the PR comments which + do not have meta-reviews and gh-board links to the PR review comment + in GitHub PR web app. Doing it on GitHub has the benefit of letting users + have their usual tools, e.g. + [refined-github](https://github.com/sindresorhus/refined-github). + +2. In the ultimate version, gh-board would show the review comments. + In the future, people could do meta-reviews directly on gh-board. This also + provides an approach to doing meta-reviews anonymously. + +#### Performance Requirements + +Currently, +[fetch-issues.js](https://github.com/coala/gh-board/blob/46f1ad5ce83d5d72aa95ea00703b92c2fc38b2dc/script/fetch-issues.js) +pre-fetches issues and pull requests to speed up gh-board performance and +save users' API hits. Meta-reviews are also to be fetched during this procedure. + +#### Other Features + +The GSoC project provides a blueprint for future enhancement. Additional +features could be implemented to make the meta-review system more +sophisticated: + +1. A basic ranking list will be built first. In the best case, we should + offer an overall ranking list as well as a recent ranking list. + +2. Offer an option for users to do meta-reviews directly on gh-board. In the + future users can do meta-reviews on gh-board anonymously (currently, there + is no way for people to do anonymous meta-reviews on GitHub page). Note + that people may need context to do meta-reviews. The diff hunk to which + the comment applies needs to be displayed on gh-board so that people have + a basic context to do meta-reviews directly on gh-board. + +3. Display visitor’s own meta-reviews if the visitor logs in. People may want + to see meta-reviews they have done before. + +4. Provide notifications of meta-reviews to a user if they receive any + meta-review. Suppose I ask for someone's agreement and they use πŸ‘ to + show their agreement, but I wouldn't receive any notification. I have to + check that page regularly. gh-board is a nice place to provide notifications + of meta-reviews. + +5. There is a potential for reverted commits to be meta-reviewed. A reverted + commit should trigger a new meta-review of the original PR, as time may + have proved that the merge was incorrect, or maybe the merge was a + self-aware time-sensitive workaround, and is no longer needed. + +6. An important technical detail of meta-review in coala on GitHub is that + the context disappears on an amend push, so meta-review by maintainers after + the merge may be without the necessary context to be properly evaluated. + However, if the commit SHA is mentioned, GitHub will keep the commit even + if it is no longer in the PR. A gitmate plugin is needed in the future to + mention any SHAs which have been reviewed, so that meta-reviews by + maintainers will have more context and be more effective. + +7. The UI of gh-board could have factual aggregated statistical information + like how many of each emoji a person has given/received, again with a legend + explaining their intended meanings, but without trying to aggregate across + the different reaction types. Note that this information should not be + displayed as a column, but as a manager-ish page like the + [Burnup Chart](https://coala.github.io/gh-board/#/r/coala:coala%7Ccoala-bears%7Cdocumentation%7Ccorobo/burnup).