diff --git a/cEP-0019.md b/cEP-0019.md new file mode 100644 index 000000000..77fa05819 --- /dev/null +++ b/cEP-0019.md @@ -0,0 +1,208 @@ +Meta-review System +=========================== + +|Metadata| | +|--------|-----------------------------------| +|cEP |0019 | +|Version |1.0 | +|Title |Meta-review System | +|Authors |Boxuan Li | +|Status |Proposed | +|Type |Process | + +Abstract +-------- + +This cEP describes the details of the process of implementing the meta-review +system as a part of the +[GSoC'18 project](https://summerofcode.withgoogle.com/projects/#5188493739819008). + +Background +---------- + +People including the author of pull request respond to comments by attaching +emojis. Those emojis are called meta-review, or +[reactions](https://developer.github.com/v4/enum/reactioncontent/). THUMBS_UP +and THUMBS_DOWN, together with other reactions, are used by people especially +the author of pull request to publicly acknowledge or reject review comments. +This provides feedback to reviewers and all other members. + +However, those reactions spread over all PRs and are not collected and +analyzed. By tracking reactions, a nice feedback loop would be created +for both senior and junior reviewers. + +Introduction +------------ + +The meta-review project is about building a meta-review system, +similar to a meta-moderation system, that can track emoji responses to +review comments. These responses are to be collected, processed and displayed +on [gh-board](https://github.com/coala/gh-board), which is a nice serverless +kanban board. + +The scope of this project would exclude non-public meta-reviews. That is, +there would not be any anonymous meta-reviews as that introduces too many +additional complexities. The public only version would be used for +a long time before adding support for non-public meta-reviews. + +Objectives +---------- + +One goal of this project is to encourage people to do meta-reviews, +systematically. + +Also, to encourage people to do more meta-reviews, statistics of meta-reviews +are to be collected and analysed. People who do meta-reviews will get scores +according to some metrics. A ranking list is to be displayed on gh-board. + +To conclude, this project builds a meta-review system on gh-board to display +statistics (score ranking, meta-reviews in need), and encourage people to +do meta-review. + +Proposed Approach +----------- + +### Collection of Meta-reviews + +Meta-review info, namely reactions, can be fetched via GitHub API V4 GraphQL. +A Sample query is as follows: + +```GraphQL +query { + repository(owner:"coala", name:"coala-bears") { + pullRequest(number:2372) { + author { + login + url + } + comments(first:10) { + nodes { + author { + login + url + } + reactions(first:10) { + totalCount + nodes { + content + user { + login + } + } + } + } + } + } + } +} +``` + +The collection of meta-reviews includes two parts: + +##### Pre-fetch meta-reviews + +Currently, `fetch-issues.js` pre-fetches issues and pull requests to speed up +gh-board performance and save users' API hits. Meta-reviews are also to be +fetched during this procedure. + +##### Fetch meta-reviews + +When a new issue/pull request is detected, gh-board will fetch that issue and +loads it asynchronously. Meta-reviews also need to be fetched during this +procedure. + +### Analysis of Meta-reviews + +This paragraph explains how meta-review information can be utilized to +generate a meaningful analysis. + +As the collection of meta-reviews, this part also includes two parts: + +##### Pre-process meta-reviews + +After pre-fetching meta-reviews, they are to be processed, which is explained +in the following sections. + +##### Process meta-reviews + +As new meta-reviews are loaded, useful information is to be extracted and +current analysis of meta-review is to be updated correspondingly. + +The following sections explain how meta-review information is used to +generate a meaningful analysis. + +#### Ranking list + +A ranking list would be displayed on gh-board, according to points +people get. A sample formula for points would be as following: + +PR = total number of positive reactions (e.g., THUMBS_UP) a person gets +for all reviews he did. +NR = total number of negative reactions (e.g., THUMBS_DOWN) a person gets +for all reviews he did. +PG = total number of positive reactions a person gives to other people for +their reviews. +NG = total number of negative reactions a person gives to other people for +their reviews. + +Note PG and NG are used to encourage people to get involved in this +reactions feedback system. Also, it may not be limited to PRs. Then final +score S would be: + +S = PR - NR + c1 * PG + c2 * NG + +where c1 = 0.5, c2 = 0.8, are coefficients. c2 is slightly larger than c1 +because people are usually reluctant to give negative reactions. + +If a person has zeros for all factors, then he is regarded as an inactive +reviewer and would not be displayed on board. + +#### Meta-reviews in Need + +Review comments that need meta-review will be displayed on gh-board. There +are three approaches: + +1. The worst case is that the tracking stores in the json which +PR do not have meta-reviews, and gh-board shows only a list of those PRs +(and the user must then find the reviews in GitHub PR web app, and do +meta-review there). + +2. An improvement is to store the identifier of the PR comments which +do not have meta-reviews and gh-board links to the PR review comment +in GitHub PR web app. + +3. In the ultimate version, gh-board would show the review comments. +In the future, people could do meta-reviews directly on gh-board. This also +provides an approach to doing meta-reviews anonymously. + +#### Other Features + +The GSoC project provides a blueprint for future enhancement. Additional +features could be implemented to make the meta-review system more +sophisticated: + +1. A basic ranking list will be built first. In the best case, we should +offer an overall ranking list as well as a recent ranking list (user +could switch to see overall/recent ranking) with sorting option. + +2. Offer an option for users to do meta-reviews directly on gh-board. This +might be a bad idea - people shouldn't do meta-review unless they know the +whole PR context. This might be a good idea - in the future users can do +meta-reviews on gh-board anonymously (there is no way for people to do +anonymous meta-reviews on GitHub page). + +3. Display visitor’s own meta-reviews if the visitor logs in. People may want +to see meta-reviews they have done before. + +4. Display visitor's received meta-reviews if the visitor logs in. +GitHub won't send a notification to a user if he receives any meta-review. +Suppose I ask for someone's agreement and he uses 👍 to show his agreement. +But I wouldn't receive any notification. I have to check that page regularly. +Meta-review system is a nice place to see the collection of all meta-reviews +I receive. + +#### Documentation including short videos + +Meta-review system sounds like something of a novelty. Documentation is needed +to clarify the use of the system. One or more short videos are to be published +to help people get acquainted with the system.