cEP-0019.md: Meta-review system

Describes how meta-reviews could be tracked and handled.

Closes #121

cEP-0019.md

@@ -0,0 +1,208 @@
+Meta-review System
+===========================
+
+|Metadata|                                   |
+|--------|-----------------------------------|
+|cEP     |0019                               |
+|Version |1.0                                |
+|Title   |Meta-review System                 |
+|Authors |Boxuan Li <[email protected]>|
+|Status  |Proposed                           |
+|Type    |Process                            |
+
+Abstract
+--------
+
+This cEP describes the details of the process of implementing the meta-review
+system as a part of the
+[GSoC'18 project](https://summerofcode.withgoogle.com/projects/#5188493739819008).
+
+Background
+----------
+
+People including the author of pull request respond to comments by attaching
+emojis. Those emojis are called meta-review, or
+[reactions](https://developer.github.com/v4/enum/reactioncontent/). THUMBS_UP
+and THUMBS_DOWN, together with other reactions, are used by people especially
+the author of pull request to publicly acknowledge or reject review comments.
+This provides feedback to reviewers and all other members.
+
+However, those reactions spread over all PRs and are not collected and
+analyzed. By tracking reactions, a nice feedback loop would be created
+for both senior and junior reviewers.
+
+Introduction
+------------
+
+The meta-review project is about building a meta-review system,
+similar to a meta-moderation system, that can track emoji responses to
+review comments. These responses are to be collected, processed and displayed
+on [gh-board](https://github.com/coala/gh-board), which is a nice serverless
+kanban board.
+
+The scope of this project would exclude non-public meta-reviews. That is,
+there would not be any anonymous meta-reviews as that introduces too many
+additional complexities. The public only version would be used for
+a long time before adding support for non-public meta-reviews.
+
+Objectives
+----------
+
+One goal of this project is to encourage people to do meta-reviews,
+systematically.
+
+Also, to encourage people to do more meta-reviews, statistics of meta-reviews
+are to be collected and analysed. People who do meta-reviews will get scores
+according to some metrics. A ranking list is to be displayed on gh-board.
+
+To conclude, this project builds a meta-review system on gh-board to display
+statistics (score ranking, meta-reviews in need), and encourage people to
+do meta-review.
+
+Proposed Approach
+-----------
+
+### Collection of Meta-reviews
+
+Meta-review info, namely reactions, can be fetched via GitHub API V4 GraphQL.
+A Sample query is as follows:
+
+```GraphQL
+query {
+  repository(owner:"coala", name:"coala-bears") {
+    pullRequest(number:2372) {
+      author {
+        login
+        url
+      }
+      comments(first:10) {
+        nodes {
+          author {
+            login
+            url
+          }
+          reactions(first:10) {
+            totalCount
+            nodes {
+              content
+              user {
+                login
+              }
+            }
+          }
+        }
+      }
+    }
+  }
+}
+```
+
+The collection of meta-reviews includes two parts:
+
+##### Pre-fetch meta-reviews
+
+Currently, `fetch-issues.js` pre-fetches issues and pull requests to speed up
+gh-board performance and save users' API hits. Meta-reviews are also to be
+fetched during this procedure.
+
+##### Fetch meta-reviews
+
+When a new issue/pull request is detected, gh-board will fetch that issue and
+loads it asynchronously. Meta-reviews also need to be fetched during this
+procedure.
+
+### Analysis of Meta-reviews
+
+This paragraph explains how meta-review information can be utilized to
+generate a meaningful analysis.
+
+As the collection of meta-reviews, this part also includes two parts:
+
+##### Pre-process meta-reviews
+
+After pre-fetching meta-reviews, they are to be processed, which is explained
+in the following sections.
+
+##### Process meta-reviews
+
+As new meta-reviews are loaded, useful information is to be extracted and
+current analysis of meta-review is to be updated correspondingly.
+
+The following sections explain how meta-review information is used to
+generate a meaningful analysis.
+
+#### Ranking list
+
+A ranking list would be displayed on gh-board, according to points
+people get.  A sample formula for points would be as following:
+
+PR = total number of positive reactions (e.g., THUMBS_UP) a person gets
+for all reviews he did.
+NR = total number of negative reactions (e.g., THUMBS_DOWN) a person gets
+for all reviews he did.
+PG = total number of positive reactions a person gives to other people for
+their reviews.
+NG = total number of negative reactions a person gives to other people for
+their reviews.
+
+Note PG and NG are used to encourage people to get involved in this
+reactions feedback system. Also, it may not be limited to PRs. Then final
+score S would be:
+
+S = PR - NR + c1 * PG + c2 * NG
+
+where c1 = 0.5, c2 = 0.8, are coefficients. c2 is slightly larger than c1
+because people are usually reluctant to give negative reactions.
+
+If a person has zeros for all factors, then he is regarded as an inactive
+reviewer and would not be displayed on board.
+
+#### Meta-reviews in Need
+
+Review comments that need meta-review will be displayed on gh-board. There
+are three approaches:
+
+1. The worst case is that the tracking stores in the json which
+PR do not have meta-reviews, and gh-board shows only a list of those PRs
+(and the user must then find the reviews in GitHub PR web app, and do
+meta-review there).
+
+2. An improvement is to store the identifier of the PR comments which
+do not have meta-reviews and gh-board links to the PR review comment
+in GitHub PR web app.
+
+3. In the ultimate version, gh-board would show the review comments.
+In the future, people could do meta-reviews directly on gh-board. This also
+provides an approach to doing meta-reviews anonymously.
+
+#### Other Features
+
+The GSoC project provides a blueprint for future enhancement. Additional
+features could be implemented to make the meta-review system more
+sophisticated:
+
+1. A basic ranking list will be built first. In the best case, we should
+offer an overall ranking list as well as a recent ranking list (user
+could switch to see overall/recent ranking) with sorting option.
+
+2. Offer an option for users to do meta-reviews directly on gh-board. This
+might be a bad idea - people shouldn't do meta-review unless they know the
+whole PR context. This might be a good idea - in the future users can do
+meta-reviews on gh-board anonymously (there is no way for people to do
+anonymous meta-reviews on GitHub page).
+
+3. Display visitor’s own meta-reviews if the visitor logs in. People may want
+to see meta-reviews they have done before.
+
+4. Display visitor's received meta-reviews if the visitor logs in.
+GitHub won't send a notification to a user if he receives any meta-review.
+Suppose I ask for someone's agreement and he uses 👍 to show his agreement.
+But I wouldn't receive any notification. I have to check that page regularly.
+Meta-review system is a nice place to see the collection of all meta-reviews
+I receive.
+
+#### Documentation including short videos
+
+Meta-review system sounds like something of a novelty. Documentation is needed
+to clarify the use of the system. One or more short videos are to be published
+to help people get acquainted with the system.