Merge pull request src-d#450 from dpordomingo/docs-v3

Docs enhancements for empathy-sessions and others

.gitignore

@@ -20,3 +20,4 @@
 
 # Configuration
 /config.yml
+/*.pem

README.md

@@ -99,13 +99,15 @@ There is [`docker-compose.yml`](/docker-compose.yml) config file for [Docker Com
 
 To do so, clone this repository or download [`docker-compose.yml`](/docker-compose.yml) directly.
 
-Create the `config.yml` file in the same directory where `docker-compose.yml` is. You can use [`config.yml.tpl`](/config.yml.tpl) as a template. Then run:
+Create the `config.yml` file in the same directory where `docker-compose.yml` is. You can use [`config.yml.tpl`](/config.yml.tpl) as a template. Make sure that you specify in the `config.yml` the repositories that will be watched by **source{d} Lookout**. Then run, passing [a valid GitHub user/token](docs/configuration.md#authentication-with-github):
 
 ```bash
 $ docker-compose pull
 $ GITHUB_USER=<user> GITHUB_TOKEN=<token> docker-compose up --force-recreate
 ```
 
+Once it is running, **source{d} Lookout** will start posting the comments returned by `dummy` analyzer into the pull requests opened at GitHub in the repositories that you configured to be watched.
+
 You can stop it by pressing `ctrl+c`
 
 If you want to try **source{d} Lookout** with your own analyzer instead of `dummy` one, you must run it in advance, then [set it into `config.yml`](/docs/configuration.md#analyzers) and then run:

config.yml.tpl

@@ -1,3 +1,4 @@
+# list of the analyzers to be called by lookoutd
 analyzers:
   - name: Dummy
     addr: ipv4://localhost:9930
@@ -20,15 +21,15 @@ providers:
     # client_id:
     # client_secret:
 
-# only for authorization with a GitHub token
+# list of repositories to watch when using authorization with a GitHub token
 repositories:
   - url: github.com/_USER_/_REPO_TO_WATCH_
     client:
       # user:
       # token:
       # minInterval: 1m
 
-# only for authorization with GitHub App
+# web interface configuration, only available if authorizing with GitHub App
 web:
   # Secret key to sign JSON Web Tokens
   signing_key:

docs/CONTRIBUTING.md

@@ -84,6 +84,48 @@ $ make test-json
 ```
 
 
+## Web Interface
+
+If you want to contribute in **source{d} Lookout** [Web Interface](web.md), you should consider the following:
+
+### Dependencies
+
+[Node.js](https://nodejs.org) `>=8` is required; you can check which version of `node` do you have, running:
+
+```shell
+$ node -v
+v10.15.0
+```
+
+### Running
+
+In case you want to locally run the web interface from sources, you can choose one of the following:
+- using web assets from binaries (embeded by `esc`), that will require you to restart the server everytime you change any web asset.
+    ```shell
+    $ make -f Makefile.web web-serve
+    ```
+    And navigate to [http://127.0.0.0:8080](http://127.0.0.0:8080)
+- using `create-react-app` dev server, with live reload for web assets changes, running in separated terminals the backend: 
+    ```shell
+    $ make -f Makefile.web web-start
+    ```
+    and the frontend:
+    ```shell
+    $ make -f Makefile.web web-dependencies # if you didn't do it yet
+    $ yarn --cwd frontend start
+    ```
+    Configure the GitHub App authorization callback URL to `http://127.0.0.0:3000/callback`, and navigate to [http://127.0.0.0:3000](http://127.0.0.0:3000)
+
+### Testing
+
+For unit-tests over the Web Interface frontend:
+
+```shell
+$ make -f Makefile.web web-dependencies # if you didn't do it yet
+$ make -f Makefile.web web-test
+```
+
+
 ## dummy Analyzer
 
 [`dummy` analyzer](analyzers-examples.md#dummy-analyzer) is part of the **source{d} Lookout** codebase but its release cycle is managed independently from the main one.

docs/architecture.md

@@ -17,7 +17,7 @@ The server also exposes **DataService** as a gRPC service.
 
 ### DataService
 
-**DataService** gRPC can be called by the analyzers to request a stream (ie. [go](https://grpc.io/docs/tutorials/basic/go.html#server-side-streaming-rpc-1), [python](https://grpc.io/docs/tutorials/basic/python.html#response-streaming-rpc)) of the files and changes being reviewed. The [ChangesRequest](https://github.com/src-d/lookout-sdk/blob/master/proto/lookout/sdk/service_data.proto#L58) or [FilesRequest](https://github.com/src-d/lookout-sdk/blob/master/proto/lookout/sdk/service_data.proto#L70) can be configured to ask either for all files, or just the changed ones, as well as UASTs, language, full file content and/or exclude some paths: by regexp, or just all [vendored paths](https://github.com/github/linguist/blob/master/lib/linguist/vendor.yml).
+**DataService** gRPC can be called by the analyzers to request a stream (ie. [go](https://grpc.io/docs/tutorials/basic/go.html#server-side-streaming-rpc-1), [python](https://grpc.io/docs/tutorials/basic/python.html#response-streaming-rpc)) of all the files in the repository being analyzed ([FilesRequest](https://github.com/src-d/lookout-sdk/blob/master/proto/lookout/sdk/service_data.proto#L70)), or only the ones that changed ([ChangesRequest](https://github.com/src-d/lookout-sdk/blob/master/proto/lookout/sdk/service_data.proto#L58)). Both requests can be configured to also require either the files content, UASTs and language; the requests can also filter by programming languages, file paths (by regexp), and exclude the [vendored paths](https://github.com/github/linguist/blob/master/lib/linguist/vendor.yml).
 
 **DataServer** also acts as a proxy to Babelfish, so if an analyzer needs to access the Babelfish API, it can point its [Babelfish client](https://docs.sourced.tech/babelfish/using-babelfish/clients) to the same **DataServer** address.
 
@@ -33,4 +33,23 @@ An analyzer is a gRPC service that will be called by the [Server](#server) to pe
 
 They are not part of **source{d} Lookout** repository so they can be developed by third parties.
 
-source{d} Lookout Server will call all the registered Analyzers to produce comments for the opened Pull Request in the watched repositories. To register new Analyzers in the configuration file, `lookoutd` will need to be restarted.
+**source{d} Lookout** Server will call all the registered Analyzers to produce comments for the opened Pull Request in the watched repositories. To register new Analyzers in the configuration file, `lookoutd` will need to be restarted.
+
+
+# External Services
+
+**source{d} Lookout** will call some external services, some of them depending on the configuration.
+
+## Babelfish
+
+Babelfish is used to parse files into [UAST](https://doc.bblf.sh/uast/uast-specification.html) when the [DataService](#dataservice) `GetChanges` or `GetFiles` methods are asked to include UASTs.
+
+Babelfish can also be directly called through **DataService** proxy.
+
+## RabbitMQ
+
+If **source{d} Lookout** is run in [distributed mode](how-to-run.md#distributed-mode), the watcher will enqueue in RabbitMQ all new repository and PR events, and the workers will take from there the events to be processed.
+
+## PostgreSQL
+
+**source{d} Lookout** uses PostgreSQL to store application internal state.

docs/configuration.md

@@ -1,6 +1,19 @@
 # Configuring source{d} Lookout
 
-**source{d} Lookout** is configured with the `config.yml` file, you can use the template [`config.yml.tpl`](/config.yml.tpl) to create your own. Use the `lookoutd` option `--config` to set the path to it, or use the default location at `./config.yml`.
+The behavior of **source{d} Lookout** is defined with two different configuration files:
+
+- [`config.yml`](#config-yml), to define the global configuration of the server.
+- [`.lookout.yml`](#lookout-yml), to override the analyzer default behavior in each repository.
+
+
+# config.yml
+
+**source{d} Lookout** is configured with the `config.yml` file, you can use the template [`config.yml.tpl`](/config.yml.tpl) to create your own. Use the `lookoutd` option `--config` to set the path to it, or use the default location at `./config.yml`. The config file is read on server startup, so `lookoutd` needs to be restarted in order to load a new configuration.
+
+The most important things you need to configure for a local installation, are:
+
+1. [Repositories](#repositories): define the repositories to be watched.
+1. [Analyzers](#analyzers): add the gRPC addresses of the analyzers to be used by **source{d} Lookout**.
 
 The main structure of `config.yml` is:
 
@@ -16,18 +29,9 @@ timeout:
     # configuration for the existing timeouts.
 ```
 
-## Quickstart
-
-The most important things you need to configure for a local installation, are:
-
-1. [Repositories](#repositories): add the URLs of the repositories to be watched.
-1. [Analyzers](#analyzers): add the gRPC addresses of the analyzers to be used by **source{d} Lookout**.
-
 For more fine grained configuration, you should pay attention to the following documentation.
 
 
-# config.yml
-
 ## Github Provider
 
 The `providers.github` key configures how **source{d} Lookout** will connect with GitHub.
@@ -42,9 +46,8 @@ providers:
     # watch_min_interval: 2s
 ```
 
-`comment_footer` key defines a format-string that will be used for custom messages for every message posted on GitHub; see how to [add a custom message to the posted comments](#custom-footer)
+`comment_footer` key defines a format-string that will be used for custom messages for every message posted on GitHub; see how to [add a custom message to the posted comments](#add-a-custom-message-to-the-posted-comments)
 
-<a id=basic-auth></a>
 ### Authentication with GitHub
 
 **source{d} Lookout** needs to authenticate with GitHub. There are two ways to authenticate with GitHub:
@@ -64,14 +67,13 @@ The credentials can be passed to `lookoutd`:
   - token: `--github-token` argument or `GITHUB_TOKEN` environment variable.
 - per watched repository, following the instructions given by the [Repositories section](#repositories) of this docs.
 
-<a id=github-app></a>
 #### Authentication as a GitHub App
 
 For a production environment, you can use **source{d} Lookout** as a [GitHub App](https://developer.github.com/apps/about-apps/).
 
 To do so, you must also unset any environment variable or option for the GitHub username and token authentication.
 
-You need to create a GitHub App following the [documentation about creating a GitHub app](https://developer.github.com/apps/building-github-apps/creating-a-github-app/). The permissions that must be set are:
+You need to create a GitHub App following the [documentation about creating a GitHub App](https://developer.github.com/apps/building-github-apps/creating-a-github-app/). The permissions that must be set are:
 
 - Repository metadata: Read-only
 - Pull requests: Read & write
@@ -89,17 +91,24 @@ providers:
     watch_min_interval: 10s
 ```
 
+Then you need to install the GitHub App in all your repositories or in a list of them.
+
 When the GitHub App authentication method is used, the repositories to analyze are retrieved automatically from the GitHub installations, so `repositories` list from `config.yml` is ignored.
 
 The update interval to discover new installations and repositories is defined by `installation_sync_interval`.
 
 The minimum watch interval to discover new pull requests and push events is defined by `watch_min_interval`.
 
+#### Web Interface
+
+The **source{d} Lookout** Web Interface to manage the installations of your GitHub App is currently under development, but you can find more details about it and its configuration at [Web Interface docs](web.md)
+
+
 ## Repositories
 
 The list of repositories to be watched by **source{d} Lookout** is defined by:
 - the `repositories` field at `config.yml`, or
-- the repositories where the GitHub App is installed if you [authenticated as a GitHub App](#github-app). In that case, the `repositories` field in `config.yml` will be ignored.
+- the repositories where the GitHub App is installed if you [authenticated as a GitHub App](#authentication-as-a-github-app). In that case, the `repositories` field in `config.yml` will be ignored.
 
 The user and token to be used for the Github authentication can be defined per repository; if you do so, it will override the globally passed user and token.
 
@@ -113,6 +122,7 @@ repositories:
       # minInterval: 1m
 ```
 
+
 ## Analyzers
 
 Each analyzer to be requested by **source{d} Lookout** should be defined under `analyzers` key.
@@ -127,9 +137,8 @@ analyzers:
         threshold: 0.8
 ```
 
-`feedback` key contains the URL used in the custom footer added to any message posted on GitHub; see how to [add a custom message to the posted comments](#custom-footer)
+`feedback` key contains the URL used in the custom footer added to any message posted on GitHub; see how to [add a custom message to the posted comments](#add-a-custom-message-to-the-posted-comments)
 
-<a id=custom-footer></a>
 ### Add a Custom Message to the Posted Comments
 
 You can configure **source{d} Lookout** to add a custom message to every comment that each analyzer returns. This custom message will be created following the rule:
@@ -143,29 +152,6 @@ Example:
 "_If you have feedback about this comment, please, [tell us](%s)._"
 ```
 
-### Customize an Analyzer from the Repository
-
-It's possible to customize the Analyzers configuration for each repository. To do that you only need to place a `.lookout.yml` file at the root directory of that repository.
-
-Example:
-```yaml
-analyzers:
-  - name: Example name
-    disabled: true
-    settings:
-        threshold: 0.9
-        mode: confident
-```
-
-The analyzer to configure must be identified with the same `name` in the `.lookout.yml` config file as in **source{d} Lookout** server configuration, otherwise, it will be ignored.
-
-The repository can disable any analyzer, but it cannot define new analyzers nor enable those that are disabled in the **source{d} Lookout** server.
-
-The `settings` for each analyzer in the `.lookout.yml` config file will be merged with the **source{d} Lookout** configuration following these rules:
-
-- Objects are deep merged
-- Arrays are replaced
-- Null value replaces object
 
 ## Timeouts
 
@@ -189,3 +175,28 @@ timeout:
   # Timeout for Bblfsh to reply to a Parse request
   bblfsh_parse: 2m
 ```
+
+
+# .lookout.yml
+
+It's possible to customize the Analyzers configuration for each repository. To do that you only need to place a `.lookout.yml` file at the root directory of that repository.
+
+Example:
+```yaml
+analyzers:
+  - name: Example name
+    disabled: true
+    settings:
+        threshold: 0.9
+        mode: confident
+```
+
+The analyzer to configure must be identified with the same `name` in the `.lookout.yml` config file as in **source{d} Lookout** server configuration, otherwise, it will be ignored.
+
+The repository can disable any analyzer, but it cannot define new analyzers nor enable those that are disabled in the **source{d} Lookout** server.
+
+The `settings` for each analyzer in the `.lookout.yml` config file will be merged with the **source{d} Lookout** configuration following these rules:
+
+- Objects are deep merged
+- Arrays are replaced
+- Null value replaces object

docs/how-to-run.md

@@ -23,6 +23,8 @@ The general steps to run source{d} Lookout in distributed mode are the same as s
 
 # Dependencies
 
+_For more details about the purpose of these external dependencies, you can take a look at [**External services** in **Architecture** documentaion](architecture.md#external-services)_
+
 **source{d} Lookout** needs a running instance of:
 
 * [bblfshd](https://github.com/bblfsh/bblfshd) to parse files into [UAST](https://doc.bblf.sh/uast/uast-specification.html).
@@ -73,15 +75,15 @@ Download the latest `lookoutd` binary from [**source{d} Lookout** releases page]
 
 ## Initialize the Database
 
-_For non-default configuration, please take a look into [**`lookoutd` Command Options**](#options)_
+_For non-default configuration, please take a look into [**`lookoutd` Command Options**](#appendix-lookoutd-command-options)_
 
 ```shell
 $ lookoutd migrate
 ```
 
 ## Start source{d} Lookout
 
-_For non-default configuration, please take a look into [**`lookoutd` Command Options**](#options)_
+_For non-default configuration, please take a look into [**`lookoutd` Command Options**](#appendix-lookoutd-command-options)_
 
 For a single server watching GitHub and processing events, just run:
 
@@ -91,7 +93,7 @@ $ lookoutd serve [--dry-run] [--github-token=<token> --github-user=<user>]
 
 ### Distributed Mode
 
-_For non-default configuration, please take a look into [**`lookoutd` Command Options**](#options)_
+_For non-default configuration, please take a look into [**`lookoutd` Command Options**](#appendix-lookoutd-command-options)_
 
 In order to run it in a distributed mode, the _watcher_ and the _workers_ must be run separately.
 
@@ -107,8 +109,7 @@ and as many _workers_ you need:
 $ lookoutd work [--dry-run] [--github-token=<token> --github-user=<user>]
 ```
 
-<a id=options></a>
-# Appendix: `lookoutd` Command Options
+# Appendix: lookoutd Command Options
 
 `lookoutd` binary includes some subcommands as described above, and they accept many different options; you can use:
 - `lookoutd -h`, to see all the available subcommands.
@@ -118,7 +119,7 @@ Here are some of the most relevant options for `lookoutd`:
 
 - [dry-run mode](#dry-run-mode)
 - [authentication options](#authentication-options)
-- [number of concurrent events to process](#concurrent-events)
+- [number of concurrent events to process](#number-of-concurrent-events-to-process)
 - [dependencies URIs](#dependencies-uris)
 - [logging options](#logging-options)
 
@@ -138,7 +139,6 @@ To post the comments returned by the Analyzers into GitHub, you can configure th
 | `serve`, `work`, `watch` | `GITHUB_USER`  | `--github-user=` |
 | `serve`, `work`, `watch` | `GITHUB_TOKEN`  | `--github-token=` |
 
-<a id=concurrent-events></a>
 ## Number of Concurrent Events to Process
 
 You can adjust the number of events that each _worker_ or the single _server_ will process concurrently when running `serve` or `work` subcommands (if you set it to `0`, it will process as many as the number of processors you have):

docs/lookout-sdk.md

@@ -40,6 +40,8 @@ To perform a `NotifyPushEvent` call to an analyzer and serve the **source{d} Loo
 $ lookout-sdk push
 ```
 
+`lookout-sdk` will exit with a non `0` exit code if the gRPC call to the analyzer ends with an error or it could not be performed.
+
 In the next section, you will find a more detailed example considering the most usual options for running `lookout-sdk` against any analyzer from two given revisions.
 
 
@@ -90,7 +92,7 @@ $ lookout-sdk review \
   "ipv4://localhost:9999"
 ```
 
-_For more options to run `lookout-sdk`, take a look into [**lookout-sdk Command Options**](#options)_
+_For more options to run `lookout-sdk`, take a look into [**lookout-sdk Command Options**](#appendix-lookout-sdk-command-options)_
 
 - If analyzer gRPC address is omitted, it will be `ipv4://localhost:9930`.
 - If `--git-dir` is omitted, the current dir will be used.
@@ -101,8 +103,7 @@ _For more options to run `lookout-sdk`, take a look into [**lookout-sdk Command
 Everything explained above for `lookout-sdk review` calling `NotifyReviewEvent`, applies also to `NotifyPushEvent` when using `lookout-sdk push`.
 
 
-<a id=options></a>
-# Appendix: `lookout-sdk` Command Options
+# Appendix: lookout-sdk Command Options
 
 `lookout-sdk` binary include some subcommands as described above, and they accept many different options; you can use:
 - `lookout-sdk -h`, to see all the available subcommands.