Update docs

Updates the documentation on a few places to make it easier for new
people to get whats going on.

README.md

@@ -1,5 +1,15 @@
 # Wikibase release pipeline
 
+## Repository overview
+
+The wikibase release pipeline contains scripts used for building, testing and publishing Wikibase docker images and tarballs.
+
+It contains a set of build targets defined in the [Makefile](./Makefile) which can be executed in two different ways.
+
+When [building](docs/topics/pipeline.md), use the [build.sh](build.sh) wrapper script to do the building inside a docker-container, this way files aren't littered around the repository but only in the `artifacts/` directory.
+
+For [testing](docs/topics/testing.md), you can use `test`, `test-all` make targets.
+
 ## Documentation
 
 The pipeline documentation can be found [here](docs/index.md).

docs/index.md

@@ -1,6 +1,7 @@
 ## Wikibase release pipeline
 
 ### Documentation
+- [Overview](topics/release-overview.md)
 - [Running the pipeline](topics/pipeline.md)
 - [Testing](topics/testing.md)
 - [Publishing](topics/publishing.md)

docs/topics/pipeline.md

@@ -1,29 +1,4 @@
 # The pipeline
-
-## The local.env file
-
-On build a local.env file can be used to override any default settings
-
-#### Use locally installed selenium test dependencies
-```
-SKIP_INSTALL_SELENIUM_TEST_DEPENDENCIES=1
-```
-#### Override for branch name of extensions cloned from gerrit
-```
-GERRIT_EXTENSION_BRANCH_NAME=REL1_35
-```
-#### Lower the compression rate to make local builds faster
-```
-GZIP_COMPRESSION_RATE=1
-```
-#### Log level for selenium tests see wdio.conf.js for options
-```
-SELENIUM_LOG_LEVEL=trace
-```
-
-#### Settings related to tarball publishing
-See [publishing](publishing.md).
-
 ## The versions folder
 
 The `versions` folder contains .env files with the specific versions of the components to be build for that specific release.
@@ -36,6 +11,21 @@ WIKIBASE_BRANCH_NAME=REL1_35
 MEDIAWIKI_BRANCH_NAME=REL1_35
 ...
 ``` 
+## Github Actions Workflows
+
+At it's core this repository is just a set of make targets that gets orchestrated by Github Actions workflows. We currently have the following workflows.
+#### [.github/workflows/built_and_test.yml](.github/workflows/built_and_test.yml)
+
+This is the main workflow that builds Wikibase and related software, tests it and then produces release candidate artifacts that can be used for further acceptance testing and publishing.
+#### [.github/workflows/lint.yml](.github/workflows/lint.yml)
+
+This is internal linting for the release pipeline, it does not lint any of the release artifacts.
+#### [.github/workflows/test.yml](.github/workflows/test.yml)
+
+This workflow is used to run the tests against a previous build without having to download them.
+#### [.github/workflows/publish.yml](.github/workflows/publish.yml)
+
+This workflow is used to publish the artifacts of a workflow from Github.
 
 ## Running the pipeline on github
 
@@ -53,13 +43,86 @@ To manually run the pipeline go to to actions, click the workflow and run the wo
 
 ![Queuing the pipeline](../images/queue_job.gif "Queuing the pipeline")
 
+## Artifacts on github
+
+After triggering the pipeline a set of jobs will start running, which after a successful run will result in a number of artifacts as described in the diagram below.
 
-## Building artifacts locally
+![Queuing the pipeline](../../diagrams/output/overview.svg "Queuing the pipeline")
+
+`BuildArtifacts` - contains the release candidate artifact for each component that was built.
+
+`BuildMetadata` - contains artifacts describing what was built for each component that is included. Also contains the artifacts produced by the finishing `version` job that describe which versions were used when building / testing.
+
+`TestArtifacts` - contains logs and screenshots from testing.
+
+`TestExampleArtifacts` - contains logs from testing done to the `example/` folder
+
+`TestUpgradeArtifacts` - contains logs from upgrades from older versions to this newly built one.
+
+## Running the pipeline locally
+
+To execute the entire build pipeline you can use the `all` make target.
+
+```sh
+./build.sh all versions/wmde1.env
+```
 
-To build artifacts using docker run the following command which would execute the wikibase pipeline step and produce the artifacts in the `artifacts` folder using the `wmde1.env` environment file. See the [Makefile](../../Makefile) for more build options.
+To build single artifacts locally you can issue the following command.
 
 ```sh
-bash build.sh wikibase versions/wmde1.env
+./build.sh wikibase versions/wmde1.env
 ```
 
-[Wikibase REL1_35]: https://gerrit.wikimedia.org/g/mediawiki/extensions/Wikibase/+/refs/heads/REL1_35
+The output ends up in the `artifacts/` folder and will be built as described by the `versions/wmde1.env` environment file.
+
+
+See the [Makefile](../../Makefile) for more build options.
+
+## Local artifacts
+
+When building locally the artifacts will be output into the root of the `artifacts/` folder as they are not into zip packages as on Github.
+
+
+### Clean local artifacts
+To remove any locally produced artifacts you can run the following commands.
+
+```sh
+make clean
+```
+
+To clean the git_cache used for building
+
+```sh
+make clean-cache
+```
+### Downloaded artifacts
+
+Any downloaded artifacts from workflow runs on github actions will be saved as a folder under `artifacts/<WORKFLOW_ID>` and are exempt from any cleaning commands.
+
+For more info on downloading artifacts from github see [publishing](publishing.md).
+
+[Wikibase REL1_35]: https://gerrit.wikimedia.org/g/mediawiki/extensions/Wikibase/+/refs/heads/REL1_35
+
+## The local.env file
+
+On build a local.env file can be used to override any default settings
+
+#### Use locally installed selenium test dependencies
+```
+SKIP_INSTALL_SELENIUM_TEST_DEPENDENCIES=1
+```
+#### Override for branch name of extensions cloned from gerrit
+```
+GERRIT_EXTENSION_BRANCH_NAME=REL1_35
+```
+#### Lower the compression rate to make local builds faster
+```
+GZIP_COMPRESSION_RATE=1
+```
+#### Log level for selenium tests see wdio.conf.js for options
+```
+SELENIUM_LOG_LEVEL=trace
+```
+
+#### Settings related to tarball publishing
+See [publishing](publishing.md).

docs/topics/release-overview.md

@@ -0,0 +1,7 @@
+# Wikibase release process overview
+
+Wikibase and related software gets release through this repository. The process for releasing involves defining the release in this repository, testing all the components of the release together and finally publishing them.
+
+The diagram below describes the general steps involved in making a release.
+
+![Release workflow overview](../../diagrams/output/release-workflow.svg "Release workflow overview")