Adds some documentation for the Sweep Hive feature (#26)

* add some docs

* add sweep documentation

* update docs

ci/util.py

@@ -1,3 +1,4 @@
+import os
 import subprocess
 
 
@@ -6,7 +7,8 @@ def pants(*args):
         [
             "pants",
             *args,
-        ]
+        ],
+        env={**os.environ, "PANTS_CONCURRENT": "true"},
     )
 
 

docs/src/SUMMARY.md

@@ -5,6 +5,7 @@
 # Getting started
 
 - [Setup](setup.md)
+- [🚅 Cheat Sheet](cheats.md)
 
 # Documentation (WIP, imported)
 
@@ -21,6 +22,7 @@
 # Workflows
 
 - [🖥 Local training](local-training.md)
+- [⚗ Sweeps in Hive](workflows/sweeps.md)
 
 # Concepts
 

docs/src/cheats.md

@@ -0,0 +1,17 @@
+# 🚅 Quick Reference
+
+#### Live-edit documentation:
+
+Starts a HTTP server on `:8000` and continuously rebuilds these docs in the background.
+
+```shell
+$ pants run scripts/docs.py
+```
+
+#### Use a different emote version
+
+Update the lockfiles to use a different version of emote. Note that if you specify a branch, you'll still need to rerun the script if the branch changes.
+
+```shell
+$ pants run ci/emote-override.sh -- TREE_ISH
+```

docs/src/workflows/sweeps.md

@@ -0,0 +1,109 @@
+# ⚗ Sweeps in Hive
+
+Hive supports a very simple form of sweeping using handlebars. This enables quick exploration of a set of parameters with less manual editing.
+
+To use, this, start by checking the `Use sweeps` checkbox at the top of the configuration editor.
+
+![Checkbox location above the configuration editor](checkbox.png)
+
+After doing this, Hive will add an extra button to your configuration editor for configuring sweeps:
+
+![New sweep.json tab to the right](newtab.png)
+
+This tab is where we'll edit our config. There's a default configuration provided; to help you get started. The format is as follows:
+
+
+```json
+{
+    "mode": "grid",
+    "parameters": {
+        "template-key": [ value-1, value-2, ...],
+        "another-key": [ "string-v1", "string-v2" ]
+    }
+}
+```
+
+Currently only atom values (strings, numbers, booleans, null) are supported! If you have more complex needs, please make a feature request and we can design it. Note that values will be inserted verbatim.
+
+As an example, we could define a grid search of hidden-layers and action frequency like this:
+
+```json
+{
+    "mode": "grid",
+    "parameters": {
+        "hidden": [ 256, 512, 1024 ],
+        "action-period": [ 3, 4, 5 ]
+    }
+}
+```
+
+Each of the keys `hidden` and `action-period` must now appear in the configuration *and* your trial name. Let's start with the trial name:
+
+```
+hidden-size-${hidden}-frequency-${action-period}
+```
+
+When you click start, the string will in turn be substituted as the following:
+
+```
+hidden-size-256-frequency-3
+hidden-size-256-frequency-4
+hidden-size-256-frequency-5
+hidden-size-512-frequency-3
+hidden-size-512-frequency-4
+hidden-size-512-frequency-5
+hidden-size-1024-frequency-3
+hidden-size-1024-frequency-4
+hidden-size-1024-frequency-5
+```
+
+Next, we'll do the same for our `protocol.json` for the `hidden` key, and to our `game.json` for the `action-period` key. Note that you'll always quote the key in the JSON, even if it's a number. The quotes are removed again when the value is substituted, except if the value is a string.
+
+
+```json
+// protocol.json
+{
+    "learning-rate": 0.0001,
+    "batch-size": 4000,
+    "hidden": "${hidden}",
+    ...
+}
+```
+
+```json
+// game.json
+{
+    "version": 1,
+    "creature_id": "79A7E7E6657A0D0F0C9175DF20748FF3",
+    "action_period": "${action_period}",
+    "observation_settings": { ... }
+}
+```
+
+The result is that we'll have 9 trials, each with a different combination of hidden size and action frequency. As you can see in the second image, there is a bit of an guide, which will help you get started.
+
+## Configuration details
+
+**`mode`**
+
+The `mode` value can be either `"grid"` or `"sequence"`.
+
+The `grid` mode will create a cartesian product of all the values, which means you'll generate a lot of trials very quickly. To avoid issues on the cluster, there's currently a cap of 10 experiments for this reason.
+
+the `sequence` mode will create a sequence of all the values. To ensure all keys are present it will use the first value for each key as default. If we start by visualizing the configuration as a grid, the sequence mode will only explore the cardinal directions starting from the top left corner.
+
+**`parameters`**
+
+The `parameters` key is a dictionary of keys and values. The keys must be present in the configuration and the trial name. The values are a list of values to substitute for the key.
+
+**`sampleSize`** *(optional)*
+
+The `sampleSize` key can be used to limit the number of trials that are generated. This is useful if you've got a large grid and you only want to test a few values. See `seed` for more details.
+
+**`seed`** *(optional)*
+
+The `seed` key can be used to set the seed for the random number generator. This is useful if you want to generate the same set of trials multiple times. If you don't set this, the generator will be seeded from a monotonic high-resolution clock.
+
+## Limitations
+
+* Sweep configuration happens in your browser and isn't stored anywhere, so you'll have to re-enter it if you refresh the page. This also means that you can't use the same sweep configuration for multiple experiments.

scripts/BUILD

@@ -0,0 +1 @@
+python_sources()

scripts/docs.py

@@ -0,0 +1,41 @@
+import http.server
+import os
+import socketserver
+import subprocess
+import threading
+from functools import partial
+
+
+def pants(*args):
+    return subprocess.check_output(
+        [
+            "pants",
+            *args,
+        ],
+        env=os.environ
+        | {
+            "PANTS_CONCURRENT": "true",
+        },
+    )
+
+
+class Server(socketserver.TCPServer):
+    allow_reuse_address = True
+
+
+def build_docs():
+    pants("--loop", "package", "docs:docs")
+
+
+def main():
+    threading.Thread(target=build_docs).start()
+
+    Handler = partial(http.server.SimpleHTTPRequestHandler, directory="dist/docs/book/")
+    server = Server(("", 8000), Handler)
+    server.allow_reuse_address = True
+
+    with server as httpd:
+        httpd.serve_forever()
+
+
+main()