re-org

site/content/docs/guide/plugin/for-developers.md

@@ -1,4 +1,8 @@
-## Creating a New Plugin
+---
+title: For Developers
+---
+
+# Creating a New Plugin
 
 A Hipcheck plugin is a separate executable artifact that Hipcheck downloads,
 starts, and communicates with over a gRPC protocol to request data. A plugin's
@@ -14,11 +18,11 @@ compile their plugin for, and more likely will need to support a handful of
 target platforms. This can be simplified through the optional use of container
 files as the plugin executable artifact.
 
-Once a plugin author writes their plugin, compiles, packages, and distribute it (@Todo - add link to
-distributing section), Hipcheck users can specify the plugin in their policy
-file for Hipcheck to fetch and use in analysis.
+Once a plugin author writes their plugin, compiles, packages, and distribute it
+(@Todo - add link to distributing section), Hipcheck users can specify the 
+plugin in their policy file for Hipcheck to fetch and use in analysis.
 
-#### Plugin CLI
+## Plugin CLI
 
 Hipcheck requires that plugins provide a CLI which accepts a `--port <PORT>`
 argument, enabling Hipcheck to centrally manage the ports plugins are listening
@@ -29,7 +33,7 @@ Once started, the plugin should continue running, listening for gRPC requests
 from Hipcheck, until shut down by the Hipcheck process. For information on the
 Hipcheck gRPC protocol, see [here](@Todo).
 
-### The Rust SDK
+## The Rust SDK
 
 The Hipcheck team maintains a library crate `hipcheck-sdk` which provides
 developers with tools for greatly simplifying plugin development in Rust. This
@@ -42,23 +46,22 @@ to all the essential types it exposes. If you want to manage your imports to
 avoid potential type name collisions you may do so, otherwise simply write `use
 hipcheck_sdk::prelude::*`.
 
-#### Defining Query Endpoints
+### Defining Query Endpoints
 
 The Hipcheck plugin communication protocol allows a plugin to expose multiple
 named query endpoints that can be called by Hipcheck core or other plugins.
 Developers may choose to use the `query` [attribute macro](#using-proc-macros)
 to mark functions as endpoints, or [manually implement](#manual-implementation)
 the `Query` trait.
 
-##### Using Proc Macros
+#### Using Proc Macros
 
-The SDK offers an attribute proc macro for marking `async` functions with an
+The SDK offers an attribute proc macro `query` for marking `async` functions with an
 appropriate signature as query endpoints. The function signature must be of the
 form
 
 ```rust
-async fn [FUNC_NAME](engine: &mut PluginEngine, input: [INPUT_TYPE]) ->
-Result<[OUTPUT_TYPE]>
+async fn [FUNC_NAME](engine: &mut PluginEngine, input: [INPUT_TYPE]) -> Result<[OUTPUT_TYPE]>
 ```
 
 Where:
@@ -72,12 +75,12 @@ apply the `#[query]` attribute to the function.
 
 Importantly, this attribute will create a struct with Pascal-case version of
 your function name (e.g. `foo_bar()` -> `struct FooBar`). You will need this
-`struct` name to implement `Plugin::queries()` [below](#the-plugin-trait).
+struct name to implement `Plugin::queries()` [below](#the-plugin-trait).
 
 For a description of how the `PluginEngine` is used to query other plugins, see
 [below](#querying-other-plugins).
 
-##### Manual Implementation
+#### Manual Implementation
 
 For each query endpoint you want to define, you must create a struct that
 implements the `Query` trait from the `prelude`. `Query` is declared as such:
@@ -105,7 +108,7 @@ static MY_QUERY_KEY_SCHEMA: &str = include_str!("../schema/my_query_key_schema.j
 static MY_QUERY_OUTPUT_SCHEMA: &str = include_str!("../schema/my_query_output_schema.json");
 ```
 
-##### The `Query::run()` Function
+#### The `Query::run()` Function
 
 The `run()` function is the place where your actual query logic wil go. Let's
 look at it in more detail. It's an `async` function since the underlying SDK
@@ -131,7 +134,7 @@ simply perform the calculations, serialize the output type to a JSON value, and
 return it wrapped in `Ok`. However, many plugins will rely on additional data from other
 plugins. In the next subsection we will describe how to do that in more detail.
 
-##### Querying Other Plugins
+#### Querying Other Plugins
 
 As mentioned above, the `run()` function receives a handle to a `PluginEngine` instance which exposes
 the following generic function:
@@ -162,14 +165,14 @@ query; by omitting the `/query` from your target string, you are targetting the
 default query endpoint for the plugin. If you don't want to pass a `String` to
 `target`, you can always instantiate a `QueryTarget` yourself and pass that.
 
-#### The `Plugin` Trait
+### The `Plugin` Trait
 
 At this point, you should have one struct that implements `Query` for each
 query endpoint you want your plugin to expose. Now, you need to implement the
 `Plugin` trait which will tie everything together and expose some additional
 information about your plugin to Hipcheck. The `Plugin` trait is as follows:
 
-```
+```rust
 trait Plugin: Send + Sync + 'static {
 
 	const PUBLISHER: &'static str;
@@ -193,7 +196,7 @@ pub struct NamedQuery {
 type DynQuery = Box<dyn Query>;
 ```
 
-The `PUBLISHER` and `NAME` associated `str`s allow you to declare the publisher
+The associated strings `PUBLISHER` and `NAME` allow you to declare the publisher
 and name of the plugin, respectively.
 
 The `set_config()` function allows Hipcheck users to pass a set of `String`
@@ -223,7 +226,7 @@ the Hipcheck policy file.
 If you do define a default query endpoint, `Plugin::explain_default_query()` should return a `Ok(Some(_))` containing a string that explains the default query.
 
 Lastly, if yours is an analysis plugin, users will need to write [policy
-expressions](@Todo - link) to interpret your plugin's output. In many cases, it
+expressions](policy-expr) to interpret your plugin's output. In many cases, it
 may be appropriate to define a default policy expression associated with your
 default query endpoint so that users do not have to write one themselves. This
 is the purpose of `default_policy_expr()`. This function will only ever be
@@ -234,7 +237,7 @@ generally will be compared against an integer/float threshold, you can return a
 `(lte $ <THRESHOLD>)` where `<THRESHOLD>` may be a value received from
 `set_config()`.
 
-#### Running Your Plugin
+### Running Your Plugin
 
 At this point you now have a struct that implements `Plugin`. The last thing to do is write some boilerplate code for starting the plugin server. The Rust SDK exposes a `PluginServer` type as follows:
 
@@ -256,7 +259,7 @@ impl<P: Plugin> PluginServer<P> {
 
 So, once you have parsed the port from the CLI `--port <PORT>` flag that
 Hipcheck passes to your plugin, you simply pass an instance of your `impl
-Plugin` struct to `PluginServer::register()`, then call `listen(<PORT>)` on the
+Plugin` struct to `PluginServer::register()`, then call `listen(<PORT>).await` on the
 returned `PluginServer` instance. This function will not return until the gRPC
 channel with Hipcheck core is closed.
 

site/content/docs/guide/plugin/for-users.md

@@ -0,0 +1,178 @@
+---
+title: For Users
+---
+
+When running Hipcheck, users provide a "policy file", which is a KDL-language
+configuration file that describes everything about how to perform the analysis.
+It specifies which top-level plugins to execute, how to configure them, how to
+interpret their output using [policy expressions](#policy-expressions), and how
+to weight the pass/fail result of each analysis when calculating a final score.
+
+Let's now look at an example policy file to examine its parts more closely:
+
+```
+plugins {
+    plugin "mitre/activity" version="0.1.0"
+    plugin "mitre/binary" version="0.1.0"
+    plugin "mitre/fuzz" version="0.1.0"
+    plugin "mitre/review" version="0.1.0"
+    plugin "mitre/typo" version="0.1.0"
+    plugin "mitre/affiliation" version="0.1.0"
+    plugin "mitre/entropy" version="0.1.0"
+    plugin "mitre/churn" version="0.1.0"
+}
+
+analyze {
+    investigate policy="(gt 0.5 $)"
+    investigate-if-fail "mitre/typo" "mitre/binary"
+
+    category "practices" {
+        analysis "mitre/activity" policy="(lte $ 52)" weight=3
+        analysis "mitre/binary" policy="(eq 0 (count $))" {
+            binary-file "./config/Binary.toml"
+        }
+        analysis "mitre/fuzz" policy="(eq #t $)"
+        analysis "mitre/review" policy="(lte $ 0.05)"
+    }
+
+    category "attacks" {
+        analysis "mitre/typo" policy="(eq 0 (count $))" {
+            typo-file "./config/Typos.toml"
+        }
+
+        category "commit" {
+            analysis "mitre/affiliation" policy="(eq 0 (count $))" {
+                orgs-file "./config/Orgs.toml"
+            }
+
+            analysis "mitre/entropy" policy="(eq 0 (count (filter (gt 8.0) $)))" {
+                langs-file "./config/Langs.toml"
+            }
+            analysis "mitre/churn" policy="(lte (divz (count (filter (gt 3) $)) (count $)) 0.02)" {
+                langs-file "./config/Langs.toml"
+            }
+        }
+    }
+}
+```
+
+As you can see, the file has two main sections: a `plugins` section, and an
+`analyze` section. We can explore each of these in turn.
+
+## The `plugin` Section
+
+This section defines the plugins that will be used to run the analyses
+described in the file. These plugins are defined with a name, version, and an
+optional manifest field (not shown in the example above) which provides a link
+to the plugin's download manifest. For an example of the manifest field, see
+[@Todo - link to For-Developers section]. In the future, when a Hipcheck plugin
+registry is established, the manifest field will become optional. In the
+immediate term it will be practically required.
+
+At runtime, each plugin will be downloaded by Hipcheck, its size and checksum
+verified, and the plugin contents decompressed and unarchived to produce the
+plugin executable artifacts which will be stored in a local plugin cache.
+Hipcheck will do the same recursively for all plugins.
+
+In the future Hipcheck will likely add some form of dependency resolution to
+minimize duplication of shared dependencies, similar to what exists in other
+more mature package ecosystems. For now the details of this mechanism are left
+unspecified.
+
+## The `analysis` Section
+
+Whereas the `plugin` section is simply a flat list telling Hipcheck which
+plugins to resolve and start up, the `analysis` section composes those
+analyses into a score tree.
+
+The score tree is comprised of a series of nested "categories" that eventually
+terminate in analysis leaf nodes. Whether an analysis appears in this tree
+determines whether Hipcheck actually queries it, so although you can list
+plugins in the `plugins` section that do not appear in the `analysis` section,
+they will not be run. On the contrary, specifying a plugin in the `analysis`
+section that is not in the `plugins` section is an error.
+
+### The Score Tree
+
+Each category and analysis node in the tree has an optional `weight` field,
+which is an integer that dictates how much or little that node's final score of
+0 or 1 (pass and fail, respectively) should count compared to its neighbors at
+the same depth of the tree. If left unspecified, the weight of a node defaults
+to `1`.
+
+Once all the weights are normalized, an individual analysis's contribution to
+Hipcheck's final score for a target can be calculated by multiplying its own
+weight and the weight of all its parent categories up to the top of the
+`analysis` section. As each analysis produces a pass/fail result, the
+corresponding `0` or `1` is multiplied with that analysis's contribution
+percentage and added to the overall score.
+
+Users may also run `hc scoring --policy <FILE_PATH>` to see a version of the
+score tree with normalized weights for a given policy file.
+
+See [the Complete Guide to Hipcheck's section on scoring][hipcheck_scoring]
+for more information on how Hipcheck's scoring mechanism works.
+
+### Configuration
+
+A plugin author may choose to provide a set of parameters so that users may
+configure the plugin's behavior. These can be set inside the corresponding
+brackets for each analysis node. For example, see the `binary-file`
+configuration inside `mitre/binary`. The provided key-value pairs are passed to
+their respective plugins at startup.
+
+### Policy Expressions
+
+Hipcheck plugins return data or measurements on data in JSON format, such that
+other plugins could be written to consume and process their output. However,
+the scoring system of Hipcheck relies on turning the output of each top-level
+plugin into a pass/fail evalution. In order to facilitate transforming plugin
+data into a boolean value, Hipcheck provides "policy expressions", which are a
+small expression language. See [here](policy-expr) for a reference on the policy
+expression language.
+
+Users can define the pass/fail policy for an analysis node in the score tree
+with a `policy` key. As described in more detail in the policy expression
+reference, a policy used in analysis ought to take one or more JSON pointers
+(operands that start with `$`) as entrypoints for part or all of the JSON object
+returned by the analysis to be fed into the expression. Additionally,
+all policies should ultimately return a boolean value, with `true`
+meaning that the analysis passed.
+
+Instead of users always having to define their own policy expressions, plugins
+may define a default pass/fail policy that may depend on configuration items
+that the plugin accepts in the `analysis` section of the policy file. If a
+plugin's default policy is acceptable, the user does not need to provide a
+`policy` key when placing that plugin into a scoring tree in their policy file.
+If the default policy is configurable, the user can configure it by setting the
+relevant configuration item for the plugin. Note that any user-provided policy
+will always override the default policy.
+
+Finally, if the policy expression language is not powerful enough to express a
+desired policy for a given analysis, users may define their own plugin which
+takes the analysis output, performs some more complicated computations on it,
+and use that as their input for the score tree.
+
+### Final Scoring and Investigation
+
+Once the policies for each top-level analysis has been evaluated, the score
+tree produces the final score. Hipcheck now looks at the `investigate` field of
+the policy file.
+
+This node accepts a `policy` key-value pair, which takes a policy expression as
+a string. The input to the policy expression is the numeric output of the
+scoring tree reduction, therefore a floating pointer number between 0.0 and 1.0
+inclusive. This defines the policy used to determine if the "risk score"
+produced by the score tree should result in Hipcheck flagging the target of
+analysis for further investigation.
+
+The `investigate-if-fail` node enables users of Hipcheck to additionally mark
+specific analyses such that if those analyses produce a failed result, the
+overall target of analysis is marked for further investigation regardless of
+the risk score. In this case, the risk score is still calculated and all other
+analyses are still run.
+
+// In this way, Hipcheck provides users extensive flexibility in both defining
+risk and the set of measurements used to evaluate it. Additionally, the plugin
+system enables users and third-party developers to write their own plugins to
+extend Hipcheck's measurement and analysis capabilities.