preprocessor.md

Mustbuild Preprocessor

The preprocessor supports the Mustbuild language extensions.

The core of the preprocessor is based on Jsonnet, a side-effect free configuration language for generating JSON. The generated JSON can be interpreted as Justbuild expression language, which serves as an intermediate representation between the Mustbuild frontend and backend. Each input file to the build system (TARGET, RULES, EXPRESSIONS, etc.) will be preprocessed automatically. Preprocessing is idempotent (processing JSON will not result in logical changes) and thereby compatibility with Justbuild projects is maintained.

Subcommand preprocess

Using the subcommand preprocess, the preprocessor can also be run manually to produce the intermediate JSON representation:

$ echo "var('val')" | must preprocess -
{
  "name": "val",
  "type": "var"
}

The command line option --root can be used to specify the file root, the preprocessed file belongs to. This is particularly useful if the file contains imports, which are restricted to exactly one file root.

IDE integration

The Mustbuild preprocessor can also be easily integrated into existing IDEs to lint your code and preview the generated Justbuild expressions. Simply set the helper tool must-lint as the compiler for your IDE's Jsonnet plugin (see IDE integration).

Example code

The following EXPRESSIONS file is implemented using the Mustbuild language extensions:

{
  // test if list contains item
  contains: {
    vars: ['list', 'item'],
    expression: or(foreach('x', var('list'), eq(var('x'), var('item')))),
  },
}

Compare this to the output below, which shows the Justbuild expressions generated by the preprocessor. Note that the field expression, which was implemented as a one-liner, expands to 13 lines of JSON code.

{
  "contains": {
    "vars": ["list","item"],
    "expression": {
      "type": "or",
      "$1": {
        "type": "foreach",
        "var": "x",
        "range": {"type": "var", "name": "list"},
        "body": {
          "type": "==",
          "$1": {"type": "var", "name": "x"},
          "$2": {"type": "var", "name": "item"}
        }
      }
    }
  }
}

Using Jsonnet

As side-effect free configuration language, Jsonnet is a perfect match for generating JSON to be consumed by a deterministic build system. Advantages of Jsonnet over JSON are

• a simplified syntax
• support for comments
• type assertions
• file-local variables and functions
• code reuse across multiple files via imports
• and many more...

Multi-line strings

One of Jsonnet's advantages is the support for multi-line strings (operator |||), which produce a single JSON string with proper encoding:

$ echo '|||
  #!/bin/sh

  while [ -n "$1" ]; do
    echo "got $1"; shift
  done
|||' | jsonnet -
"#!/bin/sh\n\nwhile [ -n \"$1\" ]; do\n  echo \"got $1\"; shift\ndone\n"

Caveats

Jsonnet is a very powerful language. However, using it as a preprocessor language for Justbuild expressions implicate a few important restrictions:

• Object literals
  Object literals {} are still being considered to be Justbuild expressions. Therefore, in an evaluated context, objects (aka maps) can only be created using expressions that evaluate to objects.

• Jsonnet imports
  Imports can only be used for files of the same file root. File paths with leading / are considered to be anchored at the root of the (potentially content-defined) file root. It is not possible (and desireable) to import files from outside the file root.

• Jsonnet operators
  All Jsonnet operators (+, -, *, /, ==, <, >, [], etc.) can only be used if none of the operands is a Mustbuild expression.
  Example: select(true,"foo","bar") == "foo" will always be false

• Jsonnet standard library
  While not enforced, using the Jsonnet standard library is highly discouraged, as currently no guarantees are given which Jsonnet version is actually being used.