At a high level, monopacker constructs a Packer configuration and then invokes packer for you. So understanding the basic concepts of Packer (builders, provisioners, and so on) is critical to understanding how Monopacker works.
The following is a "top-down" description of how Monpacker operates. The "bottom" is where most of the interesting stuff occurs, so read on!
Warning some words here have multiple meanings!
- template - We use Jinja2 templates (textually substituting things like
{{..}}) as well as a Packer template (which is just what Packer calls its input data) - builder - Monpacker defines Monopacker builders that map loosely to Packer builders, but also specify scripts that are included in Packer provisioners.
At the top level, monopacker builds a "Packer template", which is the input to the Packer executable. The Packer docs explain in detail the contents of this file.
In this Packer template, each Monopacker builder defined in builders/ is included as a Packer builder.
The Packer provisioners are set up such that, after some initial shared setup, one provisioner runs for each builder.
This is a shell provisioner for linux builders and a powershell provisioner for windows builders, configured to run the scripts for that builder.
There is a one-to-one correspondance between Monopacker builders and images (ignoring duplication of images to multiple regions). That is, there is a distinct builder defined for each necessary combination of worker implementation, cloud, and platform. Additional builders can be defined for specific purposes, such as specific software installed or specific secrets.
Since this arrangement results in a great deal of duplication across builders, the configuration is designed to maximize the reuse of components.
Each builder is defined in a YAML file in ./builders.
The builder's YAML file must specify a template, which corresponds to a Jinja2 template file in ./template/builders.
Builders can specify any template, and multiple builders can specify the same template.
Builders have a few pieces of configuration:
template(required): which template in ./template/builders to useplatform(required): the os the builder uses, such as linux or windowsbuilder_var_filesandbuilder_vars: variables for use in the template (see below)script_directories: directories containing scripts to run to build the image
Reuse is accomplished through sharing variable files and script directories between builders.
For example, a script directory foo-worker-linux might configure the foo-worker implementation on linux.
This would then be included in script_directories for the foo-worker-linux-azure, foo-worker-linux-aws, and foo-worker-linux-alibaba builders.
Templates typically have variables that must be specified. For example,
the vagrant builder references the following variables:
builder.vars.base_image_name
builder.vars.image_suffix
builder.vars.source_path
builder.vars.provider
These come from builder_var_files or from builder_vars set in the builder.
The builder can specify builder_var_files:
builder_var_files:
- default_linux
- vagrant_virtualbox
Which refer to YAML files in the ./template/vars directory.
Variables redefined in multiple builder_var_files will be overwritten by each subsequent file as it is loaded.
builder_vars override any variables specified in builder_var_files.
Variables are merged deeply, with sub-dictionaries also merged recursively.
Note that in the builder template variables are namespaced under builder.vars -
this avoids conflicts in templating where multiple builders specify the same variables.
In the builder_var_files and builder_vars these variables do not have a namespace prefix.
A complete variable file for the vagrant builder might look like this:
base_image_name: vagrant-builder-worker
image_suffix: docs-edition
source_path: ubuntu/bionic64
provider: virtualbox
Alternatively, those variables could have been specified in the builder's YAML file
as a YAML map under the key builder_vars, which also serves to override variables
that have already been set by one of the builder_var_files specified.
Like scripts, variables allow reuse.
A common tactic is to provide a defaults file as well as purpose-specific files that override some of the defaults.
For example default_linux might provide some default variables for a linux environment, with aws_bionic overriding some of those for Ubuntu Bionic on AWS.
A few variables get special treatment.
Environment variables in builder.vars.env_vars will be set for all scripts.
These are specified as a dictionary in the builder vars, and automatically converted to NAME=value format for Packer.
The required variable builder.vars.execute_command sets the Packer execute_command configuration.
Similarly, builder.vars.ssh_timeout sets the Packer start_retry_timeout configuration.
Monopacker runs scripts in the order you specify.
When you specify script_directories, all scripts in each of those directories are
executed first by order of directories in the list, then by lexicographic order
of script names in each directory.
For example, if your builder specifies:
script_directories:
- my_scripts
- my_other_scripts
# ls -1 my_scripts
01-foo.sh
02-bar.sh
# ls -1 my_other_scripts
0001-qux.sh
- All scripts in
my_scriptswill be executed before any scripts inmy_other_scripts. - Scripts
my_scriptswill be executed in lexicographic order, so01-foo.shwill be executed before02-bar.sh.
Secrets are handled in code shared between all builders and providers.
A run of monopacker build takes a --secrets_file=.. option pointing to a file containing serets, with the format
---
- name: cookie-recipe
path: /etc/bakery/recipes/cookie.txt
value: |
1 part sugar
...Each item in the file has a name (optional), a path to which it should be written on the built instances, and the value of the secret.
Secrets are internally written to a tarfile which is transferred to the instance and untarred there.
Adding a builder is as simple as adding a YAML file, specifying a template,
and making sure that the template's variables are all accounted for by a combination of
builder_var_files and builder_vars.
In the unlikely scenario that you want to add an entirely new builder template
simply create a .jinja2 file with the name of your choice under ./template/builders.
Ensure that your builder template has a name key set to {{builder.vars.name}}, as this is how monopacker templating
maps builders to provisioners in the Packer template.
Monopacker has support for handling restarts during the build process.
If a script's name includes 'reboot', Monopacker will add a pause after the step to ensure that the host is back up before continuing.
Monopacker has support for generating SBOMs.
SBOM generation is enabled by adding a variable file to your builder that sets monopacker_generate_sbom to true. Here's an example variable file:
---
monopacker_generate_sbom: true
# defaults to ""
monopacker_sbom_command_args: "-b $MONOPACKER_BUILDER_NAME -c $MONOPACKER_GIT_SHA"
# default vaule is "monopacker_ubuntu_sbom.py"
# monoopacker_sbom_script: monopacker_ubuntu_sbom.py
You might be using {{foo}} syntax to reference a variable that does not exist (in this case, foo).
If you're trying to specify variable that Packer will supply, make sure your value is wrapped in quotes.
A number of things could be going wrong here.
Ensure that the builder template properly
references all variables as being namespaced under builder.vars and that your builder_var_files
and builder_vars do not have any namespacing. See above for a more thorough description.
Variables are set only in Packer's context.
Environment variables are set when scripts run on the target host.