CFEngine contains a powerful language for controlling all aspects of a system. CFEngine runs primarily on UNIX and UNIX-like operating systems, but can also run on Windows.
Goal for this presentation: Learn 20% of CFEngine enabling 80% of your work.
- Mature
- Small
- Fast
- Flexible
- Portable
- Vertical Sysadmin’s CFEngine courses
- CFEngine Reference Manual
- CFEngine Training & Professional Services (Email)
- Beyond Automation with CFEngine3 (Video Training)
This presentation was made with love using org-mode 9.2.4, cfengine 3.12.2, and reveal.js 3.8.0. You can get a copy of this presentation any time on Github.
http://github.com/nickanderson/CFEngine-zero-to-hero-primer
These are the major components of CFEngine that you will encounter on a day to day basis.
cf-agentcf-execdcf-serverdcf-monitord
cf-agent is the command you will use most often. It is used to run
policy(code) and ensure your system is in the desired state. If you are running
any CFEngine command from the command line, there’s a greater than 99% chance
that this is it.
cf-execd is a periodic task scheduler. You can think of it like cron with an
understanding of CFEngine classes.
By default CFEngine runs and enforces policies every five minutes. cf-execd
is responsible for making that happen.
cf-serverd runs on the CFEngine server, as well as all clients.
- On servers it is responsible for serving files to clients.
- On clients it accepts
cf-runagentrequests - In Enterprise it serves reports to queries from
cf-hub
cf-runagent allows you to request ad-hoc policy runs. I rarely use it.
cf-monitord monitors various statistics about the running system. This
information is made available in the form of classes and variables.
You’ll almost never use cf-monitord directly. However the data provided by
cf-monitord is available to cf-agent.
It is very likely that you have only ever used imperative languages. Examples of imperative languages include C, Perl, Ruby, Python, shell scripting, etc. Name a language. It’s probably imperative.
CFEngine is a declarative language. The CFEngine language is merely a description of the final state. CFEngine uses convergence to arrive at the described state.
bundle agent main | #!/bin/env/bash
{ |
packages: | rpm -q openssh-server || yum install openssh-server
| yum check-update openssh-server
"openssh-server" | if [ $? -eq 100 ]; then # update available
policy => "present", | yum upgrade openssh-server
version => "latest"; | fi
}
Imperative languages execute step by step in sequence.
- Goes in order from start to finish.
- If interrupted the state is inconsistent.
- Subsequent executions typically repeat tasks already done.
- Potentially causing damage.
For Example:
- Script appends line to file and restarts daemon.
- Second execution appends duplicate line and restarts daemon.
- Daemon doesn’t accept duplicate lines and service refuses to run.
Imperative starts at known state A and transforms to known state B.
It is not a list of steps to achieve an outcome but a description of the desired state. Because of this any deviation from the desired state can be detected and corrected.
In other words, a declarative system can begin in any state, not simply a known state, and transform into the desired state.
Declarative states a list of things which must be true. It does not state how to make them true.
When a system has reached the desired state it is said to have reached convergence.
Promise theory is the fundamental underlying philosophy that drives CFEngine.
It is a model of voluntary cooperation between individual, autonomous actors or agents who publish their intentions to one another in the form of promises.
A file (e.g., /etc/apache2/httpd.conf) can make promises about its own
contents, attributes, etc. But it does not make any promises about a process.
A process (e.g., httpd) can make a promise that it will be running. But it
does not make any promises about its configuration.
The configuration file and the process are autonomous. Each makes promises about itself which cooperate toward an end.
- Promise Theory (animated video series)
- Basic concepts (part 1)
- The rules of delegation (part 2)
- Scaling cooperation (part 3)
- Scaling goals (part 4)
bundle bundle_type bundle_name
{
type:
context::
"promiser" -> { "promisee", "stakeholder" }
----------|
attribute1 => "value", |
attribute2 => body, |-- Promise Body
attributeN => bundle; |
} ----------|
bundle bundle_type bundle_name
{
type:
context::
"promiser" -> { "promisee", "stakeholder" }
----------|
attribute1 => "value", |
attribute2 => body, |-- Promise Body
attributeN => bundle; |
} ----------|
bundle_type- The type of bundle the promise resides in. Promises must be
inside of a bundle. Different bundle types are handled by
different agents (
cf-agent,cf-serverd,cf-monitord).
bundle bundle_type bundle_name
{
type:
context::
"promiser" -> { "promisee", "stakeholder" }
----------|
attribute1 => "value", |
attribute2 => body, |-- Promise Body
attributeN => bundle; |
} ----------|
bundle_name- The name of the bundle the promise resides in.
bundle bundle_type bundle_name
{
type:
context::
"promiser" -> { "promisee", "stakeholder" }
----------|
attribute1 => "value", |
attribute2 => body, |-- Promise Body
attributeN => bundle; |
} ----------|
type:- Marks the start of a section of promises. The kind of promise being
made (e.g.,
files,commands,packages, etc …).
bundle bundle_type bundle_name
{
type:
context::
"promiser" -> { "promisee", "stakeholder" }
----------|
attribute1 => "value", |
attribute2 => body, |-- Promise Body
attributeN => bundle; |
} ----------|
context::- Optional, defaults to
any::(no restriction, run on any host). The context restriction applies until the next context/class expression or until it’s reset to default at the start of the next promise type.
bundle bundle_type bundle_name
{
type:
context::
"promiser" -> { "promisee", "stakeholder" }
----------|
attribute1 => "value", |
attribute2 => body, |-- Promise Body
attributeN => bundle; |
} ----------|
promiser- What is making the promise. (e.g., a file).
bundle bundle_type bundle_name
{
type:
context::
"promiser" -> { "promisee", "stakeholder" }
----------|
attribute1 => "value", |
attribute2 => body, |-- Promise Body
attributeN => bundle; |
} ----------|
promisee/stakeholder- An optional recipient or beneficiary of the promise (who cares about the promise). Promisees provide documentation for cross referencing, primarily for humans.
bundle bundle_type bundle_name
{
type:
context::
"promiser" -> { "promisee", "stakeholder" }
----------|
attribute1 => "value", |
attribute2 => body, |-- Promise Body
attributeN => bundle; |
} ----------|
promise body- A collection of promise attributes (not to be confused with a body used as an attribute value)
bundle bundle_type bundle_name
{
type:
context::
"promiser" -> { "promisee", "stakeholder" }
----------|
attribute1 => "value", |
attribute2 => body, |-- Promise Body
attributeN => bundle; |
} ----------|
value- A value to be used by a promise attribute. Attributes that take values (in contrast to bodies and bundles) are not complex and have a limited range of input. Note, promise attributes that do not take bundles or bodies must be quoted.
bundle bundle_type bundle_name
{
type:
context::
"promiser" -> { "promisee", "stakeholder" }
----------|
attribute1 => "value", |
attribute2 => body, |-- Promise Body
attributeN => bundle; |
} ----------|
body- A collection of attribute values. Note, promise attributes that take bodies must not be quoted.
bundle bundle_type bundle_name
{
type:
context::
"promiser" -> { "promisee", "stakeholder" }
----------|
attribute1 => "value", |
attribute2 => body, |-- Promise Body
attributeN => bundle; |
} ----------|
bundle- A collection of promises. Complex promise attributes like
edit_linetake bundles. Note, promise attributes that take bundles must not be quoted;
- Separated by commas
- Vary by promise type
- Value is quoted string or unquoted object (function/body/bundle)
bundle agent main {
files:
linux::
"/tmp/example" -> { "Instructor", "Students" }
create => "true",
touch => "true",
action => warn_only;
}
# cf-agent --no-lock --file ./examples/example_promise.cf warning: Warning promised, need to create file '/tmp/example'
- collection of promises
- logical grouping
- can have parameters
- ARE NOT FUNCTIONS
bundle type name
{
type:
context::
"promiser" -> { "promisee" }
attribute1 => "value",
attribute2 => value;
type:
context::
"promiser" -> { "promisee" }
attribute1 => "value",
attribute2 => value;
}
Bundles apply to the binary that executes them. E.g., agent bundles apply to
cf-agent while server bundles apply to cf-serverd.
Bundles of type common apply to any CFEngine binary.
- ensure the
apache2package is installed - ensure the content in the config file is correct
- ensure content is present for serving
- ensure proper permissions of files
- ensure the
httpdprocess is running - ensure the
httpdprocess is restarted when the configuration changes
This matters when you call the same bundle more than one time within a given execution.
- Classic arrays are cleared at the beginning of a bundle actuation.
- Lists, strings, ints, reals, and data-containers are preserved but can be
re-defined if not guarded with
if => isvariable(). - bundle scoped classes are cleared at the end of the bundles execution
- namespace scoped classes are not cleared automatically, though they can be explicitly undefined.
bundle common globals
{
vars:
"tool_path" string => "/srv/tools"
}
bundle server my_access_rules
{
access:
"$(globals.tool_path)"
admit_ips => { "192.168.0.0/24" };
}
bundle agent my_policy
{
vars:
"config[PermitRootLogin]" string => "no";
"config[Port]" string => "22";
files:
"/etc/ssh/sshd_config"
edit_line => set_line_based( "my_policy.config", " ", "\s+", ".*", "\s*#\s*");
}
bundle monitor measure_cf_serverd
{
vars:
"pid[cf-serverd]"
string => readfile( "$(sys.piddir)/cf-serverd.pid", 4k );
"reg_stat[rss]" string =>"(?:[^\s+]*\s+){23}([^\s]+)(?:.*)";
measurements:
"/proc/$(pid[cf-serverd])/stat"
handle => "cf_serverd_vsize",
stream_type => "file",
data_type => "int",
history_type => "weekly",
units => "pages in memory",
match_value => line_match_value(".*", $(reg_stat[rss]) );
}
I stated before that the attributes of a promise, collectively, are called the body. Depending on the specific attribute the value of an attribute can be an external body.
A body is a collection of attributes. These are attributes that supplement the promise.
body TYPE NAME(OPTIONAL, PARAMS)
{
ATTRIBUTE => "value";
ATTRIBUTEn => { "more", "values" };
}
The difference between a bundle and a body is that a bundle contains promises while a body contains only attributes.
- A bundle is a collection of promises.
- A body is a collection of attributes that are applied to a promise.
The distinction is subtle, especially at first and many people are tripped up by this.
In a body, each attribute ends with a semicolon.
(Note that in a bundle each promise ends with a semicolon, while attributes of each promise are separated by commas)
bundle agent main {
files:
"/tmp/file"
perms => m(600);
}
PRO TIP: The =cf-locate= script in core/contrib can help you find and view body and bundle definitions within your policy set.
cf-locate --plain --full "perms m\(.*"body perms m(mode)
# @brief Set the file mode
# @param mode The new mode
{
mode => "$(mode)";
}
Bundles and bodies can be paramaterized for abstraction and re-usability. In other words you can define one and call it even passing in parameters which will implicitly become variables.
body type name (my_param) {
attribute1 => "$(my_param)";
}
The parameter my_param is accessed as a variable by $(my_param).
The Masterfiles Policy Framework is the default policy that ships with CFEngine. The standard library is included.
The CFEngine Standard Library comes bundled with CFEngine in the
masterfiles/lib/ directory.
The standard library contains ready to use bundles and bodies that you can include in your promises and is growing with every version of CFEngine. Get to know the standard library well, it will save you much time.
$ cf-agent --no-locks --log-level=info --file ./test.cf --bundle bundlename
Note: Make sure you use the correct file and bundle name! For any examples including a bundle named main or __main__ you can skip specifying the bundle.
By default promises (excluding defaults, vars, classes, and methods) are locked for 1 minute once they are evaluated.
for i in $(seq 3); do
cf-agent -f ./examples/reports_show_locking.cf
sleep 30
doneR: I started running CFEngine 3.14.0a.4e12fcf75 at 'Fri Jul 19 11:23:01 2019' R: Hello World! R: I started running CFEngine 3.14.0a.4e12fcf75 at 'Fri Jul 19 11:23:31 2019' R: I started running CFEngine 3.14.0a.4e12fcf75 at 'Fri Jul 19 11:24:01 2019' R: Hello World!
bundle agent main
{
reports:
"I started running CFEngine $(sys.cf_version) at '$(sys.date)'"
action => immediate;
"Hello World!";
}
body action immediate
# @brief Evaluate the promise at every `cf-agent` execution.
{
ifelapsed => "0";
}
bundle agent main
{
commands:
"/bin/echo Hello World!";
}
# cf-agent --no-lock --file ./examples/commands_echo_hello_world.cf notice: Q: ".../bin/echo Hello": Hello World!
bundle agent main {
files:
"/etc/shadow" perms => perms_for_shadow_files;
"/etc/gshadow" perms => perms_for_shadow_files;
reports:
"Please run this policy as root"
if => not( strcmp( "$(sys.user_data[gid])", "0" ) );
}
body perms perms_for_shadow_files {
owners => { "root" };
groups => { "root" };
mode => "0640";
}
- This is an agent bundle (meaning that it is processed by
cf-agent). - Its purpose is to set the permissions on
/etc/shadowand/etc/gshadow. - It uses an external body named
perms_for_shadow_files. - The body only needs to be defined once and can be reused for any number of promises.
Note: The values for owners and groups is enclosed in curly braces. This is
because these attributes take a list of strings (aka, an slist).
bundle agent example {
files:
"/etc/motd" copy_from => cp("/repo/motd");
}
body copy_from cp (from) {
servers => { "$(sys.policy_hub)" };
source => "$(from)";
compare => "digest";
}
bundle server my_access_rules
{
access:
policy_server|am_policy_hub::
"/repo"
admit_ips => { "192.168.0.1/24" },
admit_keys => { "SHA=12345" };
}
bundle agent example {
files:
"/etc/motd" copy_from => cp("/repo/motd");
}
- The file
/etc/motdshould be a copy of a file described by thecpcopy_frombody.
body copy_from cp (from) {
servers => { "$(sys.policy_hub)" };
source => "$(from)";
compare => "digest";
}
source- The path to the file that should be copied.
servers- Servers which the file should be attempted to be copied from.
compare- How to determine if the file differs and requires update.
bundle server my_access_rules
{
access:
policy_server|am_policy_hub::
"/repo"
admit_ips => { "192.168.0.1/24" },
admit_keys => { "SHA=12345" };
}
admit_ips- List of IPs or subnets that should be allowed to copy from
/repo. admi_keys- List of cfengine ids that should be allowed to copy from
/repo.
bundle agent main {
files:
"/etc/ssh/sshd_config" edit_line => deny_root_ssh;
}
bundle edit_line deny_root_ssh {
delete_lines:
"^PermitRootLogin.*";
insert_lines:
"PermitRootLogin no";
}
Can be one of several types:
- strings
- lists
- numbers (int/real)
- data (JSON/YAML/CSV/ENV)
Reference: Special Variables, Language Concepts -> Variables, Promise Types and Attributes -> vars
CFEngine doesn’t have for loops, but it implicitly iterates over lists and data structure values.
bundle agent main
{
vars:
"l" slist => { "two", "one", "three" };
"d" data => '[ "three", "one", "two"]';
"d2" data => '{ "one":"1", "two":"2", "three":"3"}';
reports:
"l contains $(l)";
"d contains $(d)";
"d2 contains $(d2)";
}
# cf-agent --no-lock --file ./examples/list-iteration.cf R: l contains two R: l contains one R: l contains three R: d contains three R: d contains one R: d contains two R: d2 contains 1 R: d2 contains 2 R: d2 contains 3
bundle agent main
{
vars:
"d" data => '{ "key": { "subkey": "value" } }';
"a[key][subkey]" string => "value";
reports:
"$(const.dollar)(d[key][subkey]) == $(d[key][subkey])";
"$(const.dollar)(a[key][subkey]) == $(a[key][subkey])";
"d contains$(const.n)$(with)" with => string_mustache( "{{%-top-}}", d );
"a contains$(const.n)$(with)" with => string_mustache( "{{%-top-}}", a );
}
A class is like a tag (like tagging a photo). Classes are used to give a promise context. Valid characters in classes are [A-Za-z0-9_] (alphanumeric and underscores). There are two types of classes.
- Built in classes. These hard classes are classes that CFEngine
will create automatically. Hard classes are determined based on the system
attributes. For example a server running Linux will have the class
linux. - User defined classes. These soft classes are classes that are defined by you. You can create them based on the outcome of a promise, based on the existence of other classes, or for no reason.
Use cf-promsies --show-classes to see the first order of resolved classes.
cf-promises --show-classes | headClass name Meta tags 127_0_0_1 inventory,attribute_name=none,source=agent,hardclass 172_17_0_1 inventory,attribute_name=none,source=agent,hardclass 172_27_224_133 inventory,attribute_name=none,source=agent,hardclass 192_168_122_1 inventory,attribute_name=none,source=agent,hardclass 192_168_42_189 inventory,attribute_name=none,source=agent,hardclass 4_cpus source=agent,derived-from=sys.cpus,hardclass 64_bit source=agent,hardclass Afternoon time_based,cfengine_internal_time_based_autoremove,source=agent,hardclass Day19 time_based,cfengine_internal_time_based_autoremove,source=agent,hardclass
bundle agent apache_config {
files:
debian::
"/etc/apache2/apache2.conf"
copy_from => remote_cp("/cfengine/repo/debian/apache2.conf","$(sys.policy_hub)");
redhat::
"/etc/httpd/conf/httpd.conf"
copy_from => remote_cp("/cfengine/repo/redhat/httpd.conf","$(sys.policy_hub)");
solaris::
"/etc/apache2/2.2/httpd.conf"
copy_from => remote_cp("/cfengine/repo/solaris/httpd.conf","$(sys.policy_hub)");
}
- Copy the appropriate config file for the given platform
- Promises outside of the specified context are skipped
bundle agent example {
files:
solaris::
"/tmp/hello/world"
create => "true";
"/tmp/foo/bar"
create => "true";
linux::
"/dev/shm/hello_world"
create => "true";
commands:
"/bin/echo Hello World";
}
- New class expression sets context for following promises
- New promise type resets context to
any(implicit default)
bundle agent main
{
reports:
redhat:: # <- This context has no promises.
64_bit:: # <- This context has one promise. (not additive)
"I am $(sys.flavor) running on $(sys.arch)";
}
- No promises are defined in the
redhatcontext - One promise is defined in the
64_bitcontext - Nesting class expressions does not make them additive
bundle agent apache_config {
commands:
apache_config_repaired::
"/usr/sbin/apache2ctl graceful";
files:
"/etc/apache2/apache2.conf"
copy_from => remote_cp("/cfengine/repo/debian/apache2.conf",
$(sys.policy_hub)),
classes => results("bundle", "apache_config");
}
- Only when the apache config is updated define
apache_config_repaired. - Only when
apache_config_repairedis defined execute the command to restart the service.
commands:
apache_config_repaired.debian::
"/usr/sbin/apache2ctl graceful";
apache_config_reparied.redhat::
"/usr/sbin/apachectl graceful";
| Operator | Meaning | Example |
|---|---|---|
. and & | boolean and | debian.Tuesday:: |
ǀ and ǀǀ | boolean or | TuesdayǀWednesday:: |
! | boolean not | !Monday:: |
( ) | Explicit grouping | (debianǀredhat).!ubuntu.!centos:: |
Since 3.7.0 CFEngine is able to dereference variables directly within class
expressions. Note that quotes surrounding the entire expression ending before
the :: are required.
bundle agent main
{
vars:
"variable_containing_class" string => "cfengine";
reports:
"$(variable_containing_class)"::
"'$(variable_containing_class)' is defined";
"!$(variable_containing_class)"::
"'$(variable_containing_class)' is NOT defined";
}
# cf-agent --no-lock --file ./examples/example_variable_class_expressions.cf R: 'cfengine' is defined
I said that only Debian systems will run debian:: and only Red Hat will run
redhat::. This isn’t exactly true.
- Ubuntu is based on Debian, and so will have both
ubuntuanddebiandefined as hard classes. - Likewise, CentOS is based on Red Hat and so will have both
centosandredhatdefined as hard classes. - MPF defines
redat_pureanddebian_pure.
- Very early definition
- Loaded if
def.jsonis found next to policy entry - Classes based on system discovery (platform/networks/arch)
- Variables defined in
defbundle scope
- Define
supported_platformif the classubuntu_14,ubuntu_16, orubuntu_17is defined. - Define
by_hostnameif the classnickanderson_thinkpad_w550sis defined.
{
"classes": {
"supported_platform": [ "ubuntu_\\d+" ],
"by_hostname": [ "nickanderson_thinkpad_w550s" ]
},
"vars": {
"myvar1": "defined from augments",
"myvar2": "defined from augments"
}
}bundle agent main
{
reports:
"I defined '$(const.dollar)(def.myvar1)' as '$(def.myvar1)'";
supported_platform::
"This is a supported platform";
by_hostname::
"You can define classes from augments based on defined hostname";
}
cf-agent --no-lock --file ./examples/augments/augments.cfR: I defined '$(def.myvar1)' as 'defined from augments' R: This is a supported platform R: You can define classes from augments based on defined hostname
bundle common def
{
vars:
"myvar1" string => "Defined in policy";
"myvar2" string => "Defined in policy", if => not( isvariable( myvar2 ) );
}
bundle agent main
{
reports:
"I defined '$(const.dollar)(def.myvar1)' as '$(def.myvar1)'";
"I defined '$(const.dollar)(def.myvar2)' as '$(def.myvar2)'";
supported_platform::
"This is a supported platform";
by_hostname::
"You can define classes from augments based on defined hostname";
}
cf-agent --no-lock --file ./examples/augments/augments-policy-wins.cfR: I defined '$(def.myvar1)' as 'Defined in policy' R: I defined '$(def.myvar2)' as 'defined from augments' R: This is a supported platform R: You can define classes from augments based on defined hostname
Merge more specific augments (based on sys vars) on top.
{
"vars": {
"myvar1": "defined from augments for all",
"myvar2": "defined from augments for all"
},
"augments": [ "$(sys.policy_entry_dirname)/$(sys.os).json" ]
}{
"vars": {
"myvar2": "override for linux hosts"
}
}bundle agent main
{
reports:
"'$(const.dollar)(def.myvar1)' is '$(def.myvar1)'";
"'$(const.dollar)(def.myvar2)' is '$(def.myvar2)'";
}
cf-agent --no-lock --file ./examples/augments-multiple/promises.cfR: '$(def.myvar1)' is 'defined from augments for all' R: '$(def.myvar2)' is 'override for linux hosts'
bundle agent apache {
processes:
".*apache2.*"
restart_class => "apache2_not_running";
commands:
apache2_not_running::
"/etc/init.d/apache2 start";
}
bundle agent stop_bluetooth {
processes:
"bluetoothd"
process_stop => "/etc/init.d/bluetooth stop";
}
This policy uses a processes promise to check the process table (with ps)
for the regular expression .*bluetoothd.*. If it is found the process_stop
command is executed.
bundle agent stop_bluetooth {
processes:
".*bluetoothd.*"
signals => { "term", "kill" };
}
This policy uses a processes promise to check the process table (with ps)
for the regular expression .*bluetoothd.*. Any matching process is sent the
term signal, then sent the kill signal.
bundle agent apache {
services:
"www"
service_policy => "start";
}
This uses the services promise type to ensure that Apache is always running.
The standard_services bundle implementation currently covers systemd,
chkconfig, the service command, svcadm and systemV init scripts. Proper
functionality relies on each installed service correctly implementing a service
check as appropriate for the init system in use.
bundle agent stop_bluetoothd {
services:
"bluetoothd"
service_policy => "stop";
}
This policy uses a services promise type to ensure that Bluetooth services are
not running. Again, this only works for services that are defined under
standard_services in the standard library and requires cfengine 3.4.0 or
higher.
The same restrictions about distros apply to stopping services promises.
Services promises are really an abstraction on bundles.
apt_get, pkgsrc, freebsd_ports, slackpkg, msiexec, yum, nimclient,
zypper, pkg
bundle agent install {
packages:
"zsh"
policy => "present",
package_module => yum,
version => "latest";
}
- The
policyofpresentwill make sure the package is installed on the system, while apolicyofabsentwill ensure a package is not installed. - The
package_moduleofyumis included in the Masterfiles Policy Framework. - The
versionoflatestmeans the installed version should be the latest available. Alternatively you can provide an explicit version.
alpinelinux, freebsd, opencsw, solaris_install, apt,
freebsd_portmaster, pacman, windows_feature, apt_get, generic, pip,
yum, apt_get_permissive, ips, rpm_filebased, yum_group,
apt_get_release, msi_explicit, rpm_version, yum_rpm, brew,
msi_implicit, smartos, yum_rpm_enable_repo, dpkg_version, npm,
smartos_pkg_add(repo), yum_rpm_permissive, emerge, npm_g, solaris,
zypper,
bundle agent install {
packages:
"zsh"
package_policy => "addupdate",
package_method => apt,
package_select => ">=,
package_version => "4.3.10-14";
}
- The
package_policyofaddupdatewill install or upgrade. Usingaddwill only install, never upgrade,upgradewill upgrade only anddeletewill uninstall. - The
package_methodofaptis in the standard library, look there for other package methods (e.g., rpm, ips, etc.). - The
package_selectof>=means the installed version must be equal to or newer than the specified version or it will be replaced. Using<=would downgrade, if thepackage_methodsupports downgrading and ==== will require the exact version.
- Full file management
- Partial file management
Hello from {{{vars.sys.fqhost}}}!
{{#classes.linux}}I am a Linux Box!{{/classes.linux}}
{{^classes.windows}}I am NOT a Windows Box{{/classes.windows}}
bundle agent main{
files:
"/tmp/example"
create => "true",
edit_template => "$(this.promise_dirname)/template.mustache",
template_method => "mustache";
}
-top-- Special key representing the complete data given to the templating engine.
@- Expands to the key that is currently iterating.
%- Variable prefix causing the data to be rendered as the multi-line JSON representation of the data given to the templating engine.
$- Variable prefix causing the data to be rendered as the serialized JSON representation of the data given to the templating engine.
packagesmatching() returns data. Render the multiline JSON representation of the data.
bundle agent main
{
vars:
"p" data => packagesmatching( "emacs.*", ".*", ".*", ".*");
"r" string => string_mustache( "{{%-top-}}", p ),
if => not(isvariable( r ) );
reports:
"$(r)";
}
- Render raw values with
{{{VAR}}}or{{& VAR}}. Mustache html escapes by default. - Use string_mustache() to render mustache into a string.
- template_data() Helps to separate CFEngine specifics from templates.
bundle agent tidy {
files:
"/var/log/.*"
file_select => days_old("7"),
delete => tidy;
}
This policy will delete any files in /var/log/ older than 7 days. The
days_old() and tidy bodies are included in the standard library,
To delete a file indiscriminately, omit the file_select.
Look up =file_select= and =tidy= in the reference-manual to find more ways to use this.
cat /var/cfengine/policy_server.datcf-promises --show-vars | grep sys.policy_hub
ps -ef | grep [c]f-
You should expect to find cf-execd, cf-serverd, and cf-monitord on all
hosts. Additional processes will be seen on Enterprise Hubs
ls -lh /var/cfengine/promise_summary.log
ls /var/cfengine/outputscat /var/cfengine/outputs/previous
cf-hub --hail <IP|HOSTNAME> --verbose --query rebasecf-hub -H <IP|HOSTNAME> -v -q delta
[root@hub ~]# cf-hub -H 10.10.10.11 -q rebase error: Abort transmission: got " Unspecified server refusal (see verbose server output)" from 10.10.10.11
- Usually indicates the host does not trust the hub.
- Is the host bootstrapped to the hub you expect?
- Run cf-serverd on the client with
--verboseand--no-forkto see why it’s refusing
- Firewall blocking inbound connections on port
5308 cf-serverdnot running on remote host
Before starting you need to have cfengine installed on the server and the client and the server FQDN must be set properly in DNS (or use the IP addresses). This is ideally handled by your provisioning process. Along with automating server function you should also be automating your provisioning process.
Some ways of automating provisioning are kickstart, preseed, fai, cobbler, disk imaging, instance cloning, etc, etc. This, of course, is not a complete list.
Edit /var/cfengine/masterfiles/def.cf to set the acl list for the IP
addresses of your network, then run:
cf-agent --bootstrap $(hostname --fqdn) cf-agent -KI
Simply run:
cf-agent --bootstrap server.fqdn.example.com
You can use the server’s IP address instead of the DNS name.
Policy is distributed from /var/cfengine/masterfiles on the server (also known as
the policy_hub) and are copied to /var/cfengine/inputs. All clients then
copy /var/cfengine/inputs from the server.
CFEngine logs to /var/cfengine/promise_summary.log. Here’s an example log message:
1463018982,1463018990: Outcome of version CFEngine Promises.cf 3.7.0 (agent-0):\ Promises observed - Total promise compliance: 93% kept, 3% repaired,\ 4% not kept (out of 148 events).\ User promise compliance: 93% kept, 2% repaired, 5% not kept (out of 130 events). CFEngine system compliance: 94% kept, 6% repaired, 0% not kept (out of 18 events).
Note: The timestamp is a Unix epoch.
CFEngine will also send an email to the configured address in body executor
control= any time there is output from an agent run that differed from the
previous run.
And finally you can use the -I flag to have CFEngine inform you of repairs.
(Shown here along with the -K flag which ignores any lock timers).
cf-agent -KI
Running the agent in verbose mode ( cf-agent --verbose | cf-agent -v )
provides all of the details about each promise and its result
bundle agent main
{
files:
"/tmp/example"
handle => "example_file_exists_and_contains_date",
create => "true",
edit_line => lines_present( $(sys.date) );
}
bundle edit_line lines_present(lines)
# @brief Ensure `lines` are present in the file. Lines that do not exist are appended to the file
# @param List or string that should be present in the file
#
# **Example:**
#
# ```cf3
# bundle agent example
# {
# vars:
# "nameservers" slist => { "8.8.8.8", "8.8.4.4" };
#
# files:
# "/etc/resolv.conf" edit_line => lines_present( @(nameservers) );
# "/etc/ssh/sshd_config" edit_line => lines_present( "PermitRootLogin no" );
# }
# ```
{
insert_lines:
"$(lines)"
comment => "Append lines if they don't exist";
}
In the verbose output as each promise is actuated a BEGIN promsie is emitted
with the promise handle or filename and line number position if it does not have
a handle. In the example output we can see that the promise for /tmp/example
was REPAIRED.
Promises can be configured to log their outcomes to a file with log_kept,
log_repaired, and log_failed attributes in an action body.
bundle agent main
{
commands:
"/bin/true"
action => log_my_repairs( '/tmp/repaired.log' );
reports:
"/tmp/repaired.log"
printfile => cat( $(this.promiser) );
}
body action log_my_repairs( file )
{
log_repaired => "$(file)";
log_string => "$(sys.date) REPAIRED $(this.promiser)";
}
CFEngine enterprise provides details logging without special configuration.
The changes reporting interface is the easiest way to what repairs the agent is making to your infrastructure.
Changes can also be queried from the changes rest api. Here we query for repairs made
by files type promises.
[root@hub ~]# curl https://hub/api/v2/changes/policy?promisetype=files
{
"data": [
{
"bundlename": "cfe_internal_update_policy",
"changetime": 1512427971,
"hostkey": "SHA=01fe75e93ca88bbd381eb720e9b43d0840ea8727aae8fc84391c297c42798f5c",
"hostname": "hub",
"logmessages": [
"Copying from 'localhost:/var/cfengine/masterfiles/cf_promises_release_id'"
],
"policyfile": "/var/cfengine/inputs/cfe_internal/update/update_policy.cf",
"promisees": [],
"promisehandle": "cfe_internal_update_policy_files_inputs_dir",
"promiser": "/var/cfengine/inputs",
"promisetype": "files",
"stackpath": "/default/cfe_internal_update_policy/files/'/var/cfengine/inputs'[1]"
},
{
"bundlename": "cfe_internal_setup_knowledge",
"changetime": 1512428912,
"hostkey": "SHA=01fe75e93ca88bbd381eb720e9b43d0840ea8727aae8fc84391c297c42798f5c",
"hostname": "hub",
"logmessages": [
"Owner of '/var/cfengine/httpd/htdocs/application/logs/./log-2017-12-04.log' was 0, setting to 497",
"Group of '/var/cfengine/httpd/htdocs/application/logs/./log-2017-12-04.log' was 0, setting to 497",
"Object '/var/cfengine/httpd/htdocs/application/logs/./log-2017-12-04.log' had permission 0644, changed it to 0640"
],
"policyfile": "/var/cfengine/inputs/cfe_internal/enterprise/CFE_knowledge.cf",
"promisees": [],
"promisehandle": "cfe_internal_setup_knowledge_files_doc_root_application_logs",
"promiser": "/var/cfengine/httpd/htdocs/application/logs/.",
"promisetype": "files",
"stackpath": "/default/cfe_internal_management/methods/'CFEngine_Internals'/default/cfe_internal_enterprise_main/methods/'hub'/default/cfe_internal_setup_knowledge/files/'/var/cfengine/httpd/htdocs/application/logs/.'[1]"
}
],
"total": 2,
"next": null,
"previous": null
}
See Also: query rest api
The custom reports interface and associated query rest api allow more flexible reports to be run.
Queries can be made against the promiselog table. This query finds the
promises that are repaired the most excluding internal cfengine related promises
and promises from the stdlib.
-- Find most frequently repaired promises excluding lib and cfe_internal directories
SELECT namespace,bundlename,promisetype,promisehandle, promiser, count(promiseoutcome)
AS count
FROM promiselog
WHERE promiseoutcome = 'REPAIRED'
AND policyfile
NOT ilike '%/lib/%'
AND policyfile
NOT ilike '%cfe_internal%'
GROUP BY namespace, bundlename, promisetype,promisehandle,promiser
ORDER BY count DESCReference: query api examples
WARNING: These logs are purged upon collection by the hub.
In Enterprise 3.7 each agent run logs to a CSV file named for the time the agent
started in $(sys.workdir)/state/promise_log/.
The fields are promise hash, policy file, release id, unknown (waiting on
developer feedback), namespace, bundle, promise type, stack path (call
tree), promise handle, promisees, log messages
719b756d3dc8fd7bdd20284c1fd894ae40bac55d8790855b074159db8fe187ae,/var/cfengine/inputs/cfe_internal/enterprise/CFE_hub_specific.cf,<unknown-release-id>,114,default,cfe_internal_update_folders,files,/var/cfengine/master_software_updates/windows_i686,/default/cfe_internal_management/methods/'CFEngine_Internals'/default/cfe_internal_enterprise_main/methods/'hub'/default/cfe_internal_update_folders/files/'/var/cfengine/master_software_updates/windows_i686'[40],cfe_internal_update_folders_files_create_dirs,"[""goal_updated""]","[""Created directory '/var/cfengine/master_software_updates/windows_i686/.'""]"
WARNING: These logs are purged upon collection by the hub.
Beginning with Enterprise 3.9 we began logging promise outcomes to a JSON format
in $(sys.statedir)/promise_log.jsonl.
Each promise outcome is logged along with the bundle name, promise handle, log messages near the promise actuation, the promise namespace, policy filename, promise hash, promise type, promisees, promiser, release id, stack path (call path), and the timestamp of the agent ran.
Here is an example of the output:
{
"execution": {
"bundle":"file_make_mustache",
"handle":"",
"log_messages":[
"Created file '/var/cfengine/httpd/conf/httpd.conf.staged', mode 0600",
"Updated rendering of '/var/cfengine/httpd/conf/httpd.conf.staged' from mustache template '/var/cfengine/inputs/cfe_internal/enterprise/templates/httpd.conf.mustache'"
],
"namespace":"default",
"policy_filename":"/var/cfengine/inputs/lib/files.cf",
"promise_hash":"ebc3dce615bcdb724e53a9761a24f2e7ed4f2e01aed1ce85dc217a9d3429fed7",
"promise_outcome":"REPAIRED",
"promise_type":"files",
"promisees":[
"CFEngine Enterprise",
"Mission Portal"],
"promiser":"/var/cfengine/httpd/conf/httpd.conf.staged",
"release_id":"<unknown-release-id>",
"stack_path":"/default/cfe_internal_management/methods/'CFEngine_Internals'/default/cfe_internal_enterprise_mission_portal/methods/'Apache Configuration'/default/cfe_internal_enterprise_mission_portal_apache/methods/'Stage Apache Config'/default/file_make_mustache/files/'/var/cfengine/httpd/conf/httpd.conf.staged'[0]"
},
"timestamp":1470326639
},
{
"execution":{
"bundle":"mission_portal_apache_from_stage",
"handle":"",
"log_messages":[
"Updated '/var/cfengine/httpd/conf/httpd.conf' from source '/var/cfengine/httpd/conf/httpd.conf.staged' on 'localhost'"
],
"namespace":"default",
"policy_filename":"/var/cfengine/inputs/cfe_internal/enterprise/mission_portal.cf",
"promise_hash":"d730f2911834395411e4f3168847fc6cc522955f97652de41e02c8bc15f3f761",
"promise_outcome":"REPAIRED",
"promise_type":"files",
"promisees":[
"CFEngine Enterprise",
"Mission Portal"
],
"promiser":"/var/cfengine/httpd/conf/httpd.conf",
"release_id":"<unknown-release-id>",
"stack_path":"/default/cfe_internal_management/methods/'CFEngine_Internals'/default/cfe_internal_enterprise_mission_portal/methods/'Apache Configuration'/default/cfe_internal_enterprise_mission_portal_apache/methods/'Manage Final Apache Config'/default/mission_portal_apache_from_stage/files/'/var/cfengine/httpd/conf/httpd.conf'[0]"
},
"timestamp":1470326639
}Inevitably, something will go wrong, and you will need to dig deep to figure something out. Lucky for you, I have some tips for debugging.
Again, using -K to disable locks is useful.
CFEngine’s verbose output can be fantastic for debugging. Use the -v flag to
turn it on.
cf-agent -Kv | grep -A 5 "BEGIN bundle"When viewing verbose output, look for BUNDLE <name> for the bundle that you
suspect is having trouble.
verbose: B: BEGIN bundle main verbose: B: ***************************************************************** verbose: P: ......................................................... verbose: P: BEGIN promise 'promise_promises_cf_4' of type "reports" (pass 1) verbose: P: Promiser/affected object: 'Hello World!' verbose: P: Part of bundle: main
CFEngine will tell you exactly what is going on with each promise, in excruciating detail.
verbose: Using literal pathtype for '/tmp/touch' verbose: No mode was set, choose plain file default 0600 info: Created file '/tmp/touch', mode 0600 verbose: Handling file existence constraints on '/tmp/touch' verbose: A: Promise REPAIRED verbose: P: END files promise (/tmp/touch...)
CFEngine supports comments as part of its data structure. Every promise can
have a comment attribute whose value is a quoted text string.
bundle agent example {
files:
"/etc/bind/named.cache"
copy_from => scp("$(def.files)/bind/named.cache"),
comment => "More recent copy of named.cache than shipped with bind";
}
Comments show up in the verbose output.
verbose: P: Container path : '/default/main/files/'/etc/bind/named.cache'[0]' verbose: P: verbose: P: Comment: More recent copy of named.cache than shipped with bind. verbose: P: .........................................................
The comment should always be why the promise is being made. Up until now none of the examples have used comments to save space on the slide. When writing your policies for real every promise should have a meaningful comment.
You’ll thank me when this saves the day.
When debugging, promise handles are also useful. Again, every promise can have
a handle attribute whose value is a quoted canonical string.
bundle agent example{
files:
"/etc/bind/named.cache"
copy_from => scp("$(def.files)/bind/named.cache"),
handle => "update_etc_bind_named_cache",
comment => "More recent copy of named.cache than shipped with bind";
}
CFEngine will tell you the handle of each promise in the verbose output.
verbose: P: BEGIN promise 'update_etc_bind_named_cache' of type "files" (pass 1) verbose: P: Promiser/affected object: '/etc/bind/named.cache' verbose: P: Part of bundle: example verbose: P: Base context class: any
By giving each promise a unique handle you can swiftly jump back and forth between your debug output and your policy file. When writing your policies for real every promise should have a unique handle.
You’ll thank me when this saves the day.
When debugging, promise stakeholders aka promisees are useful for understanding who cares about a given promise.
bundle agent example {
files:
"/etc/bind/named.cache" -> { "Operations", "Nick Anderson" }
copy_from => scp("$(def.files)/bind/named.cache"),
handle => "update_etc_bind_named_cache",
comment => "More recent copy of named.cache than shipped with bind";
}
CFEngine will tell you additional info about each promise.
verbose: Additional promise info: handle 'update_etc_bind_named_cache'\
source path './t.cf' at line 4 promisee {'Operations','Nick Anderson'}\
comment 'More recent copy of named.cache than shipped with bind.'
When debugging variables and classes promise meta data is useful to help identify variables and classes with specific attributes.
bundle agent main{
classes:
"my_class" expression => "any", meta => { "mytag" };
vars:
"my_var" string => "value", meta => { "mytag" };
"my_vars" slist => variablesmatching(".*", "mytag" );
"my_classes" slist => classesmatching(".*", "mytag" );
reports:
"My var: $(my_vars)";
"My class: $(my_classes)";
}
Note: Promise meta data is not currently displayed in the CFEngine’s verbose output.
Here’s a list of topics that I didn’t cover. This is to give you a taste of the rest of the power that is behind CFEngine. Dig deeper by checking them out in the reference manual.
vars:promises — Varables, strings, integers and reals (and lists of each).methods:promises — Create a self-contained bundle that can be called like a function.storage:promises — For local or remote (NFS) filesystems.edit_xml:promises - Promise by path, CFEngine does the XML for you.- Monitoring — Using data from
cf-monitord.
- Don’t edit the standard library. Create a
site_lib.cfand add your custom library bundles and bodies there. This helps with upgrading because you won’t have to patch your changes into the new version of the library. When you feel a bundle or body is ready for public use you can submit it to CFEngine by opening a pull request on Github. - Make built-in classes and user defined classes easy to distinguish by sight.
CFEngine creates hard classes
all_lower_case_separated_by_underscores. Whenever I define classes myself I useCamelCase. - Not sure how to organize =masterfiles=?
- A Case Study in CFEngine Layout by Brian Bennett.
- Example a10042
- Use =git= to revision control
masterfiles. - Syntax errors? Only read the very first error. Fix it, then try again. A missing character in one promise will throw the whole file off.
- Checkout the Augments file
- Checkout jq (because you can use it with mapdata() in 3.9+)
- Read the reference manual (all of it)
For example:
bundle agent main
{
classes:
"my-illegal-class";
reports:
"$(with)" with => join( " ", classesmatching( "my.illegal.class" ) );
}
R: my_illegal_class
The agent assumes you intended to canonify the string in the spirit of auto correction it canonifies it for you.
This courtesy is not extended when checking classes. You must explicitly canonify your string when using it in a class expression.
For example:
bundle agent main
{
vars:
"hostname" string => "$(sys.uqhost)";
reports:
any::
"$(hostname) contains invalid class characters";
"The class expression containing a nonvalid character is not a valid class expression";
"The agent silently skips the section";
"$(hostname)"::
"hello";
any::
"See that explicit canonification works";
"Hi"
if => canonify( $(hostname) );
}
R: nickanderson-thinkpad-w550s contains invalid class characters R: The class expression containing a nonvalid character is not a valid class expression R: The agent silently skips the section R: See that explicit canonification works R: Hi
For example:
bundle agent main
{
files:
"/tmp/dir/."
create => "true",
perms => m(600);
vars:
"mode" string => filestat( "/tmp/dir", permoct );
reports:
"/tmp/dir mode is $(with)" with => filestat( "/tmp/dir", permoct );
}
This is configurable behavior but by default if you promise a directory should be readable (list the files within the directory) the agent assumes that you also meant for it to be executable so that it can be entered and access the file and directories inside.
To disable the feature set rxdirs to false in the perms body you are
using.
For example:
bundle agent main
{
files:
"/tmp/dir/."
create => "true",
perms => my_m_norxdir(600);
vars:
"mode" string => filestat( "/tmp/dir", permoct );
reports:
"/tmp/dir mode is $(with)" with => filestat( "/tmp/dir", permoct );
}
body perms my_m_norxdir(mode)
{
rxdirs => "false";
inherit_from => m( $(mode) ); # body inheritance available since 3.8.0
}
How not to cfengine commands:
redhat.64_bit:: ” cd etc;if grep ‘10.135.130.11\|10.135.130.12\|10.135.128.11\|10.135.128.12’ /etc/resolv.conf; then /bin/sed -i ‘s/10.135.130.11/10.135.139.11;s/10.135.130.12/10.135.139.12/;s/10.135.128.11/10.135.139.11/;s/10.135.128.12/10.135.139.12/’ /etc/resolv.conf ; service network restart; fi” contain => in_shell;