cf-health-check.md

CFEngine Health Check

Purpose: Provide CFEngineers with a compact checklist for assessing CFEngine health.

Bibliography This document is based on the following best-practice guides:

• CFEngine 3 Best Practices (Archive)
• CFEngine Enterprise Best Practices
• Evolve Thinking's Best Practices

Contributors:

• Aleksey Tsalolikhin
• Nick Anderson
• Neil Watson
• Joe Moore

Writing Policy

Thinking CFEngine

• Do not think procedurally, instead declare your intentions as CFEngine promises. http://evolvethinking.com/cfengine-best-practices-part-2/
• Less is more, leave decisions on how to do something to CFEngine. http://evolvethinking.com/cfengine-best-practices-part-2/
• Focus on the end goal, not the procedure. http://evolvethinking.com/cfengine-best-practices-part-2/
• Don't try to control the order of execution. Always keep coding to a minimum. Embrace normal ordering.

Style

• Follow a consistent policy style.
• Keep your policy in bundles and files to manageable amounts, making it easier to understand. Don't group together things that should be separate. Arranging files. How to decide when to make a bundle.
• Use meaningful names for your files and bundles. Make it obvious what they are about. How to choose and name bundles.
• Use naming conventions for bundles, handles, and classes. http://evolvethinking.com/cfengine-best-practices-part-2/

Make and Use Libraries

• Don't duplicate code when you could use parameterized bundles. When to use a parameterized bundle or method. Build reusable bundles.
• Don't reinvent the wheel. CFEngine Standard Library Use frameworks
• Keep a site library instead of modifying CFEngine Standard Library.

Document

• Always document the intention of your promises using "comment" attribute.
• Use "promisees" to document who or what will be impacted by your promises.
• Use "meta" promises and attributes to document metadata such as who wrote the code, when, etc. (e.g. meta => { "author=John Smith", "version="2.0" }; )

Classes Promises

• Classify your system before making changes to it. (Put class bundles first in bundlesequence.)
• Put classes in common bundles when you need to use them in multiple bundles. When should classes be in common bundles?

Variables Promises

• Define your variables in the current bundle or in a common generic global bundle as much as possible. When should variables be in common bundles? When should variables be in local bundles?
• Use variables as pointers to paths and servers, rather than coding them directly into promises.
• Always use lists to make the same promise about multiple objects.

Commands Promises

• Don't embed shell commands in policy when you could use native CFEngine code. Never embed simple shell commands. Avoid writing custom scripts Use shell commands sparingly. Beware the shell environment. Don't use command promises to cheat.
• Always use system variables to get information about system resources, rather than running external commands.

Files Promises

• Use separate promises for file permissions and content. http://evolvethinking.com/cfengine-best-practices-part-2/
• Avoid promises that are too broad. http://evolvethinking.com/cfengine-best-practices-part-2/
• Promise whole files and not just a portion of a file. http://evolvethinking.com/cfengine-best-practices-part-2/
• Try to combine tests and operations during file searches

Collaborating with others

• Use branch and merge workflows. See also GitHub branch and merge howto.
• Delegate responsibility if appropriate in your organization. Vet and agglomerate policy from different sources.

Handling secrets

• Don't put secrets in your policy. Use cf-keycrypt, a password vault or GPG to secure your secrets. http://cfengine.com/wp-content/uploads/2015/02/cfgcamp-2015.pdf
• If you put secrets in your policy, you can limit selected inputs to agents, but do so with caution. http://evolvethinking.com/cfengine-best-practices-deployment-upgrades-and-scaling/

Smart system administration

• Use "ifelapsed" to reduce frequency of housekeeping tasks (such as updating databases or performing business tasks) to reduce agent load. (https://auth.cfengine.com/manuals/cf3-bestpractice#Batch-Jobs)
• Collect garbage on your systems (e.g. remove old log files)
• Manage name service (/etc/resolv.conf)
• Ensure your services stay up and running, or to take down services that should not be running.
• Use packages (rather than tarballs or building from source) to install software.
• Avoid running CFEngine without lock protection
• Don't maintain cron jobs - use CFEngine's time classes. Never manage more than one cron job

Misc.

• Use version control, everywhere. http://evolvethinking.com/cfengine-best-practices-deployment-upgrades-and-scaling/
• Start with simple prototypes.
• Separate data from policy. http://evolvethinking.com/cfengine-best-practices-part-2/
• Make policy reliable even when the server is unavailable. http://evolvethinking.com/cfengine-best-practices-deployment-upgrades-and-scaling/
• Don't mess with update.cf or failsafe.cf http://evolvethinking.com/cfengine-best-practices-part-2/

Quality Control / Testing

• Use editor plugins to provide syntax highlighting to catch errors early.
• Use a pre-commit hook to catch errors early. Or use automated testing of policy (Jenkins, etc.) before distributing it. Check syntax with cf-promises
• Try to test formally, including unit tests.
• Test on multiple architectures. Use reporting for scale.

Making Changes to Production

Preparation

• Label new policy items uniquely for tracking.
• Consider a "default_repository" in case you have to examine history of changes to files managed by CFEngine.
• Test prior to releasing to production environment.
• Have a policy and schedule concerning major changes.
• Automate policy distribution by setting up your policy hub to update "masterfiles" from Version Control.
• Think through your changes

Execution

• Never change system policy when humans are absent
• Try to make many small changes rather than one large change to reduce risk.
• Test in the production environment on a small number of machines whenever possible.
• Deploy changes in progressive waves to decrease risk.

Hub health

• Run your CFEngine Postgres database on a dedicated SSD for best performance.
• Set "splaytime" to reduce load on your hub.
• Monitor Hub server utilization to make sure it's within normal parameters with room to spare.
• Have more than one hub in case your hub dies or has hardware issues, of the same grade of hardware so it can handle full production load.
• Make CFEngine policy servers in redundant pairs. http://evolvethinking.com/cfengine-best-practices-deployment-upgrades-and-scaling/
• Clean up your hub to remove entries for nodes that have been decommissioned (to improve hub performance and increase readability for humans).
• Try to stay up to date on your CFEngine software version as the software is continously improved.
• Don't install CFEngine RPM onto an image and then "bake" and deploy the image; same key on nodes = management headaches and hub performance issues.

Upgrades

• CFEngine 2 and 3 can run in parallel for gradual migration. http://evolvethinking.com/cfengine-best-practices-deployment-upgrades-and-scaling/
• Upgrade policy and CFEngine 3 with extensive planning and testing. http://evolvethinking.com/cfengine-best-practices-deployment-upgrades-and-scaling/

Using CFEngine

• Use cf-monitord to detect and investigate anomalies to understand and increase system stability.
• Always monitor promise compliance and investigate non-compliances.
• Try to monitor promise repairs to increase system stability.
• Have a site security policy. Use CFEngine to implement hardening measures, and to monitor important assets.

TODO

When this document settles down, open a pull request to https://github.com/cfengine/documentation/ in https://github.com/cfengine/documentation/tree/master/guide/writing-and-serving-policy