Desired State / Configuration Management

[JimMadge]

Let's explore options for desired state configuration and configuration management.

Some things we could achieve,

• Separate VM state from IAC
  • Ease development burden
  • Use more appropriate tools for infrastructure and configuration
• Make the configuration of VMs more robust
• Periodic checks that VMs are correctly configured
• Automatically fix VMs whose state is not correct
• Idempotent configuration

Some tools/options we can explore,

• Ansible
• Ansible pull (clients fetch playbook from VCS rather than playbooks being pushed from a manager)
• Azure DSC (Linux)
• Azure Automanage Machine Configuration (using Powershell DSC 3 (preview))
• Chef
• Puppet
• Salt

See a previous issue.

[JimMadge]

Ansible

• Popular automation tool very often used for configuration management
• Actively developed, supported by RedHat
• Strong community with many community roles and collections
• Agentless, no client daemon
• Minimal client requirements
  • Linux: OpenSSH, Python (almost certainly installed by default)
  • Windows: Powershell 3.0, .NET 4.0, WinRM
• YAML syntax
• Common tasks can be collected into roles and tested with molecule

Use

Normally operates by a 'control node' pushing Python scripts to 'managed nodes' over SSH.
The scripts are then executed on the managed nodes.

This means,

• You should have access to a super user account on each managed node via SSH
• You need to be able to connect to every managed node

These could conflict with the DSH architecture.
A way around this could be to have a single control node inside the safe haven.

Alternatively, ansible-pull could be used to make each managed node periodically fetch and run a playbook.
This would mean every node would require Ansible, any roles and dependencies for the modules being used.

Other interfaces

• Ansible runner (CLI, Python module)
• AWX/Tower

[JimMadge]

DSH Architecture and DSC

The architecture of DSH, often deliberately designed to isolate groups and components, presents challenges to some models of DSC/configuration management.

In particular, any system which relies on (or assumes) each managed node is,

• reachable from a single control node or,
• able to reach a single authoritative source of configuration

may be tricky as connecting all (or most, or many) nodes to a single host/network could create a route to undermine isolation.

However, that is how many DSC/configuration management systems are intended to work.
We should think about the security implications, if we can reduce the risk and if they are offset by the extra confidence we would gain in the correct configuration of machines (and the ability to continually ensure their state is correct).

Options

A single (per SHM, or per SHM/SRE) management cluster

This could be

• AWX deployment
• VM to run Ansible from
• Chef/Puppet/Salt control node

NSG rules allowing necessary connections to all vnets with machines that need to be managed.
In the case of Ansible only outbound connections should be needed.
For tools where managed nodes 'phone home' like Chef/Puppet both inbound and outbound may be needed.

Compromising this infrastructure could be quite serious as it would allow privilege escalation (although perhaps indirectly) on managed VMs.

[JimMadge]

Azure Options

Currently is seems that the DSC extension for Linux is being retired (and is incompatible with the log analytics extension!).
It is to be replaced with the Azure Automanage Machine Configuration extension.

Machine Configuration uses Azure Policies to enforce Configuraton resources to sets of machines.

Maybe the structure is this?

• Create a Configuration resource
• Create a policy to apply that Configuration
• Machine Configuration can apply Configurations
• Automanage can automate this

Summary,

• Only supports Ubuntu up to 20.??
• Requirements on each client (there is a policy to deploy these across many machines)
• Managed identities required on each client
• Connection to a large list of endpoints required to communicate with management service
• Logs written to local filesystem
• Custom configurations are zip archives containing a compiled MOF file, modules and metadata !?!
• Configurations are published as resources in Azure
• DSC resources are written in Powershell
  • Requires writing Get, Set and Test methods
• Built in DSC resources are quite limited

[jemrobinson]

All the pros are related to ease-of-use in azure

• nice dashboard
• easy to enable for VMs without internet access (we use private endpoints)
• easy to install the VM agent
• can do pretty much anything that you can write a Test/Get/Set for in Powershell
• easy to compile a .ps1 file to MOF in Azure (currently doing this in Pulumi)

[JimMadge]

I can see the utility of a dashboard for people managing/maintaining a deployment. Quickly check if tasks are running or nodes are correctly configured. AWX could give us the same in Ansible world.

can do pretty much anything that you can write a Test/Get/Set for in Powershell

This feels like a bit of a double-edged sword. From a brief look there were not many build in resources compared to other CM tools so might require writing our own more often than we would like.

easy to compile a .ps1 file to MOF in Azure (currently doing this in Pulumi)

That is great, helps keep the configuration as-code in the repository.

[JimMadge]

AWX

Good overview here

• Dashboard/web interface for monitoring/managing Ansible
• Can create schedules and run ad-hoc jobs
• Manage inventories
• Can fetch projects from source control
  • Could be very useful for automatically importing our configuration
  • Could pin version of imported Ansible to safe haven version
  • Could provide a route to upgrade/downgrade configuration (but not infrastructure)
• Regular 'check mode' jobs could be used to test compliance
• Hooks for notifications (email, slack, etc.)
  • Could create alerts for failed jobs
• Authentication options (including AD)
• Can be deployed as a kubernetes cluster