Integrating cf-remote in an existing setup

[MasinAD]

We already integrated several clients manually but now I want to explore cf-remote.

At the moment there are several open questions re cf-engine install.

The install command distinguishes between --hub and --client as well as between --hub-package and --client-package. AFAICT the distinction in the packages is only necessary for the enterprise edition, is that right? The distinction between --hub and --client now seems to have less impact. It seems that --hub targets machines that should act as 'hub', and if I understand that term correctly a 'hub' is some kind of controlling server, basically what the Salt Master is for Saltstack or the Puppetmaster for Puppet. On a --demo system, it gets a def.json preinstalled.

Now, I don't really get where to execute cf-remote. Do I run it on my personal machine, orchestrating a fleet of clients bootstrapped to a hub? Do I run it on the machine I deemed as 'hub' but what is the --hub intended for, then?

Our CFEngine server is a FreeBSD machine. This is clearly wrong on both sides, mine and cf-remote's:

⬢[masin.wiedner@toolbox ~]$ cf-remote install --edition community --hub root@it-pm --clients root@gamma
root@it-pm
OS            : Freebsd 13
Architecture  : amd64
CFEngine      : Not installed
Policy server : it-pm.wikimedia.de
Binaries      : pkg

But CFEngine is installed, package cfengine321 (my colleague did that, don't know if it's the proper package). Seems like cf-remote does not recognize the package already being installed. In remote.py line 216 cf-remote queries the path of the cf-agent binary. A few lines later, the path gets hardcoded which causes cf-remote to fail to determine the installed version on FreeBSD resulting in the assumption CFEngine wasn't installed. The FreeBSD package does not install binaries into /var/cfengine/bin.

I think, FreeBSD is not officially supported by cf-remote. I'd be fine with that if –and now we're getting back to an earlier question– I was able to run cf-remote on my personal machine while still telling it which machine to use as a 'hub'. I'm still not entirely sure if 'hub' and 'policy server' mean the same thing.

But when I run cf-remote on the controlling server, cf-remote gets stuck at the bootstrapping step:

root@it-pm:~ # cf-remote install --edition community --clients [email protected] --remote-download --bootstrap it-pm.wikimedia.de

[email protected]
OS            : Debian 11
Architecture  : x86_64
CFEngine      : 3.21.4 (Community)
Policy server : None
Binaries      : dpkg, apt

Downloading 'https://cfengine-package-repos.s3.amazonaws.com/community_binaries/Community-3.21.4/agent_debian11_x86_64/cfengine-community_3.21.4-1.debian11_amd64.deb' on '[email protected]' using curl
Installing: 'cfengine-community_3.21.4-1.debian11_amd64.deb' on '[email protected]'
CFEngine 3.21.4 (Community) was successfully installed on '[email protected]'
Bootstrapping: '172.16.66.3' -> 'it-pm.wikimedia.de'

Why might this be? The policy server has been locked down I think

		"trustkeysfrom": ["127.0.0.1/32"],

Yet again, I think cf-remote serves a purpose of quickly provisioning a fleet of machines, bootstrapping them to the hub, and then the admin needs to lockdown the hub. I'd expect that cf-remote automates the steps of copying the localhost.pub to the hub and then run cf-key --trust-key on the copied public key but that doesn't seem to happen nor is it considered. Or am I doing something wrong? After manually trusting the key, the bootstrap step completes.

Is there a way to provide vars.agent_name somehow when using cf-remote, e.g. --clients gamma:[email protected] for setting the agent_name "gamma" for this client? When using puppet theres something like puppet config set certname `. I'm not saying, CFEngine should do everything as This Other Tool™, it's just a suggestion to get inspiration on how others achieve their stuff.

And lastly, cf-remote show promises to

Show hosts spawned by or added to cf-remote
I know how to spawn hosts using cf-remote but that's no use-case here. How would I add hosts to cf-remote? cf-keys shows 23 entries at the moment but cf-remote show shows – none.

Sorry for all those questions. I feel a little bad because @olehermanse did such a great job advertising it to me, yet I still run into lots of blockers. I try to comfort myself with my questions maybe being helpful for improving the docs or the tool itself :-).

[olehermanse]

@MasinAD Lots of things to answer here so tell me if I miss something;

Now, I don't really get where to execute cf-remote. Do I run it on my personal machine, orchestrating a fleet of clients bootstrapped to a hub? Do I run it on the machine I deemed as 'hub' but what is the --hub intended for, then?

Up to you. I prefer to run some tools like this on my machine (terminal, SSH, editor, scp, cfbs, cf-remote, etc.). Since cf-remote uses SSH to communicate with the hosts, you will need to run it from a machine which has network access and SSH access to all the machines where you plan to install CFEngine with it.

I'd like to recommend going through the getting started tutorial on our website:

https://docs.cfengine.com/docs/3.21/getting-started.html

It tries to make these things a bit more clear. It defaults to using CFEngine Enterprise, and letting you test some of the Enterprise features. But it should be valuable for you, IMO, even if you don't intend on using Enterprise after finishing the tutorial, since it introduces the various tools and concepts.

It seems that --hub targets machines that should act as 'hub', and if I understand that term correctly a 'hub' is some kind of controlling server

Yes. In a normal setup, the hub is the host responsible for serving policy (which clients fetch), and in CFEngine Enterprise it is also where the Web UI and reporting database is located. CFEngine is higly customizable / programmable - you can set up hosts to fetch policy from multiple servers, or even build something completely distributed ("peer-to-peer"), but I'm trying to focus on the easy / default case here.

For small infrastructures it's normal to have 1 hub. For larger infrastructures it's common to have multiple. The 2 most common reasons for having multiple are: Separate environments (dev, test, prod) and scalability - we currently support up to 5000 hosts per hub.

The install command distinguishes between --hub and --client as well as between --hub-package and --client-package. AFAICT the distinction in the packages is only necessary for the enterprise edition, is that right?

Kind of. The short answer is that --hub-package controls which package is installed on the host specified in the --hub argument, while --client-package controls which package is installed on the hosts in --clients. Both can be ommitted, in which case cf-remote will attempt to determine the correct package based on the platform of the host(s). This can still be useful for community, for example:

cf-remote install --hub centos-7-hub --clients ubuntu-20-clients --hub-package ./packagename.rpm --client-package ./packagename.deb --bootstrap centos-7-hub

(In this example I am using saved groups, which come from cf-remote save or cf-remote spawn).

In Enterprise there are different packages for hubs and clients (since the hub package contains everything needed for the reporting database and Web UI - Apache, PostgreSQL, etc.). Furthermore, due to this separation, the default policy (Masterfiles Policy Framework) only exists in the hub packages, since it's not needed in the client. In Community, we've included the policy in the packages to make it easy, just one package for hub or client. So yea, in community you can use the same packages (provided you are talking about running the hub and the clients on the same OS).

I'd expect that cf-remote automates the steps of copying the localhost.pub to the hub and then run cf-key --trust-key on the copied public key but that doesn't seem to happen nor is it considered. Or am I doing something wrong? After manually trusting the key, the bootstrap step completes.

This is a good idea. It's halfway there, in the form of the cf-remote scp command for transferring files, and the cf-remote install --trust-key option, for trusting a specific key from file. I think we've discussed it before, but haven't added it yet. I've created a ticket for adding it: https://northerntech.atlassian.net/browse/CFE-4404

Is there a way to provide vars.agent_name somehow when using cf-remote, e.g. --clients gamma:[email protected] for setting the agent_name "gamma" for this client?

Yes, there is cf-remote save - it's covered in the getting started tutorial mentioned above, looks like we could also add an example for it in the cf-remote README.

I think, FreeBSD is not officially supported by cf-remote.

CFEngine does not officially support FreeBSD - we don't provide packages on FreeBSD and we don't regularly test on FreeBSD. Still, there are some people running CFEngine on FreeBSD, compiling it themself, and we are happy to accept their patches. In some cases we do go in and test and fix an issue on another platform like FreeBSD, but naturally, this is not a very high priority compared to the requests from paying customers.

In order to improve the support for FreeBSD, there are 2 good routes I see:

1. Contribute to this open source project by testing cf-remote and CFEngine on FreeBSD, submit bugs you find and try to fix the issues which are the most important. Scratch your own itch, and improve the world for everyone else on FreeBSD.
2. Or, contact us, about a more formal customer agreement, where we can, for a price, officially support FreeBSD, including building official packages, prioritize fixing bugs, run automated tests, help you via technical support, etc.

But when I run cf-remote on the controlling server, cf-remote gets stuck at the bootstrapping step

Hard to say why this is happening. To debug it further, I'd try to log in on the host and run the same command as cf-remote. (If you use cf-remote --log-level DEBUG you can see everything it does). Looks like in this case it's

cf-agent --bootstrap it-pm.wikimedia.de

FYI: It's not very common to bootstrap to domain names - people don't usually trust DNS for this kind of setup. CFEngine does support resolving the domain name, but once it starts changing, it becomes tricky, and when there inevitably is a DNS problem (The problem is always DNS), it's problematic. If you can avoid that and use a static IP address for the hub, I'd recommend doing that.

[MasinAD]

I must have missed cf-remote save. Thanks for pointing in that direction. And many more thanks for another excellent answer to my woes :-). Honestly, I didn't expect the "Getting Started" to reference cf-remote at all because it looks more like a distinct project than an officially supported management tool. But that problem's onto me as I only joined later after my colleague did the heavy lifting of setting up the server environment (and that's the reason it's not Debian! I can't leave him unsupervised 😄).

For the moment, a good approach for our setup might be to follow these steps:

1. cf-remote save --role client --name GROUPNAME --hosts SSH_USER@HOST
2. cf-remote install --edition community --clients GROUPNAME --remote-download
3. *magic*
4. cf-remote install --clients GROUPNAME --bootstrap HUB

That third step is a little bit tricky. I think we're both talking about different procedures:

• Having a way for the client to trust the server
• Having a way for the server to trust the client

At the moment, I try to semi-automatize the latter variant. I run cf-remote on my personal workstation and want to provision a client. I need the /var/cfengine/ppkeys/localhost.pub of the client to put it onto the hub so I can cf-key -t PUBKEY before bootstrapping. That's why it stuck on the bootstrapping step before: The server did not trust the client's key.

cf-remote scp --hosts HOSTS allows me to put files on the hosts but at the moment I'd need to fetch a file from there and put it onto the hub. Maybe I should rather scp root@client:/var/cfengine/ppkeys/localhost.pub . && scp localhost.pub root@hub:/tmp/ && ssh root@hub cf-key -t /tmp/localhost.pub.

I think I could implement at least part of that in cf-remote. I'll give it a try.

The fourth step might not work as I expect as cf-remote seems to attempt a reinstall before bootstrapping. But I'll take a look at it when I figured out the third step.

[olehermanse]

@MasinAD Right, if CFEngine is installed, cf-remote install is not the appropriate step. When the package is installed, you'll need to:

1. Establish trust (mutually, both on hub and client).
2. Run the bootstrap command.

If you trust the network during the setup phase, 1 can be skipped and you use "automatic trust" during bootstrap. (Controlled by the trustkeysfrom attribute). If not, you have to get the keys transferred securely, this is something we usually refer to as secure bootstrap.

Running the bootstrap command can be done by:

sudo cf-agent -B <hub_ip>

Or via cf-remote:

cf-remote sudo -H <hosts> "cf-agent -B <hub ip>"

I'll just leave some resources here which you might find useful:

1. Secure bootstrap in CFEngine documention
2. Documentation for trustkeysfrom
3. Blog post on configuring allowed hosts / IP addresses and the modules related to this
4. Module for disabling automatic trust, with explanations of how that works, why you'd want to do it, etc..

I'll close this discussion by marking my reply above as answer. Feel free to open more question threads for other issues you run into, they might help others who encounter the same situations.