Bootstrap failed: Failed to establish TLS connection: underlying network error ()

[olehermanse]

I encountered this error message today, and even though I know how to solve it, I thought it could be helpful if we made an "accepted" answer here to help other users which may run into it.

When bootstrapping a client to the hub, one might encounter something that looks like a network error:

# sudo /var/cfengine/bin/cf-agent --bootstrap 1.2.3.4
  notice: Bootstrap mode: implicitly trusting server, use --trust-server=no if server trust is already established
   error: Failed to establish TLS connection: underlying network error ()
   error: No suitable server found for '/var/cfengine/inputs'
   error: Promise belongs to bundle 'failsafe_cfe_internal_update' in file '/var/cfengine/inputs/failsafe.cf' near line 121
   error: Errors encountered when actuating files promise '/var/cfengine/inputs'
   error: Failed to establish TLS connection: underlying network error (Connection reset by peer)
   error: No suitable server found for '/var/cfengine/modules'
   error: Promise belongs to bundle 'failsafe_cfe_internal_update' in file '/var/cfengine/inputs/failsafe.cf' near line 130
   error: Errors encountered when actuating files promise '/var/cfengine/modules'
   error: Failed to establish TLS connection: underlying network error (Connection reset by peer)
   error: No suitable server found for '/var/cfengine/inputs'
   error: Promise belongs to bundle 'failsafe_cfe_internal_update' in file '/var/cfengine/inputs/failsafe.cf' near line 144
   error: Comment is 'If we failed to fetch policy we try again using
                    the legacy default in case we are fetching policy
                    from a hub that is not serving mastefiles via a
                    shortcut.'
   error: Errors encountered when actuating files promise '/var/cfengine/inputs'
   error: Method 'failsafe_cfe_internal_update' failed in some repairs
R: Bootstrapping from host '1.2.3.4' via built-in policy '/var/cfengine/inputs/failsafe.cf'
R: This autonomous node assumes the role of voluntary client
   error: Bootstrapping failed, no input file at '/var/cfengine/inputs/promises.cf' after bootstrap

How do you troubleshoot or fix this?

[olehermanse]

Some things to check:

1. Correct IP: Is the IP address correct? The --bootstrap argument should always have the IP address where you can reach the hub (both when running the command on the hub and on clients).
2. Firewalls: Are any firewalls blocking the connection? For the bootstrap to work, inbound connections to the hub on port 5308 must be allowed. In general, for file transfer, reporting, and remote agent runs to work, port 5308 should be open on all CFEngine hosts.
3. Network test: It can be beneficial to test the network connection with separate tools. For example: ping 1.2.3.4, or cf-net -H 1.2.3.4 connect.
4. Server logs: On the hub, look for any connections or errors in the cf-serverd logs; cat /var/log/syslog | grep cf-serverd.

If you've ruled out the things above relating to IP addresses and firewalling, this is the most likely error message you will find in the cf-serverd logs on the hub:

error: Remote host '4.3.2.1' not in allowconnects, denying connection

CFEngine has a couple of variables for which IP addresses to trust (in terms of allowing connections and file transfers). By default, the hub will trust hosts on the same /16 subnet as itself, meaning in our example, the hub which has IP address 1.2.3.4 would trust anything which starts with 1.2.. Our example client, 4.3.2.1, does not.

This default, makes it easy to test and usually works, since you are usually testing clients on the same network when first starting out. However, it is recommended to change these values to the IP addresses you actually want to trust, especially for a production setup, but also it might be necessary if you're testing with hosts with more different IP addresses.

The fastest way to customize this is to create an augments file called def.json:

{
  "variables": {
    "default:def.acl": ["1.2.3.4/16", "4.3.2.1"]
  }
}

And put that in /var/cfengine/masterfiles/ on the hub. Or if you are already using cfbs, put it in your project and add it:

cfbs add ./def.json

Some configuration changes like this might need an agent run and/or a service restart on the hub:

cf-agent -KIf update.cf && cf-agent -KI && systemctl restart cfengine3

After this you should be able to run bootstrap again from the client, successfully:

sudo /var/cfengine/bin/cf-agent --bootstrap 1.2.3.4