Creating a RKE2 cluster without control-plane node registration

[cyvcloud]

Hi there,

I'm currently trying to create a RKE2 cluster where the Kubelet on the control-plane nodes does not register the node with the api server. Basically I would like the control-plane nodes to run all necessary services for operating the cluster without them being available as nodes to run workloads on from within the cluster.

My first try was to just disable Kubelet node registration for the server nodes by providing the following config:

kubelet-arg:
- register-node=false
[...]

This works insofar as that the Kubelet indeed does not perform the registration. Unfortunately rke2-server never completes startup as it is apparently waiting for the control-plane node to appear in the node list.

$ systemctl status rke2-server
● rke2-server.service - Rancher Kubernetes Engine v2 (server)
   Loaded: loaded (/usr/lib/systemd/system/rke2-server.service; enabled; vendor preset: disabled)
   Active: activating (start) since Wed 2022-01-26 12:52:53 UTC; 32min ago
     Docs: https://github.com/rancher/rke2#readme
  Process: 4683 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
  Process: 4677 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
  Process: 4674 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=0/SUCCESS)
 Main PID: 4687 (rke2)
    Tasks: 105
   Memory: 1.7G
   CGroup: /system.slice/rke2-server.service
           ├─4687 /usr/bin/rke2 server
           ├─4707 containerd -c /var/lib/rancher/rke2/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/rke2/agent/containerd
           [...]
           └─8340 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=cgroupfs --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --container-runtime=remote --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false -althz-bind-address=127.0.0.1 --hostname-override=rocky-4gb-fsn1-1 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file-max-size=50 --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --logtostderr=false --node-ip=10.0.0.5 --node-labels= --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --register-node=false --resolv-conf=/etc/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key

Jan 26 13:24:47 rocky-4gb-fsn1-1 rke2[4687]: time="2022-01-26T13:24:47Z" level=info msg="Waiting for control-plane node rocky-4gb-fsn1-1 startup: nodes \"rocky-4gb-fsn1-1\" not found"
Jan 26 13:24:48 rocky-4gb-fsn1-1 rke2[4687]: time="2022-01-26T13:24:48Z" level=info msg="Waiting for control-plane node rocky-4gb-fsn1-1 startup: nodes \"rocky-4gb-fsn1-1\" not found"
Jan 26 13:24:48 rocky-4gb-fsn1-1 rke2[4687]: time="2022-01-26T13:24:48Z" level=debug msg="Wrote ping"
Jan 26 13:24:49 rocky-4gb-fsn1-1 rke2[4687]: time="2022-01-26T13:24:49Z" level=info msg="Waiting for control-plane node rocky-4gb-fsn1-1 startup: nodes \"rocky-4gb-fsn1-1\" not found"
Jan 26 13:24:50 rocky-4gb-fsn1-1 rke2[4687]: time="2022-01-26T13:24:50Z" level=info msg="Waiting for control-plane node rocky-4gb-fsn1-1 startup: nodes \"rocky-4gb-fsn1-1\" not found"
Jan 26 13:24:51 rocky-4gb-fsn1-1 rke2[4687]: time="2022-01-26T13:24:51Z" level=info msg="Waiting for control-plane node rocky-4gb-fsn1-1 startup: nodes \"rocky-4gb-fsn1-1\" not found"
Jan 26 13:24:52 rocky-4gb-fsn1-1 rke2[4687]: time="2022-01-26T13:24:52Z" level=info msg="Waiting for control-plane node rocky-4gb-fsn1-1 startup: nodes \"rocky-4gb-fsn1-1\" not found"
Jan 26 13:24:53 rocky-4gb-fsn1-1 rke2[4687]: time="2022-01-26T13:24:53Z" level=info msg="Waiting for control-plane node rocky-4gb-fsn1-1 startup: nodes \"rocky-4gb-fsn1-1\" not found"
Jan 26 13:24:53 rocky-4gb-fsn1-1 rke2[4687]: time="2022-01-26T13:24:53Z" level=debug msg="Wrote ping"
Jan 26 13:24:54 rocky-4gb-fsn1-1 rke2[4687]: time="2022-01-26T13:24:54Z" level=info msg="Waiting for control-plane node rocky-4gb-fsn1-1 startup: nodes \"rocky-4gb-fsn1-1\" not found"

Is there a way to configure RKE2 to achieve my goal? Thanks!

[brandond]

No, this is not currently supported. We need the node object to exist for all control-plane and etcd nodes, as we use annotations on them to assist with managing the etcd cluster membership, locating the apiserver endpoint addresses, etc.