Create k8s cluster with Principle of least privilege

[steled]

Hi, I am trying to deploy a k8s cluster using the KubeOne solution from Kubermatic. For this I resort to the Terraform example: vsphere_flatcar
On top of it, 3 VMs are created in VMware on which I subsequently create the Kubernetes cluster.

For the creation of the cluster I do not use the core user but have created a new user kubeone which has only the really necessary permissions.

The creation of the cluster works so far without problems, but I can not update the cluster to a new version.

After creating the cluster with the user kubeone the service docker.service does not start, which is needed. Another possibility would be to remove Docker completely.

My question is, does anyone have any idea what permission I still need to give the user kubeone so that also the service docker.service can be started successfully or is there a way to just remove docker completely?

Thanks a lot in advance...

P.s. Creating a cluster with the user core works without problems, with this Docker also starts normally.

[pothos]

Do you mean that the user you created can't start docker containers? You would have to add it to the docker group - but at that point it's equivalent with being root as docker containers running as root are able to gain full system access, so not much won compared to using the core user.

[steled]

Good morning @pothos,

the runtime they are using is containerd and containers get started and the cluster gets up and running.
My only problem is that KubeOne is looking that the docker.service is started. If it is started and running I can upgrade the cluster (as done with the core user).

I also added kubeone ALL=NOPASSWD:/usr/bin/systemctl start docker.service to the /etc/sudoers.d/kubeone file and than could start the service manually via the kubeone user. After that I'm also able to update the cluster but I don't want to do this manually 😛

As described at Outro it should be also work if docker is completely removed but I'm not that deep into FlatCar how to do this.

[pothos]

In the Ignition config in Terraform (the vsphere_flatcar link you gave) you can switch docker.service from being started on demand to run directly after bootup:

https://www.flatcar.org/docs/latest/container-runtimes/getting-started-with-docker/#permanently-running-a-container

[steled]

Thank you so much that worked...
Just for interest... Is it possible to completely remove docker from FlatCar and if yes how 😄

[pothos]

Docker can be disabled and even the systemd units removed but the docker command would still be in the path - so it depends on how the docker detection logic works. In the future (maybe half year ahead) we plan to split docker and containerd in tow components and it should be easy then to have docker be completely unavailable.

[steled]

So from the differences of journalctl I can see the following:

core user

Mar 23 14:35:42 kkp-test-core-cp-1 systemd[1]: Started OpenSSH per-connection server daemon (10.9.x.x:60598).
Mar 23 14:35:43 kkp-test-core-cp-1 sshd[1187]: Accepted publickey for core from 10.9.x.x port 60598 ssh2: ECDSA SHA256:<REDACTED>
Mar 23 14:35:43 kkp-test-core-cp-1 sshd[1187]: pam_unix(sshd:session): session opened for user core(uid=500) by (uid=0)
Mar 23 14:35:43 kkp-test-core-cp-1 systemd-logind[1014]: New session 3 of user core.
Mar 23 14:35:43 kkp-test-core-cp-1 systemd[1]: Started Session 3 of User core.
Mar 23 14:35:43 kkp-test-core-cp-1 sudo[1190]:     core : PWD=/home/core ; USER=root ; COMMAND=/usr/bin/cat /etc/os-release
Mar 23 14:35:43 kkp-test-core-cp-1 sudo[1190]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=500)
Mar 23 14:35:43 kkp-test-core-cp-1 sudo[1190]: pam_unix(sudo:session): session closed for user root
Mar 23 14:35:43 kkp-test-core-cp-1 systemd[1]: Starting Wait for Network to be Configured...
Mar 23 14:35:43 kkp-test-core-cp-1 systemd[1]: Finished Wait for Network to be Configured.
Mar 23 14:35:43 kkp-test-core-cp-1 systemd[1]: Reached target Network is Online.
Mar 23 14:35:43 kkp-test-core-cp-1 systemd[1]: Starting Docker Application Container Engine...
Mar 23 14:35:43 kkp-test-core-cp-1 env[1215]: time="2023-03-23T14:35:43.900186464Z" level=info msg="Starting up"
Mar 23 14:35:43 kkp-test-core-cp-1 env[1215]: time="2023-03-23T14:35:43.901251583Z" level=info msg="parsed scheme: \"unix\"" module=grpc
Mar 23 14:35:43 kkp-test-core-cp-1 env[1215]: time="2023-03-23T14:35:43.901369116Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Mar 23 14:35:43 kkp-test-core-cp-1 env[1215]: time="2023-03-23T14:35:43.901452194Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/libcontainerd/docker-containerd.sock  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
Mar 23 14:35:43 kkp-test-core-cp-1 env[1215]: time="2023-03-23T14:35:43.901515755Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Mar 23 14:35:43 kkp-test-core-cp-1 env[1215]: time="2023-03-23T14:35:43.903340028Z" level=info msg="parsed scheme: \"unix\"" module=grpc
Mar 23 14:35:43 kkp-test-core-cp-1 env[1215]: time="2023-03-23T14:35:43.903532154Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Mar 23 14:35:43 kkp-test-core-cp-1 env[1215]: time="2023-03-23T14:35:43.903609997Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/libcontainerd/docker-containerd.sock  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
Mar 23 14:35:43 kkp-test-core-cp-1 env[1215]: time="2023-03-23T14:35:43.903664697Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Mar 23 14:35:44 kkp-test-core-cp-1 env[1215]: time="2023-03-23T14:35:44.035481098Z" level=info msg="Loading containers: start."
Mar 23 14:35:44 kkp-test-core-cp-1 systemd-timesyncd[981]: Timed out waiting for reply from 167.86.115.96:123 (3.flatcar.pool.ntp.org).
Mar 23 14:35:44 kkp-test-core-cp-1 kernel: Initializing XFRM netlink socket
Mar 23 14:35:44 kkp-test-core-cp-1 env[1215]: time="2023-03-23T14:35:44.192625485Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Mar 23 14:35:44 kkp-test-core-cp-1 systemd-udevd[1225]: Using default interface naming scheme 'v250'.
Mar 23 14:35:44 kkp-test-core-cp-1 systemd-networkd[926]: docker0: Link UP
Mar 23 14:35:44 kkp-test-core-cp-1 env[1215]: time="2023-03-23T14:35:44.283788505Z" level=info msg="Loading containers: done."
Mar 23 14:35:44 kkp-test-core-cp-1 env[1215]: time="2023-03-23T14:35:44.302227228Z" level=warning msg="Not using native diff for overlay2, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled" storage-driver=overlay2
Mar 23 14:35:44 kkp-test-core-cp-1 env[1215]: time="2023-03-23T14:35:44.302580523Z" level=info msg="Docker daemon" commit=7f84219a3c graphdriver(s)=overlay2 version=20.10.18
Mar 23 14:35:44 kkp-test-core-cp-1 env[1215]: time="2023-03-23T14:35:44.302726504Z" level=info msg="Daemon has completed initialization"
Mar 23 14:35:44 kkp-test-core-cp-1 systemd[1]: Started Docker Application Container Engine.
Mar 23 14:35:44 kkp-test-core-cp-1 env[1215]: time="2023-03-23T14:35:44.345814511Z" level=info msg="API listen on /run/docker.sock"
Mar 23 14:35:50 kkp-test-core-cp-1 sudo[1354]:     core : PWD=/home/core ; USER=root ; COMMAND=/usr/bin/mkdir -p /etc/kubeone

kubeone user

Mar 23 14:42:32 kkp-test-kubeone-cp-1 systemd[1]: Started OpenSSH per-connection server daemon (10.9.x.x:60602).
Mar 23 14:42:33 kkp-test-kubeone-cp-1 sshd[1199]: Accepted publickey for kubeone from 10.9.x.x port 60602 ssh2: ECDSA SHA256:<REDACTED>
Mar 23 14:42:33 kkp-test-kubeone-cp-1 sshd[1199]: pam_unix(sshd:session): session opened for user kubeone(uid=1000) by (uid=0)
Mar 23 14:42:33 kkp-test-kubeone-cp-1 systemd[1]: Created slice User Slice of UID 1000.
Mar 23 14:42:33 kkp-test-kubeone-cp-1 systemd[1]: Starting User Runtime Directory /run/user/1000...
Mar 23 14:42:33 kkp-test-kubeone-cp-1 systemd-logind[1012]: New session 3 of user kubeone.
Mar 23 14:42:33 kkp-test-kubeone-cp-1 systemd[1]: Finished User Runtime Directory /run/user/1000.
Mar 23 14:42:33 kkp-test-kubeone-cp-1 systemd[1]: Starting User Manager for UID 1000...
Mar 23 14:42:33 kkp-test-kubeone-cp-1 systemd[1202]: pam_unix(systemd-user:session): session opened for user kubeone(uid=1000) by (uid=0)
Mar 23 14:42:33 kkp-test-kubeone-cp-1 systemd[1202]: Queued start job for default target Main User Target.
Mar 23 14:42:33 kkp-test-kubeone-cp-1 systemd[1202]: Reached target Paths.
Mar 23 14:42:33 kkp-test-kubeone-cp-1 systemd[1202]: Reached target Sockets.
Mar 23 14:42:33 kkp-test-kubeone-cp-1 systemd[1202]: Reached target Timers.
Mar 23 14:42:33 kkp-test-kubeone-cp-1 systemd[1202]: Reached target Basic System.
Mar 23 14:42:33 kkp-test-kubeone-cp-1 systemd[1202]: Reached target Main User Target.
Mar 23 14:42:33 kkp-test-kubeone-cp-1 systemd[1202]: Startup finished in 47ms.
Mar 23 14:42:33 kkp-test-kubeone-cp-1 systemd[1]: Started User Manager for UID 1000.
Mar 23 14:42:33 kkp-test-kubeone-cp-1 systemd[1]: Started Session 3 of User kubeone.
Mar 23 14:42:33 kkp-test-kubeone-cp-1 sudo[1210]:  kubeone : PWD=/home/kubeone ; USER=root ; COMMAND=/usr/bin/cat /etc/os-release
Mar 23 14:42:33 kkp-test-kubeone-cp-1 sudo[1210]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1000)
Mar 23 14:42:33 kkp-test-kubeone-cp-1 sudo[1210]: pam_unix(sudo:session): session closed for user root
Mar 23 14:42:34 kkp-test-kubeone-cp-1 systemd-timesyncd[981]: Timed out waiting for reply from 165.140.142.118:123 (3.flatcar.pool.ntp.org).
Mar 23 14:42:44 kkp-test-kubeone-cp-1 systemd-timesyncd[981]: Timed out waiting for reply from 37.247.53.178:123 (3.flatcar.pool.ntp.org).
Mar 23 14:44:14 kkp-test-kubeone-cp-1 systemd[1]: Starting Cleanup of Temporary Directories...
Mar 23 14:44:14 kkp-test-kubeone-cp-1 systemd-tmpfiles[1247]: /usr/lib/tmpfiles.d/legacy.conf:13: Duplicate line for path "/run/lock", ignoring.
Mar 23 14:44:14 kkp-test-kubeone-cp-1 systemd-tmpfiles[1247]: /usr/lib/tmpfiles.d/systemd.conf:33: Duplicate line for path "/var/lib/systemd", ignoring.
Mar 23 14:44:14 kkp-test-kubeone-cp-1 systemd[1]: systemd-tmpfiles-clean.service: Deactivated successfully.
Mar 23 14:44:14 kkp-test-kubeone-cp-1 systemd[1]: Finished Cleanup of Temporary Directories.
Mar 23 14:44:35 kkp-test-kubeone-cp-1 sudo[1273]:  kubeone : PWD=/home/kubeone ; USER=root ; COMMAND=/usr/bin/mkdir -p /etc/kubeone

The part env[1215] is completely missing for user kubeone, to me it looks like this is the important part.

[steled]

The answer, as described here, is to start docker automatically and not via the socket, see here

Thanks to @pothos for the support