Creating multi-node cluster from yaml file fails as a user. Workers are unable to join the controller. Succeeds as root.

[psycoder1]

What happened:
I tried to create a multi-worker kubertenes. I got an error on Debian Linux, using Podman.

I used the following .yaml file:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker

I run the following command:

systemd-run --scope --user -p "Delegate=yes" kind create cluster --retain  --config kind-config.yaml 

What you expected to happen:
Successful creation of a cluster with 2 worker nodes. Instead I got an error.

How to reproduce it (as minimally and precisely as possible):

1. On Debian Linux, install Podman and Kind
2. Create kind-config.yaml file with the contents from above.
3. Run systemd-run --scope --user -p "Delegate=yes" kind create cluster --retain --config kind-config.yaml

Running the following command as root succeeds

kind create cluster --retain  --config kind-config.yaml

I attached the logs from the error

Anything else we need to know?:
I love you guys!

Environment:

• kind version: kind version 0.26.0

• Runtime info: (use docker info, podman info or nerdctl info):

podman info

host:
  arch: amd64
  buildahVersion: 1.28.2
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_2.1.6+ds1-1_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.6, commit: unknown'
  cpuUtilization:
    idlePercent: 97.88
    systemPercent: 1.38
    userPercent: 0.74
  cpus: 12
  distribution:
    codename: bookworm
    distribution: debian
    version: "12"
  eventLogger: journald
  hostname: optimalwebapps
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.1.0-28-amd64
  linkmode: dynamic
  logDriver: journald
  memFree: 104969277440
  memTotal: 135038291968
  networkBackend: cni
  ociRuntime:
    name: crun
    package: crun_1.8.1-1+deb12u1_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.1
      commit: f8a096be060b22ccd3d5f3ebe44108517fbf6c30
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.0-1_amd64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 31999389696
  swapTotal: 31999389696
  uptime: 2h 5m 31.00s (Approximately 0.08 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /home/XXXXX/.config/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 3
    stopped: 0
  graphDriverName: vfs
  graphOptions: {}
  graphRoot: /home/XXXXX/.local/share/containers/storage
  graphRootAllocated: 295619928064
  graphRootUsed: 128423350272
  graphStatus: {}
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 6
  runRoot: /run/user/1000/containers
  volumePath: /home/XXXXX/.local/share/containers/storage/volumes
version:
  APIVersion: 4.3.1
  Built: 0
  BuiltTime: Thu Jan  1 02:00:00 1970
  GitCommit: ""
  GoVersion: go1.19.8
  Os: linux
  OsArch: linux/amd64
  Version: 4.3.1

• OS: Debian GNU/Linux 12 (bookworm)

• Kubernetes version: Client Version: v1.32.0
  Kustomize Version: v5.5.0

• Any proxies or other special environment settings?:
  None that I know of.
  log_bundle.tar.gz

[stmcginnis]

The kubelet is reporting:

"Failed to ensure process in container with oom score" err="failed to apply oom score -999 to PID 736: write /proc/736/oom_score_adj: permission denied"

This looks like a podman issue. Just a quick search, but it might be a common problem. That's an old and closed issue, but a lot of recent comments from folks running into it with a few workarounds.

I don't use podman myself, so I can't easily reproduce or verify a fix. But maybe look through there and see if there is a workaround in the recent comments that could help?