failed to create containerd task: OCI runtime create failed: memory.kmem.limit_in_bytes: operation not supported: unknown

[mfalkvidd]

Summary

Pod is stuck in CrashLoopBackOff
kubectl describe pod shows the following reason:

failed to create containerd task: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:326: applying cgroup configuration for process caused: failed to write 1 to memory.kmem.limit_in_bytes: write /sys/fs/cgroup/memory/kubepods/.../memory.kmem.limit_in_bytes: operation not supported: unknown

What Should Happen Instead?

Pod should start

Reproduction Steps

This started happening after a reboot. My guess is that it happened due to switching to a new kernel version: 5.19.0-32-generic

Linux 5.19.0-32-generic #33~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Mon Jan 30 17:03:34 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

22.04.1 LTS (Jammy Jellyfish)

Introspection Report

inspection-report-20230217_092820.tar.gz

Can you suggest a fix?

Commenting out all resource directives (limits and requests) in the helm chart and running helm upgrade allows the pod to start.

Are you interested in contributing with a fix?

[neoaggelos]

@mfalkvidd Was this fixed by changing something particular in the helm chart configuration? I believe the same would have happened just by recreating the pod from scratch (e.g. with a kubectl rollout restart of the deployment/daemonset/etc)

Not sure if there is something we can do for this on the MicroK8s side, perhaps you might have better luck by creating an issue at the containerd repository?

[mfalkvidd]

@neoaggelos I fixed it by commenting out all resource directives (limits and requests) in the helm chart and running helm upgrade. I do not know if kubectl rollout restart would have worked instead.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.