[1.18] Ubuntu 20.04 + Kubernetes 1.31.2: Container creation error: writing file devices.allow: Operation not permitted

[hswong3i]

When upgrading crun from 1.17 to 1.1.8, on Ubuntu 20.04 + Kubernetes 1.31.2, CRI-O couldn't start container correctly with following log message (rolling back to 1.17 solve the problem):

Oct 24 06:37:22 kube71-sg crio[1675]: time="2024-10-24 06:37:22.288118821Z" level=error msg="Container creation error: writing file `devices.allow`: Operation not permitted\n" id=9f180776-5827-436e-bad7-4e7bee0c4bbc name=/runtime.v1.RuntimeService/CreateContainer
Oct 24 06:37:24 kube71-sg crio[1675]: time="2024-10-24 06:37:24.270590118Z" level=warning msg="Failed to open /etc/passwd: open /var/lib/containers/storage/overlay/cb7e52099d0a3c97f8d41a06a9d00ee3c9a711275d4e6b6d1a7e9f9ad7d5ddb1/merged/etc/passwd: no such file or directory"
Oct 24 06:37:24 kube71-sg crio[1675]: time="2024-10-24 06:37:24.270639749Z" level=warning msg="Failed to open /etc/group: open /var/lib/containers/storage/overlay/cb7e52099d0a3c97f8d41a06a9d00ee3c9a711275d4e6b6d1a7e9f9ad7d5ddb1/merged/etc/group: no such file or directory"
Oct 24 06:37:24 kube71-sg crio[1675]: time="2024-10-24 06:37:24.285312015Z" level=error msg="Container creation error: writing file `devices.allow`: Operation not permitted\n" id=d8c243fc-ff81-4758-8b8c-af242cfb6b10 name=/runtime.v1.RuntimeService/CreateContainer
Oct 24 06:37:27 kube71-sg crio[1675]: time="2024-10-24 06:37:27.270402292Z" level=warning msg="Failed to open /etc/passwd: open /var/lib/containers/storage/overlay/eb53b78f73ceccc44bc269c94daaba6dd872295c16a9ece48aeb66c37a10eb34/merged/etc/passwd: no such file or directory"
Oct 24 06:37:27 kube71-sg crio[1675]: time="2024-10-24 06:37:27.270470977Z" level=warning msg="Failed to open /etc/group: open /var/lib/containers/storage/overlay/eb53b78f73ceccc44bc269c94daaba6dd872295c16a9ece48aeb66c37a10eb34/merged/etc/group: no such file or directory"
Oct 24 06:37:27 kube71-sg crio[1675]: time="2024-10-24 06:37:27.284248544Z" level=error msg="Container creation error: writing file `devices.allow`: Operation not permitted\n" id=c7b8d2e7-f1c5-4c10-9f27-e5d7c0b237fd name=/runtime.v1.RuntimeService/CreateContainer
Oct 24 06:37:29 kube71-sg crio[1675]: time="2024-10-24 06:37:29.286943776Z" level=error msg="Container creation error: writing file `devices.allow`: Operation not permitted\n" id=37cf0c7b-7f6a-4462-90a0-a62d003b780c name=/runtime.v1.RuntimeService/CreateContainer
Oct 24 06:37:29 kube71-sg crio[1675]: time="2024-10-24 06:37:29.321086409Z" level=error msg="Container creation error: writing file `devices.allow`: Operation not permitted\n" id=524557d6-74fb-4854-99fb-ec338b115547 name=/runtime.v1.RuntimeService/CreateContainer
Oct 24 06:37:29 kube71-sg crio[1675]: time="2024-10-24 06:37:29.355156161Z" level=error msg="Container creation error: writing file `devices.allow`: Operation not permitted\n" id=991b0bfc-acf3-4743-a409-6f99347f3d23 name=/runtime.v1.RuntimeService/CreateContainer
Oct 24 06:37:35 kube71-sg crio[1675]: time="2024-10-24 06:37:35.284218901Z" level=error msg="Container creation error: writing file `devices.allow`: Operation not permitted\n" id=147b029f-b70c-432b-ac0a-c4a360a4659a name=/runtime.v1.RuntimeService/CreateContainer
Oct 24 06:37:35 kube71-sg crio[1675]: level=warning msg="Failed to connect to agent socket at unix:///var/run/cilium/cilium.sock." containerID=ddf412af5d1ed0d4d3dff0b9df13b553a5b5dc5b1c0736dae4f0d174a8dce5d3 error="failed to create cilium agent client after 10.000000 seconds timeout: Get \"http://localhost/v1/config\": dial unix /var/run/cilium/cilium.sock: connect: no such file or directory" eventUUID=ef9baef6-f8bf-4730-b908-b85d3327809d subsys=cilium-cni
Oct 24 06:37:35 kube71-sg crio[1675]: level=info msg="Agent is down, falling back to deletion queue directory" containerID=ddf412af5d1ed0d4d3dff0b9df13b553a5b5dc5b1c0736dae4f0d174a8dce5d3 eventUUID=ef9baef6-f8bf-4730-b908-b85d3327809d subsys=cilium-cni
Oct 24 06:37:35 kube71-sg crio[1675]: level=info msg="Queueing deletion request for endpoint" containerID=ddf412af5d1ed0d4d3dff0b9df13b553a5b5dc5b1c0736dae4f0d174a8dce5d3 endpointID="container-id:ddf412af5d1ed0d4d3dff0b9df13b553a5b5dc5b1c0736dae4f0d174a8dce5d3" eventUUID=ef9baef6-f8bf-4730-b908-b85d3327809d subsys=cilium-cni
Oct 24 06:37:35 kube71-sg crio[1675]: level=info msg="wrote queued deletion file" containerID=ddf412af5d1ed0d4d3dff0b9df13b553a5b5dc5b1c0736dae4f0d174a8dce5d3 eventUUID=ef9baef6-f8bf-4730-b908-b85d3327809d subsys=cilium-cni
Oct 24 06:37:36 kube71-sg crio[1675]: time="2024-10-24 06:37:36.271964712Z" level=warning msg="Failed to open /etc/passwd: open /var/lib/containers/storage/overlay/de8a45294b615a7817734ab6bdbe3bb336ca2d30f29d2d885a35d543ac0590fa/merged/etc/passwd: no such file or directory"
Oct 24 06:37:36 kube71-sg crio[1675]: time="2024-10-24 06:37:36.272270244Z" level=warning msg="Failed to open /etc/group: open /var/lib/containers/storage/overlay/de8a45294b615a7817734ab6bdbe3bb336ca2d30f29d2d885a35d543ac0590fa/merged/etc/group: no such file or directory"
Oct 24 06:37:36 kube71-sg crio[1675]: time="2024-10-24 06:37:36.286254185Z" level=error msg="Container creation error: writing file `devices.allow`: Operation not permitted\n" id=09a606f7-89b7-424b-b17a-d6cf7de53b96 name=/runtime.v1.RuntimeService/CreateContainer

Most likely should be #1574 related?

[giuseppe]

any chance you could try #1591 ?

[giuseppe]

are you on cgroup v1?

what version of CRI-O?

[hswong3i]

Yes in cgroup v1; I also enable cgroup v1 for CentOS 7/8, openSUSE Leap 15.5/15.6, too.

Because at that ages, most OS cgroup v2 support are "claimed" as ready but always fail, so I had hard code the grub parameter with cgroup v1.

[giuseppe]

I don't think we should put more efforts in supporting cgroup v1. systemd already moved in that direction, and I think crun should do the same: #1593

What issues exactly are you encountering with cgroup v2?

[hswong3i]

I don't think we should put more efforts in supporting cgroup v1. systemd already moved in that direction, and I think crun should do the same: #1593

Agree that we shouldn't "put more efforts in supporting cgroup v1", but that should be different with "suddenly drop cgroup v1 without announcement", isn't it?

What issues exactly are you encountering with cgroup v2?

Something like Ubuntu 20.04 LTS default with cgroup v1, and going to EOL on 2025-04, before that could we put cgroup v1 into deprecate but not a sudden death?

When switch such legacy LTS from cgroup v1 to v2, I need to give a strong justification to client, else they couldn't agree with such change in production, especially within this final 6 months before EOL...

[giuseppe]

cgroup v1support won't be removed over night. For now, it is only deprecated and a warning was added.

Realistically, I think support for it can be dropped at some point next year.

[hswong3i]

cgroup v1support won't be removed over night. For now, it is only deprecated and a warning was added.

Realistically, I think support for it can be dropped at some point next year.

Honestly, it is so difficult for my OBS repo to support those legacy OS:

• CentOS 8 Stream (EOL on 2024-05)
• CentOS 7 (EOL on 2024-07)
• Debian 11 (EOL on 2024-08)
• openSUSE Leap 15.5 (EOL on 2024-12)
• Ubuntu 20.04 (EOL on 2025-04)
• openSUSE Leap 15.6 (EOL on 2025-12)

Somehow I am going to remove those legacy OS support when openSUSE Leap 16.0 get released, around 2025-10 (see https://en.opensuse.org/openSUSE:Roadmap#DRAFT_Schedule_for_Leap_16.0)... Or realistically as early as 2025-04 when Ubuntu 20.04 EOL...

P.S. the main reason for me is about existing clients supporting: as long as community still supporting those LTS, clients ALWAYS be so lazy for upgrade; even if those LTS already EOL, clients may at least asking for +3 months extended support before upgrade to the latest LTS...