Unable to enable QUIC eBPF routing.

[lzlrd]

Describe the bug

Attempting to utilise quic_bpf results in the following error:

2023/09/28 22:48:53 [alert] 1#1: failed to create BPF map (1: Operation not permitted)
nginx: [alert] failed to create BPF map (1: Operation not permitted)
2023/09/28 22:48:53 [emerg] 1#1: ngx_quic_bpf_module failed to initialize, check limits
nginx: [emerg] ngx_quic_bpf_module failed to initialize, check limits

I've attempted to set an unlimited RLIMIT_MEMLOCK, assign the relevant capabilities as defined at https://docs.nginx.com/nginx-service-mesh/reference/permissions (CAP_NET_ADMIN,CAP_NET_RAW,CAP_SYS_RESOURCE, and CAP_SYS_ADMIN), use the host network, pass the --privileged, unconfine Seccomp (--security-opt seccomp=unconfined), run NGINX as root (within the container), and all the above together but still receive the same error.

Running on the host seems to work fine, however.

To reproduce

Steps to reproduce the behavior:

1. Deploy the NGINX Docker image using podman run -d --name nginx -p 80:80/tcp -p 443:443/tcp -p 443:443/udp -v /srv/nginx:/etc/nginx -v /etc/letsencrypt:/etc/letsencrypt --restart unless-stopped --cap-add NET_ADMIN,NET_RAW,SYS_RESOURCE,BPF,PERFMON --net <REDACTED> --ulimit memlock=-1:-1 library/nginx:mainline (potentially including the flags mentioned above).
2. The container exits with the error shared above.

Expected behavior

Following a provision of the relevant capabilities and spare RLIMIT_MEMLOCK resources, QUIC eBPF routing should work fine.

Your environment

• podman version 4.6.2
• nginx:mainline

Additional context

n/a

[thresheek]

What OS and docker engine you're running on?

It works with just --privileged here on Ubuntu 22.04 aarch64 with nginx:1.25.2 and docker-ce=5:20.10.23~3-0~ubuntu-jammy.

Fails with just --cap-add BPF: docker: Error response from daemon: invalid CapAdd: unknown capability: "CAP_BPF".

[lzlrd]

@thresheek, I'm using podman version 4.6.2 on Arch Linux. CAP_BPF was been added in Linux 5.8 so a kernel version at least that old is required; could you try see if it work if you pass --cap-add NET_ADMIN --cap-add NET_RAW --cap-add SYS_RESOURCE --cap-add SYS_ADMIN (CAP_SYS_ADMIN should contain the relevant permissions for BPF)?

[thresheek]

Hi @lazerl0rd, yes, docker run --cap-add NET_ADMIN --cap-add NET_RAW --cap-add SYS_RESOURCE --cap-add SYS_ADMIN seems to work fine under my setup.

[thresheek]

Hi @lazerl0rd, have you been able to sort out your podman setup?

[lzlrd]

Hey, @thresheek. I couldn't get it working even when running the container privileged.

Unfortunately, the server has had a disk error and I haven't got round to resolving that and am unable to test further at the moment.

[raldone01]

I did a quick test with docker. SYS_ADMIN is not necessary.

The following works for me:

cap_add:
      # https://man7.org/linux/man-pages/man7/capabilities.7.html
      - BPF
      - NET_ADMIN
      - PERFMON