WSL2: Network Policies still not working correctly in v0.25.0

[thepaulmacca]

What happened:

I noticed that #3713 has now been closed - but when I apply a deny network policy, it's still not being applied. Is this the same for anyone else?

What you expected to happen:

A deny network policy to take effect when using kind create cluster

How to reproduce it (as minimally and precisely as possible):

• Apply network policy with k apply -f https://github.com/thepaulmacca/cks-course-environment/raw/refs/heads/master/course-content/cluster-setup/network-policies/default-deny/default-deny-allow-dns.yaml

• Create pods to test

# create pods
k run frontend --image=nginx
k run backend --image=nginx

# create services
k expose pod frontend --port=80
k expose pod backend --port=80

# test connections, this shouldn't work now but still does
k exec frontend -- curl backend
k exec backend -- curl frontend

Anything else we need to know?:

Using WSL2 on Windows 11

Output of wsl --version

PS C:\Users\pmcdonald> wsl --version
WSL version: 2.3.24.0
Kernel version: 5.15.153.1-2
WSLg version: 1.0.65
MSRDC version: 1.2.5620
Direct3D version: 1.611.1-81528511
DXCore version: 10.0.26100.1-240331-1435.ge-release
Windows version: 10.0.22631.4317

This works fine with Calico installed, so have to stick with that for the moment

Environment:

• kind version: (use kind version): kind v0.25.0 go1.23.1 linux/amd64
• Runtime info: (use docker info, podman info or nerdctl info):

➜ docker info
Client: Docker Engine - Community
 Version:    25.0.3
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.16.2-desktop.1
    Path:     /usr/local/lib/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.2-desktop.2
    Path:     /usr/local/lib/docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.34
    Path:     /usr/local/lib/docker/cli-plugins/docker-debug
  desktop: Docker Desktop commands (Alpha) (Docker Inc.)
    Version:  v0.0.15
    Path:     /usr/local/lib/docker/cli-plugins/docker-desktop
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /usr/local/lib/docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.25
    Path:     /usr/local/lib/docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     /usr/local/lib/docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.3.0
    Path:     /usr/local/lib/docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /usr/local/lib/docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.13.0
    Path:     /usr/local/lib/docker/cli-plugins/docker-scout
WARNING: Plugin "/usr/local/lib/docker/cli-plugins/docker-scan" is not valid: failed to fetch metadata: fork/exec /usr/local/lib/docker/cli-plugins/docker-scan: no such file or directory

Server:
 Containers: 4
  Running: 2
  Paused: 0
  Stopped: 2
 Images: 7
 Server Version: 27.2.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 nvidia runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8fc6bcff51318944179630522a095cc9dbf9f353
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
 Kernel Version: 5.15.153.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 15.48GiB
 Name: docker-desktop
 ID: 1c85ecb4-fe68-4689-a089-ad34b4c68770
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///var/run/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
WARNING: daemon is not using the default seccomp profile

• OS (e.g. from /etc/os-release):

➜ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

• Kubernetes version: (use kubectl version):

➜ k version
Client Version: v1.30.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.31.2

• Any proxies or other special environment settings?: Not that I'm aware of

[BenTheElder]

cc @aojea

[Antonio and I are at KubeCon this week]

Just to clarify: this is with the defaults for kind create cluster, no config or flags?

[thepaulmacca]

@BenTheElder yes just with kind create cluster. I'm guessing it's maybe a WSL-specific issue if it's working for others

[aojea]

@BenTheElder yes just with kind create cluster. I'm guessing it's maybe a WSL-specific issue if it's working for others

can you please do a kind export logs and create a tarball and update the logs?

if is an environment related thing and with WSL, it may be possible you. miss some kernel modules

[thepaulmacca]

@aojea sure here you go
kind-logs.tar.gz

I know it can be a bit of a pain with WSL2, as to even get Cilium to work correctly requires quite a bit of effort - which is why I've stuck with Calico for now

[aojea]

yeah, you need to have nfqueue support in the kernel for this feature

2024-11-12T08:23:21.313392557Z stderr F add rule inet kindnet-network-policies postrouting ip saddr @podips-v4 queue num 101 bypass comment "process IPv4 traffic with network policy enforcement"
2024-11-12T08:23:21.31339452Z stderr F ^^^^^
2024-11-12T08:23:21.313395866Z stderr F /dev/stdin:21:72-76: Error: Could not process rule: No such file or directory

see for reference https://www.reddit.com/r/bashonubuntuonwindows/comments/vrldwj/problems_with_nfqueue_on_wsl/

I think the module name is xt_NFQUEUE

[thepaulmacca]

@aojea thanks for this, I'll take a look!

[BenTheElder]

We may need to update the WSL2 guide to handle ensuring these modules, pending confirmation

[thepaulmacca]

I'm not sure how you want to handle this issue now to be honest - but I've decided to just stick with Calico for now, as I'm not keen on messing around with the kernel just to get things like this working. It feels like too much hassle 😅

[BenTheElder]

In the past just running any sort of Kubernetes on WSL2 required a patched kernel due to missing core functionality. Today I think you just need some kernel modules loaded. These are for the same tech (nftables) that will be the default in kube-proxy upstream in the future so we need to address that anyhow.