Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Creating LoadBalancer service blocks API server IP #1685

Open
k6av opened this issue Jun 29, 2024 · 8 comments · May be fixed by #1699
Open

Creating LoadBalancer service blocks API server IP #1685

k6av opened this issue Jun 29, 2024 · 8 comments · May be fixed by #1699
Labels
bug

Comments

@k6av
Copy link

k6av commented Jun 29, 2024

What happened?
After creating a service with type set to LoadBalancer, the Kubernetes API server becomes unreachable.

What did you expect to happen?
API server accessibility is unaffected.

How can we reproduce the behavior you experienced?
Steps to reproduce the behavior:

  1. Create a load balancer service (kubectl create service loadbalancer example --tcp=1337:1337)
  2. Try to:
    • Access any API server endpoint (kubectl get nodes), or
    • Ping the API server IP address.

System Information:

  • Kube-Router Version (kube-router --version): v2.1.2
  • Kube-Router Parameters: --enable-cni=true --run-router=true --run-firewall=true --run-service-proxy=true --run-loadbalancer=true --bgp-graceful-restart=true --kubeconfig=/var/lib/kube-router/kubeconfig --enable-ipv4=false --enable-ipv6=true --router-id=generate --service-cluster-ip-range=fdcd:5f23:4ab4:11::/112 --loadbalancer-ip-range=fdcd:5f23:4ab4:12::/112
  • Kubernetes Version (kubectl version) : v1.28.10
  • Cloud Type: on premise
  • Kubernetes Deployment Type: custom
  • Kube-Router Deployment Type: DaemonSet
  • Cluster Size: single node

Logs, other output, metrics

ipset list output before creating the service (API server is reachable) 

Name: inet6:kube-router-pod-subnets
Type: hash:net
Revision: 7
Header: family inet6 hashsize 1024 maxelem 65536 timeout 0 bucketsize 12 initval 0xe7ac8722
Size in memory: 1336
References: 2
Number of entries: 1
Members:
fdcd:5f23:4ab4:10::/64 timeout 0

Name: inet6:kube-router-node-ips
Type: hash:ip
Revision: 6
Header: family inet6 hashsize 1024 maxelem 65536 timeout 0 bucketsize 12 initval 0xe1f932ec
Size in memory: 304
References: 1
Number of entries: 1
Members:
fdcd:0cd1:5dd1:28::1 timeout 0

Name: inet6:kube-router-local-ips
Type: hash:ip
Revision: 6
Header: family inet6 hashsize 1024 maxelem 65536 timeout 0 bucketsize 12 initval 0x5689a634
Size in memory: 224
References: 1
Number of entries: 0
Members:

Name: inet6:kube-router-svip
Type: hash:ip
Revision: 6
Header: family inet6 hashsize 1024 maxelem 65536 timeout 0 bucketsize 12 initval 0x978ab45a
Size in memory: 384
References: 1
Number of entries: 2
Members:
fdcd:5f23:4ab4:11::1 timeout 0
fdcd:5f23:4ab4:11::10 timeout 0

Name: inet6:kube-router-svip-prt
Type: hash:ip,port
Revision: 7
Header: family inet6 hashsize 1024 maxelem 65536 timeout 0 bucketsize 12 initval 0xf49105d3
Size in memory: 616
References: 1
Number of entries: 4
Members:
fdcd:5f23:4ab4:11::10,tcp:53 timeout 0
fdcd:5f23:4ab4:11::10,tcp:9153 timeout 0
fdcd:5f23:4ab4:11::10,udp:53 timeout 0
fdcd:5f23:4ab4:11::1,tcp:443 timeout 0

ipset list output after creating the service (API server is unreachable) 

Name: inet6:kube-router-pod-subnets
Type: hash:net
Revision: 7
Header: family inet6 hashsize 1024 maxelem 65536 timeout 0 bucketsize 12 initval 0xe7ac8722
Size in memory: 1336
References: 2
Number of entries: 1
Members:
fdcd:5f23:4ab4:10::/64 timeout 0

Name: inet6:kube-router-node-ips
Type: hash:ip
Revision: 6
Header: family inet6 hashsize 1024 maxelem 65536 timeout 0 bucketsize 12 initval 0xe1f932ec
Size in memory: 304
References: 1
Number of entries: 1
Members:
fdcd:0cd1:5dd1:28::1 timeout 0

Name: inet6:kube-router-local-ips
Type: hash:ip
Revision: 6
Header: family inet6 hashsize 1024 maxelem 65536 timeout 0 bucketsize 12 initval 0x9e1e76c3
Size in memory: 224
References: 1
Number of entries: 0
Members:

Name: inet6:kube-router-svip
Type: hash:ip
Revision: 6
Header: family inet6 hashsize 1024 maxelem 65536 timeout 0 bucketsize 12 initval 0x5689a634
Size in memory: 544
References: 1
Number of entries: 4
Members:
fdcd:5f23:4ab4:11::5402 timeout 0
fdcd:0cd1:5dd1:28::1 timeout 0
fdcd:5f23:4ab4:11::10 timeout 0
fdcd:5f23:4ab4:11::1 timeout 0

Name: inet6:kube-router-svip-prt
Type: hash:ip,port
Revision: 7
Header: family inet6 hashsize 1024 maxelem 65536 timeout 0 bucketsize 12 initval 0xf31f40be
Size in memory: 808
References: 1
Number of entries: 6
Members:
fdcd:5f23:4ab4:11::10,udp:53 timeout 0
fdcd:5f23:4ab4:11::5402,tcp:1337 timeout 0
fdcd:5f23:4ab4:11::1,tcp:443 timeout 0
fdcd:5f23:4ab4:11::10,tcp:53 timeout 0
fdcd:5f23:4ab4:11::10,tcp:9153 timeout 0
fdcd:0cd1:5dd1:28::1,tcp:30868 timeout 0

ip6tables -L output before creating the service (API server is reachable) 

\Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
KUBE-ROUTER-INPUT  all  --  anywhere             anywhere             /* kube-router netpol - 4IA2OSFRMVNDXBVV */
KUBE-ROUTER-SERVICES  all  --  anywhere             anywhere             /* handle traffic to IPVS service IPs in custom chain */ match-set inet6:kube-router-svip dst

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
KUBE-ROUTER-FORWARD  all  --  anywhere             anywhere             /* kube-router netpol - TEMCG2JMHZYE7H7T */
ACCEPT     all  --  anywhere             anywhere             /* allow outbound node port traffic on node interface with which node ip is associated */
ACCEPT     all  --  anywhere             anywhere             /* allow inbound traffic to pods */
ACCEPT     all  --  anywhere             anywhere             /* allow outbound traffic from pods */

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
KUBE-ROUTER-OUTPUT  all  --  anywhere             anywhere             /* kube-router netpol - VEAAIY32XVBHCSCY */

Chain KUBE-KUBELET-CANARY (0 references)
target     prot opt source               destination         

Chain KUBE-NWPLCY-DEFAULT (2 references)
target     prot opt source               destination         
MARK       all  --  anywhere             anywhere             /* rule to mark traffic matching a network policy */ MARK or 0x10000

Chain KUBE-POD-FW-MCT5QETSQR7LAPG7 (7 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             /* rule for stateful firewall for pod */ ctstate RELATED,ESTABLISHED
DROP       all  --  anywhere             anywhere             /* rule to drop invalid state for pod */ ctstate INVALID
ACCEPT     all  --  anywhere             fdcd:5f23:4ab4:10::2  /* rule to permit the traffic traffic to pods when source is the pod's local node */ ADDRTYPE match src-type LOCAL
KUBE-NWPLCY-DEFAULT  all  --  fdcd:5f23:4ab4:10::2  anywhere             /* run through default egress network policy chain */
KUBE-NWPLCY-DEFAULT  all  --  anywhere             fdcd:5f23:4ab4:10::2  /* run through default ingress network policy chain */
NFLOG      all  --  anywhere             anywhere             /* rule to log dropped traffic POD name:coredns-5cfff596b9-sz64c namespace: kube-system */ mark match ! 0x10000/0x10000 limit: avg 10/min burst 10 nflog-group 100
REJECT     all  --  anywhere             anywhere             /* rule to REJECT traffic destined for POD name:coredns-5cfff596b9-sz64c namespace: kube-system */ mark match ! 0x10000/0x10000 reject-with icmp6-port-unreachable
MARK       all  --  anywhere             anywhere             MARK and 0xfffeffff
MARK       all  --  anywhere             anywhere             /* set mark to ACCEPT traffic that comply to network policies */ MARK or 0x20000

Chain KUBE-ROUTER-FORWARD (1 references)
target     prot opt source               destination         
KUBE-POD-FW-MCT5QETSQR7LAPG7  all  --  anywhere             fdcd:5f23:4ab4:10::2  /* rule to jump traffic destined to POD name:coredns-5cfff596b9-sz64c namespace: kube-system to chain KUBE-POD-FW-MCT5QETSQR7LAPG7 */
KUBE-POD-FW-MCT5QETSQR7LAPG7  all  --  anywhere             fdcd:5f23:4ab4:10::2  PHYSDEV match --physdev-is-bridged /* rule to jump traffic destined to POD name:coredns-5cfff596b9-sz64c namespace: kube-system to chain KUBE-POD-FW-MCT5QETSQR7LAPG7 */
KUBE-POD-FW-MCT5QETSQR7LAPG7  all  --  fdcd:5f23:4ab4:10::2  anywhere             /* rule to jump traffic from POD name:coredns-5cfff596b9-sz64c namespace: kube-system to chain KUBE-POD-FW-MCT5QETSQR7LAPG7 */
KUBE-POD-FW-MCT5QETSQR7LAPG7  all  --  fdcd:5f23:4ab4:10::2  anywhere             PHYSDEV match --physdev-is-bridged /* rule to jump traffic from POD name:coredns-5cfff596b9-sz64c namespace: kube-system to chain KUBE-POD-FW-MCT5QETSQR7LAPG7 */
ACCEPT     all  --  anywhere             anywhere             /* rule to explicitly ACCEPT traffic that comply to network policies */ mark match 0x20000/0x20000

Chain KUBE-ROUTER-INPUT (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             fdcd:5f23:4ab4:11::/112  /* allow traffic to primary/secondary cluster IP range - S45O6WQWYVU3GELQ */
RETURN     tcp  --  anywhere             anywhere             /* allow LOCAL TCP traffic to node ports - LR7XO7NXDBGQJD2M */ ADDRTYPE match dst-type LOCAL multiport dports ndmps:filenet-powsrm
RETURN     udp  --  anywhere             anywhere             /* allow LOCAL UDP traffic to node ports - 76UCBPIZNGJNWNUZ */ ADDRTYPE match dst-type LOCAL multiport dports 30000:filenet-powsrm
RETURN     all  --  anywhere             fdcd:5f23:4ab4:12::/112  /* allow traffic to load balancer IP range: fdcd:5f23:4ab4:12::/112 - PP4T6FE3ON4YJVID */
RETURN     tcp  --  anywhere             anywhere             /* allow LOCAL TCP traffic to node ports - LR7XO7NXDBGQJD2M */ ADDRTYPE match dst-type LOCAL multiport dports ndmps:filenet-powsrm
RETURN     all  --  anywhere             fdcd:5f23:4ab4:12::/112  /* allow traffic to load balancer IP range: fdcd:5f23:4ab4:12::/112 - PP4T6FE3ON4YJVID */
RETURN     tcp  --  anywhere             anywhere             /* allow LOCAL TCP traffic to node ports - LR7XO7NXDBGQJD2M */ ADDRTYPE match dst-type LOCAL multiport dports ndmps:filenet-powsrm
RETURN     all  --  anywhere             fdcd:5f23:4ab4:12::/112  /* allow traffic to load balancer IP range: fdcd:5f23:4ab4:12::/112 - PP4T6FE3ON4YJVID */
KUBE-POD-FW-MCT5QETSQR7LAPG7  all  --  fdcd:5f23:4ab4:10::2  anywhere             /* rule to jump traffic from POD name:coredns-5cfff596b9-sz64c namespace: kube-system to chain KUBE-POD-FW-MCT5QETSQR7LAPG7 */
ACCEPT     all  --  anywhere             anywhere             /* rule to explicitly ACCEPT traffic that comply to network policies */ mark match 0x20000/0x20000

Chain KUBE-ROUTER-OUTPUT (1 references)
target     prot opt source               destination         
KUBE-POD-FW-MCT5QETSQR7LAPG7  all  --  anywhere             fdcd:5f23:4ab4:10::2  /* rule to jump traffic destined to POD name:coredns-5cfff596b9-sz64c namespace: kube-system to chain KUBE-POD-FW-MCT5QETSQR7LAPG7 */
KUBE-POD-FW-MCT5QETSQR7LAPG7  all  --  fdcd:5f23:4ab4:10::2  anywhere             /* rule to jump traffic from POD name:coredns-5cfff596b9-sz64c namespace: kube-system to chain KUBE-POD-FW-MCT5QETSQR7LAPG7 */
ACCEPT     all  --  anywhere             anywhere             /* rule to explicitly ACCEPT traffic that comply to network policies */ mark match 0x20000/0x20000

Chain KUBE-ROUTER-SERVICES (1 references)
target     prot opt source               destination         
ACCEPT     ipv6-icmp --  anywhere             anywhere             /* allow icmp echo requests to service IPs */ ipv6-icmp echo-request
ACCEPT     ipv6-icmp --  anywhere             anywhere             /* allow icmp ttl exceeded messages to service IPs */ ipv6-icmp time-exceeded
ACCEPT     ipv6-icmp --  anywhere             anywhere             /* allow icmp destination unreachable messages to service IPs */ ipv6-icmp destination-unreachable
ACCEPT     all  --  anywhere             anywhere             /* allow input traffic to ipvs services */ match-set inet6:kube-router-svip-prt dst,dst
REJECT     all  --  anywhere             anywhere             /* reject all unexpected traffic to service IPs */ ! match-set inet6:kube-router-local-ips dst reject-with icmp6-port-unreachable

ip6tables -L output after creating the service (API server is unreachable) 

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
KUBE-ROUTER-INPUT  all  --  anywhere             anywhere             /* kube-router netpol - 4IA2OSFRMVNDXBVV */
KUBE-ROUTER-SERVICES  all  --  anywhere             anywhere             /* handle traffic to IPVS service IPs in custom chain */ match-set inet6:kube-router-svip dst

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
KUBE-ROUTER-FORWARD  all  --  anywhere             anywhere             /* kube-router netpol - TEMCG2JMHZYE7H7T */
ACCEPT     all  --  anywhere             anywhere             /* allow outbound node port traffic on node interface with which node ip is associated */
ACCEPT     all  --  anywhere             anywhere             /* allow inbound traffic to pods */
ACCEPT     all  --  anywhere             anywhere             /* allow outbound traffic from pods */

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
KUBE-ROUTER-OUTPUT  all  --  anywhere             anywhere             /* kube-router netpol - VEAAIY32XVBHCSCY */

Chain KUBE-KUBELET-CANARY (0 references)
target     prot opt source               destination         

Chain KUBE-NWPLCY-DEFAULT (2 references)
target     prot opt source               destination         
MARK       all  --  anywhere             anywhere             /* rule to mark traffic matching a network policy */ MARK or 0x10000

Chain KUBE-POD-FW-RFIAS6HOABJQ3KE2 (7 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             /* rule for stateful firewall for pod */ ctstate RELATED,ESTABLISHED
DROP       all  --  anywhere             anywhere             /* rule to drop invalid state for pod */ ctstate INVALID
ACCEPT     all  --  anywhere             fdcd:5f23:4ab4:10::2  /* rule to permit the traffic traffic to pods when source is the pod's local node */ ADDRTYPE match src-type LOCAL
KUBE-NWPLCY-DEFAULT  all  --  fdcd:5f23:4ab4:10::2  anywhere             /* run through default egress network policy chain */
KUBE-NWPLCY-DEFAULT  all  --  anywhere             fdcd:5f23:4ab4:10::2  /* run through default ingress network policy chain */
NFLOG      all  --  anywhere             anywhere             /* rule to log dropped traffic POD name:coredns-5cfff596b9-sz64c namespace: kube-system */ mark match ! 0x10000/0x10000 limit: avg 10/min burst 10 nflog-group 100
REJECT     all  --  anywhere             anywhere             /* rule to REJECT traffic destined for POD name:coredns-5cfff596b9-sz64c namespace: kube-system */ mark match ! 0x10000/0x10000 reject-with icmp6-port-unreachable
MARK       all  --  anywhere             anywhere             MARK and 0xfffeffff
MARK       all  --  anywhere             anywhere             /* set mark to ACCEPT traffic that comply to network policies */ MARK or 0x20000

Chain KUBE-ROUTER-FORWARD (1 references)
target     prot opt source               destination         
KUBE-POD-FW-RFIAS6HOABJQ3KE2  all  --  anywhere             fdcd:5f23:4ab4:10::2  /* rule to jump traffic destined to POD name:coredns-5cfff596b9-sz64c namespace: kube-system to chain KUBE-POD-FW-RFIAS6HOABJQ3KE2 */
KUBE-POD-FW-RFIAS6HOABJQ3KE2  all  --  anywhere             fdcd:5f23:4ab4:10::2  PHYSDEV match --physdev-is-bridged /* rule to jump traffic destined to POD name:coredns-5cfff596b9-sz64c namespace: kube-system to chain KUBE-POD-FW-RFIAS6HOABJQ3KE2 */
KUBE-POD-FW-RFIAS6HOABJQ3KE2  all  --  fdcd:5f23:4ab4:10::2  anywhere             /* rule to jump traffic from POD name:coredns-5cfff596b9-sz64c namespace: kube-system to chain KUBE-POD-FW-RFIAS6HOABJQ3KE2 */
KUBE-POD-FW-RFIAS6HOABJQ3KE2  all  --  fdcd:5f23:4ab4:10::2  anywhere             PHYSDEV match --physdev-is-bridged /* rule to jump traffic from POD name:coredns-5cfff596b9-sz64c namespace: kube-system to chain KUBE-POD-FW-RFIAS6HOABJQ3KE2 */
ACCEPT     all  --  anywhere             anywhere             /* rule to explicitly ACCEPT traffic that comply to network policies */ mark match 0x20000/0x20000

Chain KUBE-ROUTER-INPUT (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             fdcd:5f23:4ab4:11::/112  /* allow traffic to primary/secondary cluster IP range - S45O6WQWYVU3GELQ */
RETURN     tcp  --  anywhere             anywhere             /* allow LOCAL TCP traffic to node ports - LR7XO7NXDBGQJD2M */ ADDRTYPE match dst-type LOCAL multiport dports ndmps:filenet-powsrm
RETURN     udp  --  anywhere             anywhere             /* allow LOCAL UDP traffic to node ports - 76UCBPIZNGJNWNUZ */ ADDRTYPE match dst-type LOCAL multiport dports 30000:filenet-powsrm
RETURN     all  --  anywhere             fdcd:5f23:4ab4:12::/112  /* allow traffic to load balancer IP range: fdcd:5f23:4ab4:12::/112 - PP4T6FE3ON4YJVID */
RETURN     tcp  --  anywhere             anywhere             /* allow LOCAL TCP traffic to node ports - LR7XO7NXDBGQJD2M */ ADDRTYPE match dst-type LOCAL multiport dports ndmps:filenet-powsrm
RETURN     all  --  anywhere             fdcd:5f23:4ab4:12::/112  /* allow traffic to load balancer IP range: fdcd:5f23:4ab4:12::/112 - PP4T6FE3ON4YJVID */
RETURN     tcp  --  anywhere             anywhere             /* allow LOCAL TCP traffic to node ports - LR7XO7NXDBGQJD2M */ ADDRTYPE match dst-type LOCAL multiport dports ndmps:filenet-powsrm
RETURN     all  --  anywhere             fdcd:5f23:4ab4:12::/112  /* allow traffic to load balancer IP range: fdcd:5f23:4ab4:12::/112 - PP4T6FE3ON4YJVID */
KUBE-POD-FW-RFIAS6HOABJQ3KE2  all  --  fdcd:5f23:4ab4:10::2  anywhere             /* rule to jump traffic from POD name:coredns-5cfff596b9-sz64c namespace: kube-system to chain KUBE-POD-FW-RFIAS6HOABJQ3KE2 */
ACCEPT     all  --  anywhere             anywhere             /* rule to explicitly ACCEPT traffic that comply to network policies */ mark match 0x20000/0x20000

Chain KUBE-ROUTER-OUTPUT (1 references)
target     prot opt source               destination         
KUBE-POD-FW-RFIAS6HOABJQ3KE2  all  --  anywhere             fdcd:5f23:4ab4:10::2  /* rule to jump traffic destined to POD name:coredns-5cfff596b9-sz64c namespace: kube-system to chain KUBE-POD-FW-RFIAS6HOABJQ3KE2 */
KUBE-POD-FW-RFIAS6HOABJQ3KE2  all  --  fdcd:5f23:4ab4:10::2  anywhere             /* rule to jump traffic from POD name:coredns-5cfff596b9-sz64c namespace: kube-system to chain KUBE-POD-FW-RFIAS6HOABJQ3KE2 */
ACCEPT     all  --  anywhere             anywhere             /* rule to explicitly ACCEPT traffic that comply to network policies */ mark match 0x20000/0x20000

Chain KUBE-ROUTER-SERVICES (1 references)
target     prot opt source               destination         
ACCEPT     ipv6-icmp --  anywhere             anywhere             /* allow icmp echo requests to service IPs */ ipv6-icmp echo-request
ACCEPT     ipv6-icmp --  anywhere             anywhere             /* allow icmp ttl exceeded messages to service IPs */ ipv6-icmp time-exceeded
ACCEPT     ipv6-icmp --  anywhere             anywhere             /* allow icmp destination unreachable messages to service IPs */ ipv6-icmp destination-unreachable
ACCEPT     all  --  anywhere             anywhere             /* allow input traffic to ipvs services */ match-set inet6:kube-router-svip-prt dst,dst
REJECT     all  --  anywhere             anywhere             /* reject all unexpected traffic to service IPs */ ! match-set inet6:kube-router-local-ips dst reject-with icmp6-port-unreachable

fdcd:0cd1:5dd1:28::1 is the address of the API server.
@k6av k6av added the bug label Jun 29, 2024
@aauren
Copy link
Collaborator

aauren commented Jun 30, 2024

Out of curiosity, what is the address of the kube-apiserver in your kubectl config file? If this is a DNS name, please resolve it first before sending it.

@k6av
Copy link
Author

k6av commented Jun 30, 2024 •
edited
Loading

Hi, thanks for the reply. The API server is addressed with https://[fdcd:0cd1:5dd1:28::]:6443 everywhere including all kubeconfigs, no DNS involved.

@k6av
Copy link
Author

k6av commented Jul 1, 2024 •
edited
Loading

It seems the API server is blocked because the node IP is added to the inet6:kube-router-svip set. Given the name of the set I'd suspect this is not supposed to happen since the node IP is not a service VIP.

@aauren
Copy link
Collaborator

aauren commented Jul 1, 2024

Good sleuthing I got distracted on another problem that I found with ipv6 while I was looking into this one.

I'll try to track down the codepath for that after I see if I can reproduce this.

Out of curiosity, do you see that IP (the one of the node) listed in kubectl describe services -A anywhere?

@k6av
Copy link
Author

k6av commented Jul 1, 2024 •
edited
Loading

Out of curiosity, do you see that IP (the one of the node) listed in kubectl describe services -A anywhere?

The node's IP is listed in the endpoint of the kubernetes service since the node hosts the static control plane pods which have host networking, but nowhere else.

I'll try to track down the codepath for that after I see if I can reproduce this.

I'll see if I can collect some logs from the kube-router pod, if that helps.

@k6av
Copy link
Author

k6av commented Jul 1, 2024 •
edited
Loading

I'm having a hard time getting logs due to the fact that the API server is unreachable, but I've dug up some older logs and noticed this excerpt where kube-router adds the node IP to the inet6:kube-router-svip set.

kube-router log excerpt 1 

I0627 09:33:40.602742    3448 linux_networking.go:298] [tcp]:[10]:[fdcd:5f23:4ab4:12::]:[128]:443 (Flags: ) didn't match any existing IPVS services, creating a new IPVS service
E0627 09:33:40.602809    3448 network_routes_controller.go:300] Failed to enable iptables for bridge. Network policies and service proxy may not work: Sysctl net/bridge/bridge-nf-call-iptables=1 : stat /proc/sys/net/bridge/bridge-nf-call-iptables: no such file or directory (option not found, Does your kernel version support this feature?)
I0627 09:33:40.602848    3448 linux_networking.go:310] Successfully added service: [tcp]:[10]:[fdcd:5f23:4ab4:12::]:[128]:443 (Flags: )
I0627 09:33:40.602869    3448 service_endpoints_sync.go:437] no FW mark found for service, nothing to cleanup
E0627 09:33:40.602872    3448 network_routes_controller.go:306] Failed to enable ip6tables for bridge. Network policies and service proxy may not work: Sysctl net/bridge/bridge-nf-call-ip6tables=1 : stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: no such file or directory (option not found, Does your kernel version support this feature?)
I0627 09:33:40.602881    3448 service_endpoints_sync.go:206] No endpoints detected for service VIP: fdcd:5f23:4ab4:12::, skipping adding endpoints...
I0627 09:33:40.602900    3448 service_endpoints_sync.go:362] no external IP addresses returned for service default:kubernetes skipping...
I0627 09:33:40.602915    3448 service_endpoints_sync.go:64] Setting up NodePort Health Checks for LB services
I0627 09:33:40.602920    3448 network_routes_controller.go:315] Starting network route controller
I0627 09:33:40.602925    3448 nodeport_healthcheck.go:38] Running UpdateServicesInfo for NodePort health check
I0627 09:33:40.602940    3448 nodeport_healthcheck.go:70] Finished UpdateServicesInfo for NodePort health check
I0627 09:33:40.602953    3448 service_endpoints_sync.go:71] Cleaning Up Stale VIPs from dummy interface
I0627 09:33:40.602963    3448 service_endpoints_sync.go:628] Cleaning up if any, old service IPs on dummy interface
I0627 09:33:40.603067    3448 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kube-router/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Authorization: Bearer <masked>" 'https://[fdcd:0cd1:5dd1:28::1]:6443/api/v1/nodes/node01'
I0627 09:33:40.603445    3448 service_endpoints_sync.go:649] Found an IP fe80::c8f7:30ff:fe27:f3b6 which is no longer needed so cleaning up
I0627 09:33:40.603469    3448 linux_networking.go:86] Ignoring link-local IP address: fe80::c8f7:30ff:fe27:f3b6
I0627 09:33:40.603487    3448 service_endpoints_sync.go:78] Cleaning Up Stale VIPs from IPVS
I0627 09:33:40.603768    3448 service_endpoints_sync.go:692] Cleaning up if any, old ipvs service and servers which are no longer needed
I0627 09:33:40.603837    3448 service_endpoints_sync.go:695] Current active service map:
{
      "fdcd:0cd1:5dd1:28::1-tcp-30511": [],
      "fdcd:0cd1:5dd1:28::1-tcp-30967": [],
      "fdcd:5f23:4ab4:11::1-tcp-443": [
          "fdcd:0cd1:5dd1:28::1:6443"
      ],
      "fdcd:5f23:4ab4:11::10-tcp-53": [],
      "fdcd:5f23:4ab4:11::10-tcp-9153": [],
      "fdcd:5f23:4ab4:11::10-udp-53": [],
      "fdcd:5f23:4ab4:11::ddae-tcp-443": [],
      "fdcd:5f23:4ab4:11::ddae-tcp-80": [],
      "fdcd:5f23:4ab4:12::-tcp-443": [],
      "fdcd:5f23:4ab4:12::-tcp-80": []
  }
I0627 09:33:40.604326    3448 service_endpoints_sync.go:85] Cleaning Up Stale metrics
I0627 09:33:40.604351    3448 service_endpoints_sync.go:88] Syncing IPVS Firewall
I0627 09:33:40.604362    3448 network_services_controller.go:628] Attempting to attain ipset mutex lock
I0627 09:33:40.604373    3448 network_services_controller.go:630] Attained ipset mutex lock, continuing...
I0627 09:33:40.605616    3448 ipset.go:595] ipset (ipv6? true) restore looks like:
create TMP-3TF2IQ5MKRP6NAJL hash:ip timeout 0 family inet6
flush TMP-3TF2IQ5MKRP6NAJL
create inet6:kube-router-local-ips hash:ip timeout 0 family inet6
swap TMP-3TF2IQ5MKRP6NAJL inet6:kube-router-local-ips
flush TMP-3TF2IQ5MKRP6NAJL
add TMP-3TF2IQ5MKRP6NAJL fdcd:5f23:4ab4:12:: timeout 0
add TMP-3TF2IQ5MKRP6NAJL fdcd:5f23:4ab4:11::ddae timeout 0
add TMP-3TF2IQ5MKRP6NAJL fdcd:5f23:4ab4:11::10 timeout 0
add TMP-3TF2IQ5MKRP6NAJL fdcd:5f23:4ab4:11::10 timeout 0
add TMP-3TF2IQ5MKRP6NAJL fdcd:0cd1:5dd1:28::1 timeout 0
add TMP-3TF2IQ5MKRP6NAJL fdcd:0cd1:5dd1:28::1 timeout 0
add TMP-3TF2IQ5MKRP6NAJL fdcd:5f23:4ab4:11::ddae timeout 0
add TMP-3TF2IQ5MKRP6NAJL fdcd:5f23:4ab4:11::10 timeout 0
add TMP-3TF2IQ5MKRP6NAJL fdcd:5f23:4ab4:11::1 timeout 0
add TMP-3TF2IQ5MKRP6NAJL fdcd:5f23:4ab4:12:: timeout 0
create inet6:kube-router-svip hash:ip timeout 0 family inet6
swap TMP-3TF2IQ5MKRP6NAJL inet6:kube-router-svip
flush TMP-3TF2IQ5MKRP6NAJL
create TMP-54ALMNP65S4KA7CE hash:ip,port timeout 0 family inet6
flush TMP-54ALMNP65S4KA7CE
add TMP-54ALMNP65S4KA7CE fdcd:5f23:4ab4:12::,tcp:80 timeout 0
add TMP-54ALMNP65S4KA7CE fdcd:5f23:4ab4:11::ddae,tcp:443 timeout 0
add TMP-54ALMNP65S4KA7CE fdcd:5f23:4ab4:11::10,tcp:53 timeout 0
add TMP-54ALMNP65S4KA7CE fdcd:5f23:4ab4:11::10,udp:53 timeout 0
add TMP-54ALMNP65S4KA7CE fdcd:0cd1:5dd1:28::1,tcp:30511 timeout 0
add TMP-54ALMNP65S4KA7CE fdcd:0cd1:5dd1:28::1,tcp:30967 timeout 0
add TMP-54ALMNP65S4KA7CE fdcd:5f23:4ab4:11::ddae,tcp:80 timeout 0
add TMP-54ALMNP65S4KA7CE fdcd:5f23:4ab4:11::10,tcp:9153 timeout 0
add TMP-54ALMNP65S4KA7CE fdcd:5f23:4ab4:11::1,tcp:443 timeout 0
add TMP-54ALMNP65S4KA7CE fdcd:5f23:4ab4:12::,tcp:443 timeout 0
create inet6:kube-router-svip-prt hash:ip,port timeout 0 family inet6
swap TMP-54ALMNP65S4KA7CE inet6:kube-router-svip-prt
flush TMP-54ALMNP65S4KA7CE
destroy TMP-3TF2IQ5MKRP6NAJL
destroy TMP-54ALMNP65S4KA7CE
I0627 09:33:40.605645    3448 ipset.go:225] running ipset command: path=`/usr/sbin/ipset` args=[restore -exist] stdin `​`​`create TMP-3TF2IQ5MKRP6NAJL hash:ip timeout 0 family inet6
flush TMP-3TF2IQ5MKRP6NAJL
create inet6:kube-router-local-ips hash:ip timeout 0 family inet6
swap TMP-3TF2IQ5MKRP6NAJL inet6:kube-router-local-ips
flush TMP-3TF2IQ5MKRP6NAJL
add TMP-3TF2IQ5MKRP6NAJL fdcd:5f23:4ab4:12:: timeout 0
add TMP-3TF2IQ5MKRP6NAJL fdcd:5f23:4ab4:11::ddae timeout 0
add TMP-3TF2IQ5MKRP6NAJL fdcd:5f23:4ab4:11::10 timeout 0
add TMP-3TF2IQ5MKRP6NAJL fdcd:5f23:4ab4:11::10 timeout 0
add TMP-3TF2IQ5MKRP6NAJL fdcd:0cd1:5dd1:28::1 timeout 0
add TMP-3TF2IQ5MKRP6NAJL fdcd:0cd1:5dd1:28::1 timeout 0
add TMP-3TF2IQ5MKRP6NAJL fdcd:5f23:4ab4:11::ddae timeout 0
add TMP-3TF2IQ5MKRP6NAJL fdcd:5f23:4ab4:11::10 timeout 0
add TMP-3TF2IQ5MKRP6NAJL fdcd:5f23:4ab4:11::1 timeout 0
add TMP-3TF2IQ5MKRP6NAJL fdcd:5f23:4ab4:12:: timeout 0
create inet6:kube-router-svip hash:ip timeout 0 family inet6
swap TMP-3TF2IQ5MKRP6NAJL inet6:kube-router-svip
flush TMP-3TF2IQ5MKRP6NAJL
create TMP-54ALMNP65S4KA7CE hash:ip,port timeout 0 family inet6
flush TMP-54ALMNP65S4KA7CE
add TMP-54ALMNP65S4KA7CE fdcd:5f23:4ab4:12::,tcp:80 timeout 0
add TMP-54ALMNP65S4KA7CE fdcd:5f23:4ab4:11::ddae,tcp:443 timeout 0
add TMP-54ALMNP65S4KA7CE fdcd:5f23:4ab4:11::10,tcp:53 timeout 0
add TMP-54ALMNP65S4KA7CE fdcd:5f23:4ab4:11::10,udp:53 timeout 0
add TMP-54ALMNP65S4KA7CE fdcd:0cd1:5dd1:28::1,tcp:30511 timeout 0
add TMP-54ALMNP65S4KA7CE fdcd:0cd1:5dd1:28::1,tcp:30967 timeout 0
add TMP-54ALMNP65S4KA7CE fdcd:5f23:4ab4:11::ddae,tcp:80 timeout 0
add TMP-54ALMNP65S4KA7CE fdcd:5f23:4ab4:11::10,tcp:9153 timeout 0
add TMP-54ALMNP65S4KA7CE fdcd:5f23:4ab4:11::1,tcp:443 timeout 0
add TMP-54ALMNP65S4KA7CE fdcd:5f23:4ab4:12::,tcp:443 timeout 0
create inet6:kube-router-svip-prt hash:ip,port timeout 0 family inet6
swap TMP-54ALMNP65S4KA7CE inet6:kube-router-svip-prt
flush TMP-54ALMNP65S4KA7CE
destroy TMP-3TF2IQ5MKRP6NAJL
destroy TMP-54ALMNP65S4KA7CE
`​`​`
I0627 09:33:40.608281    3448 network_services_controller.go:633] Returned ipset mutex lock
I0627 09:33:40.608312    3448 service_endpoints_sync.go:95] Setting up DSR Services
I0627 09:33:40.608325    3448 service_endpoints_sync.go:605] Setting up policy routing required for Direct Server Return functionality.
I0627 09:33:40.608400    3448 linux_routing.go:30] Did not find iproute2's rt_tables in location /etc/iproute2/rt_tables
I0627 09:33:40.608531    3448 linux_routing.go:30] Did not find iproute2's rt_tables in location /usr/share/iproute2/rt_tables
I0627 09:33:40.611737    3448 service_endpoints_sync.go:610] Custom routing table kube-router-dsr required for Direct Server Return is setup as expected.
I0627 09:33:40.611757    3448 service_endpoints_sync.go:613] Setting up custom route table required to add routes for external IP's.
I0627 09:33:40.611789    3448 linux_routing.go:30] Did not find iproute2's rt_tables in location /etc/iproute2/rt_tables
I0627 09:33:40.611845    3448 linux_routing.go:30] Did not find iproute2's rt_tables in location /usr/share/iproute2/rt_tables
I0627 09:33:40.617435    3448 service_endpoints_sync.go:621] Custom routing table required for Direct Server Return (external_ip) is setup as expected.
I0627 09:33:40.617489    3448 service_endpoints_sync.go:107] IPVS servers and services are synced to desired state
I0627 09:33:40.617505    3448 service_endpoints_sync.go:32] sync ipvs services took 48.173796ms

@aauren
Copy link
Collaborator

aauren commented Jul 5, 2024

Still trying to figure this one out...

If your kube controller node IP is getting added to the inet6:kube-router-svip ipset, that would definitely explain what's going on. Basically, at the end of processing the KUBE-ROUTER-SERVICES chain which is selected by the destination IP being in the inet6:kube-router-svip list, your traffic will be rejected.

Confirming this is happening should be as simple as looking at ip6tables -nvL KUBE-ROUTER-SERVICES and watching the pkts column increase on the reject rule which is the last rule in the chain.

When I tested this locally, it is what I saw:

Before adding the node's ipv6 address to inet6:kube-router-svip manually:

#ip6tables -nvL KUBE-ROUTER-SERVICES
Chain KUBE-ROUTER-SERVICES (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     58   --  *      *       ::/0                 ::/0                 /* allow icmp echo requests to service IPs */ ipv6-icmptype 128
    0     0 ACCEPT     58   --  *      *       ::/0                 ::/0                 /* allow icmp ttl exceeded messages to service IPs */ ipv6-icmptype 3
    0     0 ACCEPT     58   --  *      *       ::/0                 ::/0                 /* allow icmp destination unreachable messages to service IPs */ ipv6-icmptype 1
    0     0 ACCEPT     0    --  *      *       ::/0                 ::/0                 /* allow input traffic to ipvs services */ match-set inet6:kube-router-svip-prt dst,dst
    0     0 REJECT     0    --  *      *       ::/0                 ::/0                 /* reject all unexpected traffic to service IPs */ ! match-set inet6:kube-router-local-ips dst reject-with icmp6-port-unreachable

After adding the controller node's IPv6 address to inet6:kube-router-svip manually and running kubectl get nodes from a kubeconfig file set to the IPv6 address of the kube controller node:

#ip6tables -nvL KUBE-ROUTER-SERVICES
Chain KUBE-ROUTER-SERVICES (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     58   --  *      *       ::/0                 ::/0                 /* allow icmp echo requests to service IPs */ ipv6-icmptype 128
    0     0 ACCEPT     58   --  *      *       ::/0                 ::/0                 /* allow icmp ttl exceeded messages to service IPs */ ipv6-icmptype 3
    2   256 ACCEPT     58   --  *      *       ::/0                 ::/0                 /* allow icmp destination unreachable messages to service IPs */ ipv6-icmptype 1
    0     0 ACCEPT     0    --  *      *       ::/0                 ::/0                 /* allow input traffic to ipvs services */ match-set inet6:kube-router-svip-prt dst,dst
    2   160 REJECT     0    --  *      *       ::/0                 ::/0                 /* reject all unexpected traffic to service IPs */ ! match-set inet6:kube-router-local-ips dst reject-with icmp6-port-unreachable

So the question becomes, then, how does your node's address end up in that set?

After looking into it some more, I noticed that this set (inet6:kube-router-svip) is getting built by the output of: the ipvs services: https://github.com/cloudnativelabs/kube-router/blob/master/pkg/controllers/proxy/network_services_controller.go#L662

This is probably not the best logic, but it has been this way for years. I'm wondering if there might be something else on your system that is creating an IPVS service for you node's IP address?

Can you run ipvsadm -L -n on your controller node and send me the output?

I'm going to assume that fdcd:0cd1:5dd1:28::1 or fdcd:0cd1:5dd1:28:: will show up in that output.

Unfortunately, ipvs services don't have names, so it might be a bit of a guess to figure out what is creating that. But maybe the port will let us know? Specifically, if that port relates to a kubernetes service, then maybe kube-router is somehow errantly creating it? If not, then maybe some other part of your OS or setup is creating it?

@aauren aauren linked a pull request Jul 5, 2024 that will close this issue
Open
@aauren
Copy link
Collaborator

aauren commented Jul 5, 2024 •
edited
Loading

If this is indeed not a kube-router created IPVS service, then the following fix would resolve the issue you're experiencing: #1699

As part of that pipeline, it should build a test container that maybe you could try out as a shortcut to see if it just magically resolves this issue?

The container created from the pipeline was: cloudnativelabs/kube-router-git:PR-1699

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(NSC): ensure kube-router owns kube-router-svip cloudnativelabs/kube-router
2 participants
@aauren @k6av