[Question]How kube-proxy use yurthub

[MIBc]

/kind question

Can kube-proxy get resources throuth yurthub instead of apiserver？ How does yurthub work for kube-proxy ?

[rambohe-ch]

@MIBc Thank you for filing issue.
Now you need to configure the kube-proxy daemonset like the following setting.

      containers:
      - name: kube-proxy
        command:
        - "/bin/sh"
        - "-c"
        - |
          set -x
          if [ -e /etc/kubernetes/manifests/edge-hub-dp.yaml ]; then
              /usr/local/bin/kube-proxy --master=http://127.0.0.1:10261 --cluster-cidr={{.CIDR}} --hostname-override=$(NODE_NAME) {{if .ProxyMode }} --proxy-mode={{.ProxyMode}} {{end}}
          else
              /usr/local/bin/kube-proxy --kubeconfig=/var/lib/kube-proxy/kubeconfig.conf --cluster-cidr={{.CIDR}} --hostname-override=$(NODE_NAME) {{if .ProxyMode }} --proxy-mode={{.ProxyMode}} {{end}}
          fi

and we plan to solve this challenge in OpenYurt v0.5.0 without configure kube-proxy daemonset by instructing Yurthub endpoint info into pods automatically. if you want to contribute this feature, Please let me know it.

and we also discuss this feature in issue #351 .

[MIBc]

It's a great plan. I'm great to contribute.

[rambohe-ch]

It's a great plan. I'm great to contribute.

Thank you for your response. how about apply for the OpenYurt Member? and you can set me as sponsor.
apply example like this: openyurtio/community#18

[adamzhoul]

same issue here.

anywhere can I find a full solution doc or discussion? @rambohe-ch

kube-proxy or other addons may run in daemonset, we shall only redirect traffic to yurt-hub on edge-node.
this can be a complicated issue when supporting many many more addons besides kube-proxy,
if don't use iptables to redirect traffic.

[DrmagicE]

@adamzhoul You can set --access-server-through-hub=true for yurt-hub and restart kube-proxy. After setting, all traffic in the edge will pass through yurt-hub.

[DrmagicE]

@adamzhoul And if you are not using the docker images building from the latest source code, you may need to add rbac for group system:nodes manually. See #447 (comment)

[adamzhoul]

thanks @DrmagicE

have a quick look at the latest code.
after set --access-server-through-hub=true , MutatedMasterServiceAddr will be set to 127.0.0.1
and filter masterServiceFilter will return 127.0.0.1 as svc kubernetes clusterIP which should be apiServer IP

So, I am still wondering how it affects kube-proxy, if the kube-proxy never visits yurtHub in the first place.

1. kube-proxy reads apiServer address from configmap, and connect to apiServer directly.
2. edge-node deployed at different network-region, so the connecting to apiServer never reached.

Is there any else code I am missing?

[DrmagicE]

@adamzhoul Oh, I miss something. The kube-proxy should be configured to use in-cluster config which will be mutated by the kubelet. So:

1. For question 1, you can just remove the kubeconfig setting in the kube-proxy configmap.
2. For question 2, after setting --access-server-through-hub=true, the in-cluster config for newly created pods will be mutated by kubelet. The pods treat Yurthub (127.0.0.1) as the apiserver, so they don't connect to the real apiserver address.

[adamzhoul]

hi @DrmagicE

as far as I know, using in-cluster config requires rabc config, so kube-proxy can visit the apiServer without certification file.
who will do the config job? should we do it manually?

For question 1, you can just remove the kubeconfig setting in the kube-proxy configmap.

If I remove the kubeconfig setting in the configmap , kube-proxy in cloud-node will be affected too.
so rabc config must be done globally.

For question 2, the in-cluster config for newly created pods will be mutated by kubelet.

Sorry, I don't know much about What will kubelet do to make kube-proxy visit 127.0.0.1.
By put some sort of file inside the container?

[rambohe-ch]

@adamzhoul

1. Kube-proxy and other addons are recommended to use ServiceAccount to auth kube-apiserver, and user usually bind RBAC rights to ServiceAccount beforehand.
2. Support kube-proxy or addons access kube-apiserver through yurthub, we need to solve the following items:
   • KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT env should be configured as yurthub serve address. you can configure the --access-server-through-hub=true for yurthub, these env will be mutated without awareness.
   • ServiceAccount info(bearer token) from kube-proxy or addons can be pass through yurthub to kube-apiserver. so traffic forwarded by yurthub can be authenticated as original pod at kube-apiserver. This item have been solved and we will release new images very soon.

[adamzhoul]

thanks @rambohe-ch
this solved my questions.

after some dig into kubelet code, supply some detail.

1. after config in-cluster mode, client-go use env to connect apiServer

2. env is injected by kubelet

3. kubelet get svc from yurtHub (redirect by update kubelet config )

4. yurtHub add masterServiceFilter to return 127.0.0.1 as svc kubernetes IP

5. finally , kube-proxy pod get kubernetes env value 127.0.0.1

[rambohe-ch]

@DrmagicE @adamzhoul How about configure the default value of --access-server-through-hub to true? so pods on edge nodes will be able to access kube-apiserver without any awareness of cloud-edge difference.

[DrmagicE]

@rambohe-ch I agree.

[adamzhoul]

@rambohe-ch I agree too.

1. from the Architecture picture, it tells us kube-proxy's traffic goes to yurtHub
2. from the description below, it implies the same

[rambohe-ch]

Make sure kube-proxy use InClusterConfig to access kube-apiserver, and kube-proxy will access kube-apiserver through yurthub without configuration. this feature is supported in OpenYurt v0.5.0. the detail info, you can reference the link here: https://openyurt.io/zh/blog/How-run-business-pod-in-edge-scenarios

[hhstu]

If remove the kubeconfig setting in the configmap , kube-proxy on cloud-node can not dial to kube-apiserver after reboot machine. So it is not a good idea !

@adamzhoul @rambohe-ch

[hhstu]

  template:
    metadata:
      creationTimestamp: null
      labels:
        k8s-app: kube-proxy
    spec:
      containers:
      - command:
        - sh
        - -c
        - |
          set -x
          if [ -e /etc/kubernetes/manifests/yurt-hub.yaml ]; then
              /usr/local/bin/kube-proxy --master=http://127.0.0.1:10261 --config=/var/lib/kube-proxy/config.conf  --hostname-override=$(NODE_NAME)
          else
              /usr/local/bin/kube-proxy  --config=/var/lib/kube-proxy/config.conf  --hostname-override=$(NODE_NAME)
          fi

it is not work on edge but cloud

### kube-proxy logs
E0121 07:49:09.213839       6 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.EndpointSlice: failed to list *v1beta1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:node:lc1" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope
E0121 07:50:03.223320       6 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.EndpointSlice: failed to list *v1beta1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:node:lc1" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope
E0121 07:50:48.669400       6 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.EndpointSlice: failed to list *v1beta1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:node:lc1" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope

[rambohe-ch]

If remove the kubeconfig setting in the configmap , kube-proxy on cloud-node can not dial to kube-apiserver after reboot machine. So it is not a good idea !

@hhstu In OpenYurt v0.6.0 version, kube-proxy be able to use the same command parameter on the cloud side or the edge side. but you need to make sure that kube-proxy uses InClusterConfig to access kube-apiserver. the detail info you can reference link here: https://openyurt.io/zh/blog/How-run-business-pod-in-edge-scenarios/