Merge pull request kubernetes-sigs#631 from carreter/kind-test

Add script to quickly deploy kind cluster with custom KNP image

README.md

@@ -195,6 +195,13 @@ curl -v -p --proxy-key certs/frontend/private/proxy-client.key --proxy-cert cert
 ### Running on kubernetes
 See following [README.md](examples/kubernetes/README.md)
 
+### Running on a local kubernetes cluster with `kind`
+See this [README.md](examples/kind/README.md) for an example that creates a local kubernetes cluster using` kind` and
+deploys the proxy agent on a worker node and the proxy server on a control plane node.
+
+See this [README.md](examples/kind-multinode/README.md) for a similar example that creates a `kind` cluster with a 
+user-configurable number of control plane and worker nodes and optionally sideloads custom proxy agent and server images.
+
 ### Clients
 
 `apiserver-network-proxy` components are intended to run as standalone binaries and should not be imported as a library. Clients communicating with the network proxy can import the `konnectivity-client` module.

examples/kind-multinode/README.md

@@ -0,0 +1,82 @@
+# Set up KIND cluster with multiple KCP and worker nodes running konnectivity
+
+Change to the `examples/kind-multinode-kcp` folder and run `./quickstart-kind`. This script
+performs the following operations:
+
+1. Render config templates in `templates/` using provided values.
+2. Create a new `kind` cluster with the desired number of KCP and worker nodes.
+3. Changes `kubectl` context to point to the new `kind` cluster.
+4. Deploys `konnectivity` proxy servers and agents to the KCP and worker nodes.
+
+
+## `./quickstart-kind.sh` command-line flags
+- `--cluster-name <NAME>`: Name of the `kind` cluster to be created Default: `knp-test-cluster`
+- `--overwrite-cluster`: Overwrite existing `kind` cluster if necessary. Default: do not overwrite.
+- `--server-image <IMAGE_NAME>[:<IMAGE_TAG>]`: Proxy server image to deploy. Default: `gcr.io/k8s-staging-kas-network-proxy/proxy-server:master`
+- `--agent-image <IMAGE_NAME>[:<IMAGE_TAG>]`: Proxy server image to deploy. Default: `gcr.io/k8s-staging-kas-network-proxy/proxy-agent:master`
+- `--num-kcp-nodes <NUM>`: Number of control plane nodes to spin up. Default: 2.
+- `--num-worker-nodes <NUM>`: Number of worker nodes to spin up. Default: 1.
+- `--server-count-override <NUM>`: If this flag is >= 0, override the `--serverCount` flag in the proxy server's configuration to the provided number. Default: set `--serverCount` to equal the number of KCP nodes.
+- `--sideload-images`: Use `kind load ...` to sideload custom proxy server and agent images with the names set by `--server-image` and `--agent-image` into the kind cluster. Default: do not sideload.
+  - Use this if you don't want to publish your custom KNP images to a public registry.
+  - NOTE: You MUST specify an image tag (i.e. `my-image-name:my-image-tag` and not just `my-image-name`) and the image tag MUST NOT be `:latest` for this to work! See [`kind` docs](https://kind.sigs.k8s.io/docs/user/quick-start/#loading-an-image-into-your-cluster) for why this is necessary.
+
+## Example usage to deploy custom local KNP images
+In the repo root, build KNP and its docker images with the following:
+```shell
+make clean
+make certs
+make gen
+make build
+make docker-build
+```
+
+Verify that the new images are available in the local docker registry with `docker images`. Then, bring up the cluster:
+
+```shell
+cd examples/kind-multinode
+
+# These are the default values of the registry, image name, and tag used by the Makefile.
+# Edit them if necessary.
+REGISTRY=gcr.io/$(gcloud config get-value project)
+TAG=$(git rev-parse HEAD)
+TARGET_ARCH="amd64"
+SERVER_IMAGE="$REGISTRY/proxy-server-$TARGET_ARCH:$TAG"
+AGENT_IMAGE="$REGISTRY/proxy-agent-$TARGET_ARCH:$TAG"
+
+# Bring up the cluster!
+./quickstart-kind.sh --cluster-name custom-knp-test --server-image "$SERVER_IMAGE" --agent-image "$AGENT_IMAGE" \
+  --num-kcp-nodes 3 --num-worker-nodes 2 --sideload-images
+```
+
+## Making sure the script worked
+
+Check that the `konnectivity` pods are up and running:
+```shell
+kubectl --namespace kube-system get pods | grep konnectivity
+# Output:
+# konnectivity-agent-4db5j                                 1/1     Running   0          34m
+# konnectivity-agent-c7gj5                                 1/1     Running   0          34m
+# konnectivity-agent-h86l9                                 1/1     Running   0          34m
+# konnectivity-server-9bl45                                1/1     Running   0          34m
+# konnectivity-server-dcfz8                                1/1     Running   0          34m
+# konnectivity-server-klww5                                1/1     Running   0          34m
+# konnectivity-server-nrfz8                                1/1     Running   0          34m
+```
+
+Then create a test pod on a worker node and verify you can get logs from it:
+```shell
+kubectl run test --image httpd:2
+# Output:
+# pod/test created
+kubectl get pods
+# Output:
+# NAME   READY   STATUS    RESTARTS   AGE
+# test   1/1     Running   0          34s
+kubectl logs test
+# Output:
+# AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 10.244.5.3. Set the 'ServerName' directive globally to suppress this message
+# AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 10.244.5.3. Set the 'ServerName' directive globally to suppress this message
+# [Wed Jun 12 20:42:06.471169 2024] [mpm_event:notice] [pid 1:tid 139903660291968] AH00489: Apache/2.4.59 (Unix) configured -- resuming normal operations
+# [Wed Jun 12 20:42:06.471651 2024] [core:notice] [pid 1:tid 139903660291968] AH00094: Command line: 'httpd -D FOREGROUND'
+```

examples/kind-multinode/egress_selector_configuration.yaml

@@ -0,0 +1,15 @@
+apiVersion: apiserver.k8s.io/v1beta1
+kind: EgressSelectorConfiguration
+egressSelections:
+- name: cluster
+  connection:
+    proxyProtocol: GRPC
+    transport:
+      uds:
+        udsName: /etc/kubernetes/konnectivity-server/konnectivity-server.socket
+- name: controlplane
+  connection:
+    proxyProtocol: Direct
+- name: etcd
+  connection:
+    proxyProtocol: Direct

examples/kind-multinode/quickstart-kind.sh

@@ -0,0 +1,121 @@
+#!/bin/bash
+
+set -e
+
+# DEFAULT ARGS
+CLUSTER_NAME="knp-test-cluster"
+AGENT_IMAGE="gcr.io/k8s-staging-kas-network-proxy/proxy-agent:master"
+SERVER_IMAGE="gcr.io/k8s-staging-kas-network-proxy/proxy-server:master"
+NUM_WORKER_NODES=1
+NUM_KCP_NODES=2
+OVERWRITE_CLUSTER=false
+SIDELOAD_IMAGES=false
+SERVER_COUNT_OVERRIDE=-1
+
+# Provide usage info
+usage() {
+  printf "USAGE:\n./quickstart-kind.sh\n\t[--cluster-name <NAME>]\n\t[--server-image <IMAGE_NAME>[:<IMAGE_TAG>]]\n\t[--agent-image <IMAGE_NAME>[:<IMAGE_TAG>]]\n\t[--num-worker-nodes <NUM>]\n\t[--num-kcp-nodes <NUM>]\n\t[--overwrite-cluster]\n"
+}
+
+# ARG PARSING
+VALID_ARGS=$(getopt --options "h" --longoptions "sideload-images,cluster-name:,agent-image:,server-image:,num-worker-nodes:,num-kcp-nodes:,help,overwrite-cluster,server-count-override:" --name "$0" -- "$@") || exit 2
+
+eval set -- "$VALID_ARGS"
+while true; do
+  case "$1" in
+    --cluster-name)
+      CLUSTER_NAME=$2
+      shift 2
+      ;;
+    --agent-image)
+      AGENT_IMAGE=$2
+      shift 2
+      ;;
+    --server-image)
+      SERVER_IMAGE=$2
+      shift 2
+      ;;
+    --num-worker-nodes)
+      NUM_WORKER_NODES=$2
+      shift 2
+      ;;
+    --num-kcp-nodes)
+      NUM_KCP_NODES=$2
+      shift 2
+      ;;
+    --overwrite-cluster)
+      OVERWRITE_CLUSTER=true
+      shift 1
+      ;;
+    --sideload-images)
+      SIDELOAD_IMAGES=true
+      shift 1
+      ;;
+    --server-count-override)
+      SERVER_COUNT_OVERRIDE=$2
+      shift 2
+      ;;
+    --)
+      shift
+      break
+      ;;
+    *|-h|--help)
+      usage
+      exit
+      ;;
+  esac
+done
+
+# RENDER CONFIG TEMPLATES
+echo "Rendering config templates..."
+if [ ! -d rendered ]; then
+  echo "Creating ./rendered"
+  mkdir rendered
+fi
+echo "Adding $NUM_KCP_NODES control plane nodes and $NUM_WORKER_NODES worker nodes to kind.config..."
+cp templates/kind/kind.config rendered/kind.config
+for i in $(seq 1 "$NUM_KCP_NODES")
+do
+ cat templates/kind/control-plane.config >> rendered/kind.config
+done
+for i in $(seq 1 "$NUM_WORKER_NODES")
+do
+ cat templates/kind/worker.config >> rendered/kind.config
+done
+
+SERVER_COUNT=$NUM_KCP_NODES
+if [ "$SERVER_COUNT_OVERRIDE" -ge 0 ]; then
+  echo "Overriding default server count from $NUM_KCP_NODES to $SERVER_COUNT_OVERRIDE"
+  SERVER_COUNT=$SERVER_COUNT_OVERRIDE
+fi
+
+echo "Setting server image to $SERVER_IMAGE, agent image to $AGENT_IMAGE, and --server-count flag on server to $SERVER_COUNT"
+sed -e "s|image: .*|image: $AGENT_IMAGE|" <templates/k8s/konnectivity-agent-ds.yaml >rendered/konnectivity-agent-ds.yaml
+sed -e "s|image: .*|image: $SERVER_IMAGE|" -e "s/--server-count=[0-9]\+/--server-count=$SERVER_COUNT/" <templates/k8s/konnectivity-server.yaml >rendered/konnectivity-server.yaml
+
+
+
+# CLUSTER CREATION
+if [ $OVERWRITE_CLUSTER = true ] && kind get clusters | grep -q "$CLUSTER_NAME"; then
+  echo "Deleting old cluster $CLUSTER_NAME..."
+  kind delete clusters "$CLUSTER_NAME"
+fi
+
+echo "Creating cluster $CLUSTER_NAME..."
+kind create cluster --config rendered/kind.config --name $CLUSTER_NAME
+
+echo "Successfully created cluster. Switching kubectl context to kind-$CLUSTER_NAME"
+kubectl cluster-info --context kind-$CLUSTER_NAME
+
+# SIDELOAD IMAGES IF REQUESTED
+if [ $SIDELOAD_IMAGES = true ]; then
+  echo "Sideloading images into the kind cluster..."
+  kind --name "$CLUSTER_NAME" load docker-image "$SERVER_IMAGE"
+  kind --name "$CLUSTER_NAME" load docker-image "$AGENT_IMAGE"
+fi
+
+# DEPLOY KONNECTIVITY
+echo "Requesting creation of konnectivity proxy servers on cluster $CLUSTER_NAME..."
+kubectl apply -f rendered/konnectivity-server.yaml
+echo "Requesting creation of konnectivity proxy agents on cluster $CLUSTER_NAME..."
+kubectl apply -f rendered/konnectivity-agent-ds.yaml

examples/kind-multinode/templates/k8s/konnectivity-agent-ds.yaml

@@ -0,0 +1,95 @@
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: konnectivity-agent
+  namespace: kube-system
+  labels:
+    kubernetes.io/cluster-service: "true"
+---
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  labels:
+    k8s-app: konnectivity-agent
+  namespace: kube-system
+  name: konnectivity-agent
+spec:
+  selector:
+    matchLabels:
+      k8s-app: konnectivity-agent
+  updateStrategy:
+    type: RollingUpdate
+  template:
+    metadata:
+      labels:
+        k8s-app: konnectivity-agent
+    spec:
+      priorityClassName: system-cluster-critical
+      tolerations:
+        - key: "CriticalAddonsOnly"
+          operator: "Exists"
+        - operator: "Exists"
+          effect: "NoExecute"
+      nodeSelector:
+        kubernetes.io/os: linux
+      dnsPolicy: ClusterFirstWithHostNet
+      containers:
+      - name: konnectivity-agent-container
+        image: gcr.io/k8s-staging-kas-network-proxy/proxy-agent:master
+        resources:
+          requests:
+            cpu: 50m
+          limits:
+            memory: 30Mi
+        command: [ "/proxy-agent"]
+        args: [
+          "--logtostderr=true",
+          "--ca-cert=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt",
+          "--proxy-server-host=konnectivity-server.kube-system.svc.cluster.local",
+          "--proxy-server-port=8091",
+          "--sync-interval=5s",
+          "--sync-interval-cap=30s",
+          "--sync-forever",
+          "--probe-interval=5s",
+          "--service-account-token-path=/var/run/secrets/tokens/konnectivity-agent-token",
+          "--agent-identifiers=ipv4=${HOST_IP}"
+          ]
+        env:
+          - name: POD_NAME
+            valueFrom:
+              fieldRef:
+                fieldPath: metadata.name
+          - name: POD_NAMESPACE
+            valueFrom:
+              fieldRef:
+                fieldPath: metadata.namespace
+          - name: HOST_IP
+            valueFrom:
+              fieldRef:
+                fieldPath: status.hostIP
+        livenessProbe:
+          httpGet:
+            scheme: HTTP
+            port: 8093
+            path: /healthz
+          initialDelaySeconds: 15
+          timeoutSeconds: 15
+        readinessProbe:
+          httpGet:
+            scheme: HTTP
+            port: 8093
+            path: /readyz
+          initialDelaySeconds: 15
+          timeoutSeconds: 15
+        volumeMounts:
+          - mountPath: /var/run/secrets/tokens
+            name: konnectivity-agent-token
+      serviceAccountName: konnectivity-agent
+      volumes:
+      - name: konnectivity-agent-token
+        projected:
+          sources:
+          - serviceAccountToken:
+              path: konnectivity-agent-token
+              audience: system:konnectivity-server