Kopilot

Kopilot is a network tunnel used to proxy Kubernetes API requests to member clusters. Kopilot leverages WebSocket as the underlying connection, secured via HTTPS.

How Does It Work

The kopilot-agent running in a member cluster will first initiate a WebSocket connection to the kopilot-hub running in the host cluster. Then, the WebSocket connection will be multiplexed with Yamux and used as the proxy channel to the Kubernetes API of the member cluster.

Features

• Proxies Kubernetes API requests to multiple member clusters
• Only the host cluster needs to be externally addressable
• Connections are secured and encrypted via HTTPS
• Load-balances member cluster requests when multiple hub-agent connections are available
• Access to member clusters is protected by RBAC rules on the host cluster
• Runs on x86_64 or ARM64

Getting Started

Prerequisites

For the host cluster:

• Kubernetes 1.16+ / minikube / kind
• cert-manager 1.0+

For member clusters:

• Kubernetes 1.16+ / minikube / kind

Installation

First, ensure that cert-manager is installed on the host cluster. If it is not installed yet, you can install it as described in the cert-manager installation documentation. Alternatively, you can simply just run the single command below:

kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.3.1/cert-manager.yaml

Once cert-manager is running, you can now deploy the kopilot-hub on the host cluster:

kubectl apply -f https://github.com/smartxworks/kopilot/releases/download/v0.3.0/kopilot.yaml
export HUB_IP=$(kubectl get nodes -o jsonpath='{.items[0].status.addresses[0].address}')
echo $HUB_IP  # change if this value is incorrect
export HUB_PORT=6443
kubectl create configmap kopilot-hub -n kopilot-system --from-literal=public_addr=$HUB_IP:$HUB_PORT

Usage

First, create a Cluster object in the host cluster to represent one member cluster that needs to be proxied:

kubectl apply -f https://raw.githubusercontent.com/smartxworks/kopilot/master/samples/cluster.yaml

Then, deploy the kopilot-agent on the member cluster:

export HUB_ADDR=$(kubectl get configmap kopilot-hub -n kopilot-system -o jsonpath='{.data.public_addr}')
export MEMBER_NAMESPACE=default
export MEMBER_NAME=sample
export MEMBER_TOKEN=$(kubectl get cluster sample -o jsonpath='{.token}')
export MEMBER_KUBECONFIG=~/.kube/member.config  # change to your member cluster's kubeconfig path
curl -k "https://$HUB_ADDR/apis/subresource.kopilot.smartx.com/v1alpha1/namespaces/$MEMBER_NAMESPACE/clusters/$MEMBER_NAME/agent?token=$MEMBER_TOKEN" | kubectl apply --kubeconfig=$MEMBER_KUBECONFIG -f -

Once the kopilot-agent is running, you can now send Kubernetes API requests to the member cluster from the host cluster with proper RBAC rules:

# create a kubectl pod with proper RBAC rules
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kubectl
  namespace: kopilot-system
---
apiVersion: v1
kind: Pod
metadata:
  name: kubectl
  namespace: kopilot-system
spec:
  serviceAccountName: kubectl
  containers:
    - name: kubectl
      image: bitnami/kubectl
      securityContext:
        runAsUser: 0
        runAsGroup: 0
      command:
        - sleep
        - infinity
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kubectl
rules:
  - apiGroups:
      - subresource.kopilot.smartx.com
    resources:
      - clusters/proxy
    verbs:
      - "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubectl
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kubectl
subjects:
  - kind: ServiceAccount
    name: kubectl
    namespace: kopilot-system
EOF

# get inside the pod
kubectl exec kubectl -n kopilot-system -it -- /bin/bash

# create the member cluster's kubeconfig with service account token
export MEMBER_NAMESPACE=default
export MEMBER_NAME=sample
export MEMBER=${MEMBER_NAMESPACE}_${MEMBER_NAME}
kubectl config set-cluster $MEMBER --server=https://kubernetes.default/apis/subresource.kopilot.smartx.com/v1alpha1/namespaces/$MEMBER_NAMESPACE/clusters/$MEMBER_NAME/proxy --insecure-skip-tls-verify=true
kubectl config set-context $MEMBER --cluster=$MEMBER
kubectl config set-credentials user --token=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
kubectl config set-context $MEMBER --user=user
kubectl config use-context $MEMBER

# get pods of the member cluster
kubectl get pods -A

License

This project is licensed under the Apache-2.0 License. See the LICENSE file for more information.