Skip to content

Latest commit

 

History

History
204 lines (164 loc) · 8.07 KB

README.md

File metadata and controls

204 lines (164 loc) · 8.07 KB

Kubernetes The Hard Way The Easy Way

Creates a Single Controller Kubernetes cluster using multipass and cilium

All scripts are available to learn how it is built.

Specs

  • Ubuntu 22.04
  • Kubernetes 1.28.4
  • etcd 3.5.10
  • containerd 1.7.10
  • cni plugins 1.4.0
  • cilium 1.14.4 (via helm chart)
  • coredns 1.28.2 (via helm chart)

Requirements

  • helm 3+
    • linux: here
    • mac: brew install helm
  • curl
    • linux: sudo apt install curl
    • mac: brew install curl
  • ipcalc
    • linux: sudo apt install ipcalc
    • mac: brew install ipcalc
  • multipass
    • linux: sudo snap install multipass --classic
    • mac: brew cask install multipass
  • cfssl and cfssljson

    • linux:

      wget -q --show-progress --https-only --timestamping \
  https://storage.googleapis.com/kubernetes-the-hard-way/cfssl/linux/cfssl \
  https://storage.googleapis.com/kubernetes-the-hard-way/cfssl/linux/cfssljson
chmod +x cfssl cfssljson
sudo mv cfssl cfssljson /usr/local/bin/

    • mac: brew install cfssl

  • kubectl

    • linux:

      wget https://storage.googleapis.com/kubernetes-release/release/v1.18.5/bin/linux/amd64/kubectl
chmod +x kubectl
sudo mv kubectl /usr/local/bin/

    • mac:

      curl -o kubectl https://storage.googleapis.com/kubernetes-release/release/v1.18.5/bin/darwin/amd64/kubectl
chmod +x kubectl
sudo mv kubectl /usr/local/bin/

Procedure

  1. Create your machines:

    ./setup.sh

  2. Wait a couple of minutes for cilium and coredns to start working

    $  kubectl get all --all-namespaces
NAMESPACE     NAME                                   READY   STATUS    RESTARTS   AGE
kube-system   pod/cilium-25lnk                       1/1     Running   0          73s
kube-system   pod/cilium-d8p59                       1/1     Running   0          74s
kube-system   pod/cilium-operator-67f9b484d8-ksvzj   1/1     Running   0          74s
kube-system   pod/cilium-operator-67f9b484d8-mqvsq   1/1     Running   0          74s
kube-system   pod/coredns-coredns-67c645769c-klwlk   1/1     Running   0          73s
kube-system   pod/hubble-relay-85664d47c4-ftd5n      1/1     Running   0          74s
kube-system   pod/hubble-ui-7c6789876c-7b9mh         3/3     Running   0          74s

NAMESPACE     NAME                      TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)         AGE
default       service/kubernetes        ClusterIP   172.17.0.1    <none>        443/TCP         2m31s
kube-system   service/coredns-coredns   ClusterIP   172.17.0.10   <none>        53/UDP,53/TCP   73s
kube-system   service/hubble-relay      ClusterIP   172.17.0.83   <none>        80/TCP          74s
kube-system   service/hubble-ui         ClusterIP   172.17.0.35   <none>        80/TCP          74s

NAMESPACE     NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
kube-system   daemonset.apps/cilium   2         2         2       2            2           <none>          74s

NAMESPACE     NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/cilium-operator   2/2     2            2           74s
kube-system   deployment.apps/coredns-coredns   1/1     1            1           73s
kube-system   deployment.apps/hubble-relay      1/1     1            1           74s
kube-system   deployment.apps/hubble-ui         1/1     1            1           74s

NAMESPACE     NAME                                         DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/cilium-operator-67f9b484d8   2         2         2       74s
kube-system   replicaset.apps/coredns-coredns-67c645769c   1         1         1       73s
kube-system   replicaset.apps/hubble-relay-85664d47c4      1         1         1       74s
kube-system   replicaset.apps/hubble-ui-7c6789876c         1         1         1       74s

  3. Your cluster is ready, lets verify data encryption works

    $ kubectl create secret \
  generic kubernetes-the-hard-way --from-literal="mykey=mydata"
$ multipass exec controller-k8s -- sudo ETCDCTL_API=3 etcdctl get \
  --endpoints=https://127.0.0.1:2379 --cacert=/etc/etcd/ca.pem \
  --cert=/etc/etcd/kubernetes.pem --key=/etc/etcd/kubernetes-key.pem \
  /registry/secrets/default/kubernetes-the-hard-way | hexdump -C
00000000  2f 72 65 67 69 73 74 72  79 2f 73 65 63 72 65 74  |/registry/secret|
00000010  73 2f 64 65 66 61 75 6c  74 2f 6b 75 62 65 72 6e  |s/default/kubern|
00000020  65 74 65 73 2d 74 68 65  2d 68 61 72 64 2d 77 61  |etes-the-hard-wa|
00000030  79 0a 6b 38 73 3a 65 6e  63 3a 61 65 73 63 62 63  |y.k8s:enc:aescbc|
00000040  3a 76 31 3a 6b 65 79 31  3a 61 bb c0 45 f2 df 88  |:v1:key1:a..E...|
00000050  36 46 05 df c1 df 26 e1  e0 59 18 9f 7d 51 7a d9  |6F....&..Y..}Qz.|
00000060  28 0d 03 4e c3 14 55 01  51 d6 aa cc 50 21 a5 09  |(..N..U.Q...P!..|
00000070  86 92 89 9b 33 82 43 09  7d 5b fe bb 68 45 43 48  |....3.C.}[..hECH|
00000080  96 9a 1e a8 88 30 82 a8  2c d8 26 ea 12 19 58 da  |.....0..,.&...X.|
00000090  3a 25 ed 6b 47 1f e2 e9  31 91 e6 cf 64 bb 19 41  |:%.kG...1...d..A|
000000a0  fe 2b 7a 86 a8 be e4 c0  b6 98 2e dc 96 92 58 92  |.+z...........X.|
000000b0  c4 6b c1 85 a9 d0 ec d6  03 2d c7 2c 14 f5 da 03  |.k.......-.,....|
000000c0  ef c6 c9 2b bc 26 9c 36  ab 0c da 08 f2 8b 79 c7  |...+.&.6......y.|
000000d0  12 98 55 5f 4c 56 f7 fd  e1 71 45 16 a3 59 01 76  |..U_LV...qE..Y.v|
000000e0  97 5b d1 cc 91 92 c5 d9  05 0a                    |.[........|
000000ea

    The etcd key should be prefixed with k8s:enc:aescbc:v1:key1, which indicates the aescbc provider was used to encrypt the data with the key1 encryption key.

  4. Deployments as they are described on Kubernetes the hard way will work

    $ kubectl create deployment nginx --image=nginx
deployment.apps/nginx created

    List the pod created

    $ kubectl get pods -l app=nginx
NAME                    READY   STATUS    RESTARTS   AGE
nginx-8f458dc5b-86vs4   1/1     Running   0          36s

  5. NodePort service will work in the following way (Depends on nginx deployment described in step 4)

    $ kubectl expose deployment nginx --port 80 --type NodePort
$ NODE_PORT=$(kubectl get svc nginx \
  --output=jsonpath='{range .spec.ports[0]}{.nodePort}')
$ WORKER_IP=$(multipass info 'worker-1-k8s' | grep 'IPv4' | \
  awk '{ print $2 }')
$ curl -I "http://${WORKER_IP}:${NODE_PORT}"
HTTP/1.1 200 OK
Server: nginx/1.25.3
Date: Thu, 07 Dec 2023 05:28:35 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Tue, 24 Oct 2023 13:46:47 GMT
Connection: keep-alive
ETag: "6537cac7-267"
Accept-Ranges: bytes

  6. You've done a kubernetes!

How to see hubble-ui

  1. Forward hubble-ui port

    POD_NAME=$(kubectl get pods -n kube-system -l k8s-app=hubble-ui \
  -o jsonpath="{.items[0].metadata.name}")
kubectl port-forward -n kube-system ${POD_NAME} 8080:8081

  2. In your browser go to http://localhost:8080 -> pick a namespace with pods. Example below:

    cilium-hubble-ui

Troubleshooting

All nodes should be able to reach each other via hostname

01-config-files/distribute-config-files.sh generates multipass-hosts and later the bootstrap scripts append it to /etc/hosts on the controllers and workers

Related links