Install services

.gitignore

@@ -1 +1,2 @@
+manifests
 tmp

Guide/1-Install cluster.md

@@ -32,8 +32,9 @@ In order to prevent situation when Raspberry Pi will get different IP on each bo
 
 1. Under this guide find [./config](./config) folder and apply following changes:
    - [ ] Change file names to your MAC addresses you have noted in previous step. Keep order, each config file contain comment to note which Rasperry Pi node number is it.
-   - [ ] Set your own ssh public key under `ssh_authorized_keys` ( NOTE: github shortcut notation didn't work for me )
-   - [ ] Update `ntp_servers` might be some public ntp server or if your router provide own use that one
+   - [ ] Set your own ssh public key under `ssh_authorized_keys` ( NOTE: github shortcut notation didn't work for me ).
+   - [ ] Update `ntp_servers` might be some public ntp server or if your router provide own use that one.
+   - [ ] Update `boot_cmd` to reflect your zone.
 2. Run master image build, please note that it take some time to fetch dependencies so go take coffee or stare at something ⏳.
 
        docker run -e TARGET=raspberrypi -v $PWD/config:/app/config -v $PWD/deps:/app/deps -v $PWD/out:/app/out -v /dev:/dev --privileged docker.io/elmariofredo/picl-k3os-image-generator:v0.2
@@ -42,10 +43,10 @@ In order to prevent situation when Raspberry Pi will get different IP on each bo
 
 4. Get kubeconfig file and verify that master is up and running
 
-       ssh rancher@MASTER_1_IP sudo cat /etc/rancher/k3s/k3s.yaml | sed 's/127.0.0.1/MASTER_1_IP/g' >! ~/.kube/config
-       export KUBECONFIG=~/.kube/config
-       kubectl get nodes
-       #> n1     Ready    master   20s   v1.18.6+k3s1
+     ssh rancher@MASTER_1_IP sudo cat /etc/rancher/k3s/k3s.yaml | sed 's/127.0.0.1/MASTER_1_IP/g' > ./tmp/kube_config.yml
+     export KUBECONFIG=./tmp/kube_config.yml
+     kubectl get nodes
+     #> n1     Ready    master   20s   v1.18.6+k3s1
 
 5. Get join token from master node
 
@@ -56,7 +57,7 @@ In order to prevent situation when Raspberry Pi will get different IP on each bo
 1. Update `server_url` under [config](./config) folder for each worker
 2. Update `token` under [config](./config) folder for each worker
 3. Build image using same command
-    
+
        docker run -e TARGET=raspberrypi -v $PWD/config:/app/config -v $PWD/deps:/app/deps -v $PWD/out:/app/out -v /dev:/dev --privileged docker.io/elmariofredo/picl-k3os-image-generator:v0.2
 
 4. Burn image to rest MicroSDHC using Raspberry Pi Imager and put it into rest of Raspberry Pi.

Guide/2-Install services.md

@@ -0,0 +1,18 @@
+# Install services
+
+All services are setup using [Kustomize](https://github.com/kubernetes-sigs/kustomize) defined in [Sources](../Sources) folder.
+
+## Deploy
+
+1. Update values in
+   1. Change domain in [grafana-chart-values.yml](../Sources/monitoring-system/grafana/grafana-chart-values.yml)
+   2. Add digitalocean.com DNS token to [../tmp/dnstoken](../tmp/dnstoken)
+        access-token=YOUR_TOKEN
+      If you use different DNS provider change [issuer.yml](../Sources/cert-manager/issuer.yml) see docs for more informations https://cert-manager.io/docs/configuration/acme/dns01/
+   3. Change loadbalancer IP in [metallb](../Sources/metallb-system/configs/config) and then also in ingress [controller](../Sources/ingress-nginx/kustomization.yml)
+   4. Change email in [cert-manager](../Sources/cert-manager/issuer.yml)
+2. Run [manifests.sh](../manifests.sh) file. In case that you will run into CRD nonexistent error run command again. 
+
+## Verify
+
+Log into your grafana dashboard using [adminpass](../tmp/adminpass) credentials. After login you should see 'Pi k3s Simple Dashboard', look around there are few other dashboard preinstalled for you 😉.

README.md

@@ -8,7 +8,19 @@ Because I hated our bathroom floor heating thermostat and wanted something bette
 
 ## Why Cloud Native? Isn't it bit overhead?
 
-Maybe 😉, running K8s cluster to power simple cron job for triggering heating on and off may seem like overhead 🥴. But in the another way running so critical application like home heating on some single Raspberry Pi using undocumented scripts, is not good recipe for working marriage 🤣. Most importantly Kubernetes is my daily bread and I wanted home project where I can try new technologies and approaches.
+Maybe 😉, running K8s cluster to power simple cron job for triggering heating on and off may seem like overhead 🥴. But in the another way running so critical application like home heating on some single Raspberry Pi using undocumented scripts, is not good recipe for happy family life. Most importantly Kubernetes is my daily bread and I wanted home project, where I can try new technologies and approaches.
+
+## What
+
+- Manager nodes [Raspberry Pi 4](https://www.raspberrypi.org/products/raspberry-pi-4-model-b/) 4GB
+- OS [K3OS](https://github.com/rancher/k3os) build using [picl-k3os-image-generator](https://github.com/elmariofredo/picl-k3os-image-generator)
+- Scheduler [K3s](https://github.com/rancher/k3s)
+- Loadbalancer [Metallb](Sources/metallb-system)
+- Ingress [NGINX Ingress Controller](Sources/ingress-nginx)
+- Monitoring 
+  - [Grafana](Sources/monitoring-system/grafana)
+  - [Node exporter](Sources/monitoring-system/node-exporter)
+  - [VictoriaMetrics Operator](Sources/monitoring-system/victoriametrics)
 
 ## How does it work?
 
@@ -19,6 +31,6 @@ TODO
 Fork and clone this repo https://github.com/elmariofredo/cnt and follow this guide divided into several steps.
 
 1. [Install cluster](./Guide/1-Install%20cluster.md)
-2. [Install base services]() TODO
+2. [Install base services](./Guide/2-Install%20services.md)
 3. [Install thermostat services]() TODO
 4. [Build thermostat]() TODO

Sources/cert-manager/README.md

@@ -0,0 +1,4 @@
+# Cert Manager
+
+- Git: https://github.com/jetstack/cert-manager
+- Kustomize: https://github.com/jetstack/cert-manager/releases

Sources/cert-manager/issuer.yml

@@ -0,0 +1,16 @@
+apiVersion: cert-manager.io/v1
+kind: ClusterIssuer
+metadata:
+  name: ingress
+spec:
+  acme:
+    email: [email protected]
+    server: https://acme-v02.api.letsencrypt.org/directory
+    privateKeySecretRef:
+      name: issuer-account-key
+    solvers:
+    - dns01:
+        digitalocean:
+          tokenSecretRef:
+            name: digitalocean-dns
+            key: access-token

Sources/cert-manager/kustomization.yml

@@ -0,0 +1,18 @@
+# https://github.com/jetstack/cert-manager/releases
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+namespace: cert-manager
+
+resources:
+- https://github.com/jetstack/cert-manager/releases/download/v1.0.2/cert-manager.yaml
+- issuer.yml
+
+generatorOptions:
+  disableNameSuffixHash: true
+
+secretGenerator:
+- name: digitalocean-dns
+  envs: 
+  - ../../tmp/dnstoken
+

Sources/ingress-nginx/README.md

@@ -0,0 +1,4 @@
+# Nginx Ingress Controller
+
+- Git: https://github.com/kubernetes/ingress-nginx
+- Kustomize: https://github.com/kubernetes/ingress-nginx/tree/master/deploy/static/provider/cloud

Sources/ingress-nginx/kustomization.yml

@@ -0,0 +1,18 @@
+# https://github.com/kubernetes/ingress-nginx/tree/master/deploy/static/provider/cloud
+resources:
+- https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.34.1/deploy/static/provider/cloud/deploy.yaml
+
+patchesJson6902:
+- target:
+    version: v1
+    kind: Service
+    name: ingress-nginx-controller
+    namespace: ingress-nginx
+  patch: |-
+    - op: add
+      path: "/metadata/annotations"
+      value:
+        metallb.universe.tf/address-pool: ingress
+    - op: add
+      path: "/spec/loadBalancerIP"
+      value: 10.236.127.155

Sources/metallb-system/README.md

@@ -0,0 +1,6 @@
+# Metallb
+
+We use metallb for automatically assigning IP to active node, in case of node failure IP is attached to next working node.
+
+- Git: https://github.com/metallb/metallb
+- Kustomize: https://github.com/metallb/metallb/tree/main/manifests

Sources/metallb-system/configs/config

@@ -0,0 +1,5 @@
+address-pools:
+- name: ingress
+  protocol: layer2
+  addresses:
+  - 10.236.127.155/32

Sources/metallb-system/configs/secretkey

@@ -0,0 +1 @@
+Aish]ea9Shai

Sources/metallb-system/kustomization.yml

@@ -0,0 +1,18 @@
+# https://github.com/metallb/metallb/tree/main/manifests
+namespace: metallb-system
+
+resources:
+- github.com/metallb/metallb/manifests?ref=v0.9.3
+
+configMapGenerator:
+- name: config
+  files:
+  - configs/config
+
+secretGenerator:
+- name: memberlist
+  files:
+  - configs/secretkey
+
+generatorOptions:
+ disableNameSuffixHash: true

Sources/monitoring-system/grafana/README.md

@@ -0,0 +1,4 @@
+# Grafana
+
+- Git: https://github.com/grafana/grafana
+- HELM Chart: https://github.com/grafana/helm-charts/tree/main/charts/grafana

Sources/monitoring-system/grafana/grafana-chart-values.yml

@@ -0,0 +1,72 @@
+# https://github.com/grafana/grafana/blob/master/conf/defaults.ini
+grafana.ini:
+  server:
+    domain: graf.vejlupek.org
+    root_url: "%(protocol)s://%(domain)s/"
+    serve_from_sub_path: false
+  dashboards:
+    default_home_dashboard_path: /var/lib/grafana/dashboards/default/overview.json
+  analytics:
+    reporting_enabled: false
+    check_for_updates: false
+
+ingress:
+  enabled: true
+  annotations:
+    cert-manager.io/cluster-issuer: "ingress"
+  tls:
+  - secretName: grafana-tls
+    hosts:
+    - graf.vejlupek.org
+  hosts:
+  - "graf.vejlupek.org"
+  path: "/"
+
+testFramework:
+  enabled: false # Until https://github.com/bats-core/bats-core/issues/356 is resolved
+
+admin:
+  existingSecret: "adminpass"
+  userKey: adminuser
+  passwordKey: adminpass
+
+datasources:
+  datasources.yaml:
+    apiVersion: 1
+    datasources:
+    - name: VictoriaMetrics
+      type: prometheus
+      url: http://vmselect-vmcluster-persistent.monitoring-system.svc.cluster.local:8481/select/0/prometheus/
+      isDefault: true
+
+dashboards:
+  default:
+    overview:
+      gnetId: 13043
+      revision: 1
+      datasource: VictoriaMetrics
+    victoriametrics:
+      gnetId: 11831
+      revision: 6
+      datasource: VictoriaMetrics
+    pod-metrics:
+      gnetId: 13025
+      revision: 1
+      datasource: VictoriaMetrics
+    node-exporter:
+      gnetId: 1860
+      revision: 21
+      datasource: VictoriaMetrics
+
+dashboardProviders:
+  dashboardproviders.yaml:
+    apiVersion: 1
+    providers:
+    - name: 'default'
+      orgId: 1
+      folder: ''
+      type: file
+      disableDeletion: false
+      editable: false
+      options:
+        path: /var/lib/grafana/dashboards/default

Sources/monitoring-system/grafana/grafana-chart.yml

@@ -0,0 +1,14 @@
+# https://github.com/grafana/helm-charts/tree/main/charts/grafana
+apiVersion: helm.kustomize.mgoltzsche.github.com/v1
+kind: ChartRenderer
+metadata:
+  name: grafana
+  namespace: monitoring-system
+# repository: ./
+repository: https://grafana.github.io/helm-charts # https://github.com/grafana/helm-charts
+# chart: ./Sources/monitoring-system/grafana/
+# chart: file:///workdir/Sources/monitoring-system/grafana/grafana
+chart: grafana
+version: 5.6.7
+valueFiles:
+- grafana-chart-values.yml

Sources/monitoring-system/grafana/kustomization.yml

@@ -0,0 +1,14 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+generatorOptions:
+  disableNameSuffixHash: true
+
+secretGenerator:
+- name: adminpass
+  envs: 
+  - ../../../tmp/adminpass
+
+generators:
+- grafana-chart.yml
+

Sources/monitoring-system/kustomization.yml

@@ -0,0 +1,9 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+namespace: monitoring-system
+
+resources:
+- victoriametrics
+- node-exporter
+- grafana

Sources/monitoring-system/node-exporter/README.md

@@ -0,0 +1,4 @@
+# Node exporter
+
+- Git: https://github.com/prometheus/node_exporter
+- HELM Chart: https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-node-exporter

Sources/monitoring-system/node-exporter/kustomization.yml

@@ -0,0 +1,5 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+generators:
+- node-exporter-chart.yml

Sources/monitoring-system/node-exporter/node-exporter-chart.yml

@@ -0,0 +1,9 @@
+# https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-node-exporter
+apiVersion: helm.kustomize.mgoltzsche.github.com/v1
+kind: ChartRenderer
+metadata:
+  name: node-exporter
+  namespace: monitoring-system
+repository: https://prometheus-community.github.io/helm-charts
+chart: prometheus-node-exporter
+version: 1.11.2

Sources/monitoring-system/victoriametrics/README.md

@@ -0,0 +1,4 @@
+# VistoriaMetrics Operator
+
+- Git: https://github.com/VictoriaMetrics/operator
+- Kustomize: https://github.com/VictoriaMetrics/operator/tree/master/config/default

Sources/monitoring-system/victoriametrics/kubelet-svc.yml

@@ -0,0 +1,24 @@
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: kubelet
+  namespace: monitoring
+  annotations:
+    prometheus.io/scrape: "true"
+  labels:     
+    app: prometheus-node-exporter
+    heritage: Helm
+    release: eit
+    chart: prometheus-node-exporter-1.10.0
+    jobLabel: node-exporter
+spec:
+  type: ClusterIP
+  ports:
+    - port: 9100
+      targetPort: 9100
+      protocol: TCP
+      name: metrics
+  selector:
+    app: prometheus-node-exporter
+    release: eit

Sources/monitoring-system/victoriametrics/kustomization.yml

@@ -0,0 +1,19 @@
+# https://github.com/VictoriaMetrics/operator/tree/master/config/default
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+- github.com/victoriametrics/operator/config/default?ref=master
+- vmagent-view-metrics-rbac.yml
+- vmagent.yml
+- vmcluster.yml
+- vmservicescrape-kubelet.yml
+- vmservicescrape-metrics-server.yml
+- vmservicescrape-node-exporter.yml
+
+images:
+- name: victoriametrics/operator
+  newTag: docker-multiarch-manifest
+
+patches:
+- manager.patch.yml

Sources/monitoring-system/victoriametrics/manager.patch.yml

@@ -0,0 +1,15 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  labels:
+    control-plane: vm-operator
+  name: vm-operator
+  namespace: monitoring-system
+spec:
+  template:
+    spec:
+      containers:
+      - name: manager
+        env:
+        - name: VM_VMAGENTDEFAULT_CONFIGRELOADIMAGE
+          value: quay.io/coreos/prometheus-config-reloader:v0.42.0