Welcome to Devantler's Homelab 🚀

![NOTE] I am relocating my physical servers, and these are required for my CI pipeline to test new PRs. As such the PRs are failing untill this relocation is done, but the project is very much still alive.

This repo contains the deployment artifacts for Devantler's Homelab. The Homelab is a Kubernetes cluster that is highly automated with the use of Flux GitOps, CI/CD with Automated Testing, and much more. Feel free to look around. You might find some inspiration 🙌🏻

Show/hide folder structure

.
├── .github
│   └── workflows
├── .vscode
├── docs
│   └── images
├── k8s
│   ├── clusters
│   │   ├── homelab-local
│   │   │   ├── apps
│   │   │   ├── flux-system
│   │   │   ├── infrastructure
│   │   │   │   └── controllers
│   │   │   └── variables
│   │   └── homelab-prod
│   │       ├── apps
│   │       ├── flux-system
│   │       ├── infrastructure
│   │       │   ├── controllers
│   │       │   └── gha-runner-scale-sets
│   │       └── variables
│   ├── components
│   │   ├── flux-kustomization-post-build-variables-label
│   │   ├── flux-kustomization-sops-label
│   │   ├── helm-release-crds-label
│   │   ├── helm-release-remediation-label
│   │   └── network-policy-default-deny
│   ├── distributions
│   │   ├── k3s
│   │   │   ├── apps
│   │   │   ├── infrastructure
│   │   │   │   └── controllers
│   │   │   └── variables
│   │   └── talos
│   │       ├── apps
│   │       ├── infrastructure
│   │       │   └── controllers
│   │       │       ├── cilium
│   │       │       ├── kubelet-serving-cert-approver
│   │       │       └── longhorn
│   │       └── variables
│   └── shared
│       ├── apps
│       │   ├── fleetdm
│       │   ├── headlamp
│       │   ├── homepage
│       │   ├── open-webui
│       │   └── plantuml
│       ├── infrastructure
│       │   ├── cloudflared
│       │   ├── controllers
│       │   │   ├── capi-operator
│       │   │   ├── cert-manager
│       │   │   ├── gha-runner-scale-set-controller
│       │   │   ├── goldilocks
│       │   │   ├── k8sgpt-operator
│       │   │   ├── kyverno
│       │   │   ├── metrics-server
│       │   │   ├── reloader
│       │   │   ├── testkube
│       │   │   │   └── crds
│       │   │   ├── traefik
│       │   │   └── trivy-operator
│       │   ├── dex
│       │   ├── harbor
│       │   ├── helm-charts-oci-proxy
│       │   ├── kube-prometheus-stack
│       │   ├── middlewares
│       │   │   ├── basic-auth
│       │   │   └── forward-auth
│       │   ├── oauth2-proxy
│       │   ├── ollama
│       │   └── selfsigned-cluster-issuer
│       ├── tenants
│       └── variables
└── talos
    ├── hetzner
    └── patches
        ├── cluster
        └── nodes

79 directories

Prerequisites

For development:

• Docker
• KSail

For production:

• A Talos Cluster

Note

You can use other distributions as well, but the configuration is optimized for Talos, and thus it is not guaranteed to work with other distributions.

Usage

To run this cluster locally, simply run the following command:

ksail up homelab-local

Note

To run this cluster on your metal, would require that you have access to my SOPS keys. This is ofcourse not possible, so you would need to create your own keys and replace the existing ones, if you want to run my cluster configuration on your own metal.

• The keys that KSail uses are stored in ~/.ksail/age where one Age key is store for each cluster, and named according to the cluster name. For example ~/.ksail/age/homelab-local.
• To update SOPS to work with Ksail, you need to update the .sops.yaml file in the root of the repository, and replace the age keys with your own keys.
• To update the manifests to work with KSail, you need to replace all .sops.yaml files with new ones, that are encrypted with your own keys.

For the production cluster, you would need to do the same, but in addition to storing the keys in ~/.ksail/age, you would also need to store the keys in GitHub Secrets, such that the CI/CD pipeline can provision the keys to the cluster.

Stack

The cluster uses Flux GitOps to reconcile the state of the cluster with single source of truth stored in this repository and published as an OCI image. For development, the cluster is spun up by KSail and for production, the cluster is provisioned by Talos Omni.

The cluster configuration is stored in the k8s/* directories where the structure is as follows:

• clusters/: Contains the the cluster specific configuration for each environment.
• components/: Contains the reusable components that are used across the cluster.
• distributions/: Contains the distribution specific configuration.
• shared/: Contains the shared configuration for all clusters.
  • apps/: Contains the application specific manifests.
    • FleetDM - To provide a device management for my devices. (currently not in use, as it does not support ARM64)
    • Headlamp - To provide a lightweight and extensible Kubernetes UI.
    • Homepage - To provide a dashborad for the cluster.
    • Open WebUI - To provide a web interface and a REST API for interacting with LLM's.
    • PlantUML - To provide a web interface and a REST API for generating PlantUML diagrams.
    • Traefik - To provide an ingress controller for the cluster.
  • custom-resources/: Contains the custom resources that are used across the cluster.
    • Middlewares - Contains the middlewares that are used by Traefik.
    • Selfsigned Cluster Issuer - Contains the selfsigned cluster issuer that is used by Traefik.
  • infrastructure/: Contains the infrastructure specific manifests.
    • Cert Manager - For managing certificates in the cluster.
    • Cloudflared - For tunneling traffic to the cluster.
    • Dex - For providing OIDC authentication for the cluster.
    • Cluster API Operator - For managing the lifecycle of Kubernetes clusters.
    • GitHub Actions Runner Scale Set Controller - To manage GitHub Actions Runner Scale Sets in the cluster.
    • GitHub Actions Runner Scale Sets - To run GitHub Actions in the cluster.
    • Goldilocks - To provide and apply resource recommendations for pods.
    • Harbor - To store and distribute container images.
    • K8sGPT Operator - To analyze the cluster for improvements, vulnerabilities or bugs.
    • Kube Prometheus Stack - To provide monitoring for the cluster. (Prometheus, Grafana, Alertmanager, etc.)
    • Kyverno - To enforce policies in the cluster.
    • Longhorn - To provide distributed storage for the cluster.
    • Metrics Server - To provide metrics for the cluster.
    • OAuth2 Proxy - To provide authentication for the cluster.
    • Ollama - To run LLM's on the cluster.
    • Reloader - To reload deployments when secrets or configmaps change.
    • Testkube - To provide a testing framework for the cluster.
    • Trivy Operator - To analyze the cluster for vulnerabilities.
  • tenants: Contains Flux kustomizations to bootstrap and onboard tenants. (currently not in use)
  • variables/: Contains global variables, that are the same for all clusters.

Kustomize and Flux Kustomization Flow

To support hooking into the kustomize flow for adding or modifying resources for a specific cluster, a specific distribution, or shared across all clusters, the following structure is used:

This means that for every root level kustomization that is applied to the cluster, there should be a corresponding folder in either clusters, distributions, or shared that contains the resources that should be applied to the cluster at that scope. For example, for a root level kustomization in k8s/clusters/<cluster-name>/flux-system/infrastructure.yaml, there should be a corresponding folder in:

• k8s/clusters/<cluster-name>/infrastructure/
• k8s/distributions/<distribution-name>/infrastructure/
• k8s/shared/infrastructure/

Production Environment

Nodes

• 1x Hetzner CAX21 node (QEMU ARM64 4CPU 8Gb RAM 160Gb SSD) for both control plane and worker node
• 2x Hetzner CAX41 node (QEMU ARM64 16CPU 32Gb RAM 320Gb SSD) for both control plane and worker nodes
• 1x Apple Hypervisor ARM64 VM (Running on Mac Mini M2 Pro with access to 32GB RAM and 20 cores (overprovisioned 2/1) as a worker node

Hardware

• Unifi Cloud Gateway - For networking and firewall.
• External Samsung T5/T7 SSD Disks - For distributed storage across the cluster.

Software

• Unifi - For configuring a DMZ zone for my own nodes to run in, along with other security features.
• UTM - For running Kubernetes on Mac Mini via Apple Hypervisor.
• Talos Omni - For provisioning the production cluster, and managing nodes, updates, and the Talos configuration.
• Cloudflare - For etcd backups, DNS, and tunneling all traffic so my network stays private.
• Flux GitOps - For managing the kubernetes applications and infrastructure declaratively.
• SOPS and Age - For encrypting secrets at rest, allowing me to store them in this repository with confidence.
• KSail - For developing the cluster locally, and for running the cluster in CI to ensure all changes are properly tested before being applied to the production cluster.
• K8sGPT - To analyze the cluster for improvements, vulnerabilities or bugs. It integrates with Trivy and Kuverno to also provide security and policy suggestions.

Monthly Cost

Item	No.	Per unit	Total
Hetzner CAX21	3	7,49€	$24,9
Hetzner CAX41	1	29,99€	$33,23
Talos Omni	1	$10	$10
Cloudflare Domains	2	$0,87	$1,74
			$69,87

Star History