From dcd4757450c6a46da821f244479ef83e35f75641 Mon Sep 17 00:00:00 2001 From: Ross Fairbanks Date: Sun, 24 Mar 2024 20:17:29 +0100 Subject: [PATCH 1/2] docs: Add docs for cluster infra dev with tofu Signed-off-by: Ross Fairbanks --- .../cluster-infra-development.md | 87 +++++++++++ docs/infrastructure/create_equinix_cluster.md | 137 ------------------ 2 files changed, 87 insertions(+), 137 deletions(-) create mode 100644 docs/infrastructure/cluster-infra-development.md delete mode 100644 docs/infrastructure/create_equinix_cluster.md diff --git a/docs/infrastructure/cluster-infra-development.md b/docs/infrastructure/cluster-infra-development.md new file mode 100644 index 0000000..8e4e5c8 --- /dev/null +++ b/docs/infrastructure/cluster-infra-development.md @@ -0,0 +1,87 @@ +# Cluster infra development + +## Overview + +Our cluster runs on physical servers from Equinix Metal contributed to CNCF. + +- We use OpenTofu to manage the cluster infra and an S3 bucket from AWS to store the tofu state. +- To manage components running in the cluster we use Flux. + +## Pre-Requisites + +- Install the [tofu](https://opentofu.org/docs/intro/install/) CLI +- Fork the tooling repo https://github.com/cncf-tags/green-reviews-tooling +- Get access to the `Green Reviews Cluster Dev` vault in the TAG ENV 1Password account (please post in the #tag-env-wg-green-reviews channel in CNCF Slack so we have tracking for your request) + +## Setup + +- In a local copy of your fork change to the working directory + +```sh +cd infrastructure/equinix-metal +``` + +- Set env vars for tofu from 1Password with AWS and Equinix credentials + +```sh +export AWS_ACCESS_KEY_ID="*****" +export AWS_SECRET_ACCESS_KEY="*****" +export TF_VAR_equinix_auth_token="*****" +export TF_VAR_equinix_project_id="*****" +export TF_VAR_k3s_token="*****" +``` + +- Set env vars for your fork including a GitHub PAT for bootstrapping Flux + +```sh +export TF_VAR_cluster_name="green-reviews-dev" +export TF_VAR_flux_github_user="*** Your GitHub user ***" +export TF_VAR_flux_github_repo="green-reviews-tooling" +export TF_VAR_flux_github_token="*** Your GitHub PAT ***" +export TF_VAR_flux_github_branch="*** Your branch ***" +export TF_VAR_ssh_public_key="*** Your SSH public key ***" +``` + +- Check tofu workspaces + +```sh +tofu workspace list + default +* dev +``` + +- Ensure you are using the **dev** workspace + +```sh +tofu workspace select dev +``` + +## Making changes + +Follow the usual tofu workflow. See [core workflow](https://opentofu.org/docs/intro/core-workflow/) +for more details. + +- Make changes +- Initialize tofu + +```sh +tofu init +``` + +- Review changes + +```sh +tofu plan +``` + +- Test changes - **make sure you are using dev workspace** + +```sh +tofu apply +``` + +- Cleanup - **make sure you are using dev workspace** + +```sh +tofu destroy +``` diff --git a/docs/infrastructure/create_equinix_cluster.md b/docs/infrastructure/create_equinix_cluster.md deleted file mode 100644 index a43b2be..0000000 --- a/docs/infrastructure/create_equinix_cluster.md +++ /dev/null @@ -1,137 +0,0 @@ -# Create Equinix Metal cluster using CAPI / CAPEM - -## Overview - -- We are using machines contributed to the CNCF by Equinix Metal. -- These steps manually create a cluster using CAPI / CAPEM using a local -Kind cluster for the management cluster. -- These docs are based on the Equinix CAPI [guide](https://deploy.equinix.com/developers/guides/kubernetes-cluster-api/). -- Using [Podman](https://github.com/kubernetes-sigs/cluster-api-provider-packet/issues/496) -is recommended by Equinix for MacOS. -- We plan to automate these steps later using GitHub Actions and an IaC tool. - -## Create management Cluster - -- If you don't have a management cluster running create it with -```sh -kind create cluster -``` - -- Check that it's running locally - -```sh -kubectl cluster-info --context kind-kind -``` - -## Create Cluster - -- Log in to the Equinix Metal [console](https://console.equinix.com/) and create a new project. Get -the project API key from the project settings. If it doesn't exist create it. - -```sh -export PACKET_API_KEY="" -``` - -- Create a project ssh-key with `` - -- Install the CAPI controllers using [clusterctl](https://cluster-api.sigs.k8s.io/user/quick-start.html#install-clusterctl). - -```sh -clusterctl init --infrastructure packet -``` - -- Set env vars with cluster config. - -```sh -# Get the project ID from the project settings in the console -export PROJECT_ID="" - -# Use Paris metro (Equinix region) -export METRO="pa" - -# Use Ubuntu 22.04 with cGroup v2 for Kepler -export NODE_OS="ubuntu_22_04" - -# The pod and service CIDRs for the new cluster -export POD_CIDR="192.168.0.0/16" -export SERVICE_CIDR="172.26.0.0/16" - -# Use node type with Intel CPUs for RAPL support -export CONTROLPLANE_NODE_TYPE="m3.small.x86" -export WORKER_NODE_TYPE="m3.small.x86" - -# SSH key to use for access to nodes -export SSH_KEY="" - -# Kubernetes version to install -export KUBERNETES_VERSION="v1.28.2" -``` - -- Generate cluster manifests. - -```sh -clusterctl generate cluster wg-green-reviews \ - --kubernetes-version $KUBERNETES_VERSION \ - --control-plane-machine-count=1 \ - --worker-machine-count=1 \ - > wg-green-reviews.yaml -``` - -- Apply cluster manifests. - -```sh -kubectl apply -f wg-green-reviews.yaml -``` - -- Get kubeconfig and store it securely. - -```sh -clusterctl get kubeconfig wg-green-reviews > wg-green-reviews.kubeconfig -``` - -- Set `KUBECONFIG` env var so following commands are run on the cluster. - -```sh -export KUBECONFIG=wg-green-reviews.kubeconfig -``` - -- Install Cilium as CNI. - -```sh -helm repo add cilium https://helm.cilium.io/ -helm install cilium cilium/cilium --version 1.14.2 --namespace kube-system -``` - -- SSH to each cluster node and ensure Kepler dependencies are installed -(user is named `root`). - -```sh -apt update -apt install -y linux-headers-$(uname -r) -apt install -y linux-modules-extra-$(uname -r) -modprobe intel_rapl_common -``` - -- Install Kepler. - -```sh -helm repo add kepler https://sustainable-computing-io.github.io/kepler-helm-chart -helm install kepler kepler/kepler --namespace kepler --create-namespace -``` - -- Check Kepler container metrics are non-zero. - -```sh -kubectl exec -ti -n kepler daemonset/kepler \ - -- bash -c "curl localhost:9102/metrics" | grep 'kepler_container_package_joules_total' -``` - -## Delete Cluster - -- If the Kind cluster still exists it can be used to delete the Equinix cluster. - -```sh -kubectl delete cluster wg-green-reviews -``` - -- Otherwise delete both servers and the elastic IP via the Equinix console. From 9282291f00fe1f2042e6308de3b5c1f8f02195d1 Mon Sep 17 00:00:00 2001 From: Ross Fairbanks Date: Tue, 26 Mar 2024 14:30:26 +0100 Subject: [PATCH 2/2] Create dev workspace if it doesn't exist Signed-off-by: Ross Fairbanks --- docs/infrastructure/cluster-infra-development.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/infrastructure/cluster-infra-development.md b/docs/infrastructure/cluster-infra-development.md index 8e4e5c8..d9bf5b5 100644 --- a/docs/infrastructure/cluster-infra-development.md +++ b/docs/infrastructure/cluster-infra-development.md @@ -46,10 +46,17 @@ export TF_VAR_ssh_public_key="*** Your SSH public key ***" ```sh tofu workspace list + default * dev ``` +- If the dev workspace doesn't exist create it. + +```sh +tofu workspace new dev +``` + - Ensure you are using the **dev** workspace ```sh