Run multiple KinD clusters on Lima

Demo: https://asciinema.org/a/gZysYfnX0XehRhTnKs8p0BTWS?speed=3&i=2

Prerequisites

This guide now requires MacOS 13.0+ (Ventura).

Download and install the following tools:

• kind → brew install kind

• direnv → brew install direnv

• docker CLI (not docker desktop, just the CLI) → brew install docker

• Only for Apple Silicon Mac (M1/M2): install Rosetta → softwareupdate --install-rosetta --agree-to-license

• Install the konfig plugin for kubectl → kubectl krew install konfig

Additional tools

softwareupdate --install-rosetta xcode-select --install

Getting started

Clone this repo

First, clone the repo:

git clone [email protected]:bcollard/kind-on-lima-public.git
cd kind-on-lima

Take a look at the environment variables in .envrc and change the value of LIMA_WORKDIR to the path of the cloned repo on your machine.

Initialize the env vars for this directory:

direnv allow .

Using the scripts

Most of the scripts are available with Makefile targets.

List all the targets with

# Safely run `make` to list all the targets
make

MacBook preparation

Run the following make target to create a local data dir (LIMA_DATA_DIR env var) on your machine. A few docker images will be saved locally as archives. This may take a few minutes.

make prepare-mac-host
ls -l $LIMA_DATA_DIR

The data dir is mainly used by the docker registries to cache images.

Lima CLI

Two methods here:

• Method #1: install Lima with brew: brew install lima

• Method #2: download the Lima CLI binary with the make target (see below): make dl-install-lima ; then you are prompted for the version to download (ex. "0.14.2"). This will download the lima command-line (limactl) and save it in your LIMA_WORKDIR. Then the Makefile targets will use this binary. This way, the Lima version is pinned and you don’t have to worry about upgrading with brew.

Note	→ In any case, double check that the env vars named LIMA_BIN and LIMACTL_BIN in .envrc are aligned with the method you chose!

Manage the Lima VM

The Lima machine that will be used by the following commands is the one defined in the LIMA_INSTANCE environment variable (see the .envrc file)

Example: LIMA_INSTANCE=vz will use the vz.yaml config file for Lima. And all the following commands will be executed on the VM named vz.

It’s recommanded to use the vz instance (requires MacOS 13.0+) as it’s the fastest one. It’s based on macos Virtualization.framework.
People still using MacOS 12 or lower can use the docker instance which is using QEMU.

# list the Lima machines:
make list-machines
# create (and start) the VM:
make create
# stop the VM:
make stop
# start the VM:
make start
# delete the VM:
make delete

Configure the network E2E

The following command will:

• configure the network on your MacBook machine to allow access to the Lima VM

• pre-configures the Kind network bridge on the Lima VM so that it’s connected to the local docker registries

• configure the network on the Lima VM to allow access from the MacBook host to the KinD clusters

make config-network-end-to-end

Run this command everytime you restart the Lima VM.

Prepare the docker images

make prepare-docker-images

Spin up a new KinD cluster

#make kind-create <id> <name>
make kind-create 1 1-istio
make kind-create 2 2-gloo-edge
make kind-create 3 3-gloo-mesh-mgmt
make kind-create 4 4-gloo-mesh-cluster1
make kind-create 5 5-gloo-mesh-cluster2

The kind-create script takes care of:

• creating a new KinD cluster

• configuring MetalLB so that you can use a LoadBalancer service

• configuring the KinD network bridge to be connected to the local docker registries

Test connectivity E2E

The following command will help you test the connectivity end-to-end.

It will deploy an nginx instance (pod) on the current cluster, expose the nginx service with a Service type LoadBalancer, and then curl the nginx service from the MacBook host.

make test
make clean-test

Manage KinD clusters

# list all the kind clusters:
make kind-list
# delete a specific kind cluster with:
make kind-delete 1-istio
# delete all kind clusters with:
make kind-delete-all

History and WIP

• 2021: Lima v0.8 + QEMU + MetalLB

• 2022: Lima v0.10 + QEMU + MetalLB + registry mirrors + offline mode bootstrap

• 2023: Lima v0.15 + VZ + Cilium CNI + registry mirrors + offline bootstrap + KinD worker nodes