how to use kairos with other kubernetes distributions ?

[xinity]

is there any way to use kairos tooling to deploy another kubernetes distribution like k0s for example, instead of k3s ?

i'd love to try out kairos, but not a real fan (no harm) of k3s :(

let me know,

Cheers,

Rachid

[paynejacob]

Kairos supports other kubernetes distributions however the usage of these distributions is not well documented at the moment. I will make issues to document this more thoroughly and make it easier to use kairos with other distributions but for now I can just give a brief summary of the interface.

There are a few repositories that demonstrate this:

https://github.com/kairos-io/provider-k3s
https://github.com/kairos-io/provider-rke2
https://github.com/kairos-io/provider-kubeadm
https://github.com/kairos-io/provider-microk8s

The k3s provider readme describes the cluster configuration section. This configuration is universal for all cluster providers in Kairos.

https://github.com/kairos-io/provider-k3s

[mudler]

👋

Another way to approach this, if you are willing to spend time to, is to customize Core images https://kairos.io/docs/advanced/customizing/ or use live layers to add k0s https://kairos.io/docs/advanced/livelayering/.

Currently we don't have documentation for SDKs, but the providers are creating images as described in the customization docs with an additional binary that allows to control the Kubernetes distribution. https://github.com/kairos-io/provider-k3s is a good example as it have a very limited scope.

[cmbaron]

How are we meant to compose providers? Do we remaster the standard image to layer on provider-k3s or something else?

edit:
I built a custom image and overlayed both provider-kairos and provider-microk8s, and grabbed a few handy lines I noticed in various Earthfiles:

FROM .. base image per docs

COPY --from=quay.io/kairos/framework:v2.0.0_ubuntu-22-lts / /
COPY --from=quay.io/kairos/core-ubuntu-22-lts-microk8s:v1.25 /opt/microk8s /
COPY --from=quay.io/kairos/core-ubuntu-22-lts-microk8s:v1.25 /system/providers/agent-provider-microk8s /system/providers/agent-provider-microk8s

# Activate Kairos services
RUN systemctl enable cos-setup-reconcile.timer && \
          systemctl enable cos-setup-fs.service && \
          systemctl enable cos-setup-boot.service && \
          systemctl enable cos-setup-network.service

RUN kernel=$(ls /lib/modules | head -n1) && \
    dracut -f "/boot/initrd-${kernel}" "${kernel}" && \
    ln -sf "initrd-${kernel}" /boot/initrd && \
    depmod -a "${kernel}"

# grab provider binary from provider-kairos release
COPY --from=quay.io/kairos/kairos-ubuntu-22-lts:v2.0.0-k3sv1.26.3-k3s1 /system/providers/agent-provider-kairos /system/providers/agent-provider-kairos
COPY --from=quay.io/kairos/kairos-ubuntu-22-lts:v2.0.0-k3sv1.26.3-k3s1 /usr/bin/kairos /usr/bin

The microk8s provider does alright as a proof-of-concept. It needs just a little work to be usable*. Not sure how much work/if at all possible to be a drop-in replacement for k3s as far as the Kairos documentation is concerned.

*Issue pending

[jimmykarily]

Let's document things here: #70