containerd / nerdctl provider support

[xiaofan-linux]

kind provider already supports podman and docker. What supports containerd?

[BenTheElder]

Nothing. See some discussion here: #1369

What's your use case?

Also to be clear these are used to create the "nodes". KIND runs the Kubernetes pods on containerd inside the nodes always since around v0.2.0 off the top of my head.

[xiaofan-linux]

Nothing. See some discussion here: #1369

What's your use case?

Also to be clear these are used to create the "nodes". KIND runs the Kubernetes pods on containerd inside the nodes always since around v0.2.0 off the top of my head.

If containerd is installed on a virtual machine, can kind call containerd to create a kubernetes cluster? Not calling docker or podman

[BenTheElder]

Currently: No. Why wouldn't you be able to install docker or podman in the VM?

[xiaofan-linux]

Currently: No. Why wouldn't you be able to install docker or podman in the VM?

docker and podman can be installed. Would you like to ask if you consider adding containerd support later?

[BenTheElder]

I haven't heard a compelling reason to maintain this yet. We've found the podman support very time consuming to maintain and the differences between the tools limiting. I think containerd makes sense for running production workloads but for development and test I would still easily recommend Docker instead (or podman if you have reasons against the docker/containerd ecosystem).

It seems unlikely that developer machines and CI environments will switch to not support docker and that's where we target kind. Podman is available for the redhat ecosystem / as a local docker alternative.

I don't yet see any argument other than "just because" and it's already gotten painful trying to get full CI coverage and maintenance for the two existing options, so I don't think is yet more compelling than the expected cost.

[AkihiroSuda]

Supporting containerd via nerdctl (https://github.com/containerd/nerdctl) would be quite straight forward. Almost just alias docker=nerdctl.

[afbjorklund]

@AkihiroSuda :

Supporting containerd via nerdctl (https://github.com/containerd/nerdctl) would be quite straight forward.

But there are lots of subtle little differences between the tools, that makes this fail...

For kind, it fails rather early due to different implementations of --filter and --format:

$ kind create cluster
using docker due to KIND_EXPERIMENTAL_PROVIDER
ERROR: failed to create cluster: node(s) already exist for a cluster with the name "kind"

nerdctl ps -a --filter "label=io.x-k8s.kind.cluster=kind" --format "{{.Names}}"

FATA[0000] unknown flag: --filter

nerdctl.lima ps -a --filter "label=io.x-k8s.kind.cluster=kind" --format "{{.Names}}"

CONTAINER ID    IMAGE                              PLATFORM       COMMAND    CREATED       STATUS                   PORTS    NAMES
0f9dbfcb95f9    docker.io/arm64v8/alpine:latest    linux/amd64    "arch"     6 days ago    Exited (0) 6 days ago             
34582c470b5f    docker.io/amd64/alpine:latest      linux/amd64    "arch"     6 days ago    Exited (0) 6 days ago             

So even if both podman and nerdctl try to provide a "drop-in replacement" for docker:

This would need a separate provider implementation, to handle all the subtle little quirks ?

[dimetron]

separate provider just because of incomplete features on nerdctl sounds wrong direction.
if docker is alias for lima nerdctl we should have the fix on nerdctl side
containerd/nerdctl#349

[BenTheElder]

Conceptually yes, nerdctl should aim for drop-in compatibility to help users migrate, and that would be great for everyone, but kind interacts with more internals and API surface than most CLI users I think.

Similarly podman was also initially at least advertised as drop-in compatible, but that turns out to be rather difficult because the API surface of the docker CLI is so large. E.G. --format involves exposing the results's go structs to the caller but there's often no other way to collect the information we're requesting.

Then there's complex leaky or intentionally user-facing implementation details like networking, I think realistically containerd + nerdctl is not going to be able to ship 100% docker compatible networking, e.g. the built in dockerd DNS resolver in non-default networks (like "kind") is not something so trivially added, it requires some network hackery and a resolver running inside a host daemon.

This will probably realistically be another provider if implemented. Including blank lines, boilerplate headers, and comments, the podman provider is only ~1500 lines currently and the docker provider about 1700 lines (EDIT: and they're currently more verbose than they need to be, I think a provider can be <1000 lines, but getting one right is still complex and provider specific due to those leaky details). There are some other bits required in e.g. cgroups handling in the node image, but that's much less code, though perhaps trickier.

Support, CI, and understanding what the differences are is going to be the expensive part, not so much the additional codepaths / named provider.

[BenTheElder]

Currently nerdctl fails the hardcoded IsAvailable() checks that look for "Docker version" or "podman version".

Yeah, unfortunately we had to start sniffing for this string in the output because we had such a headache with docker in PATH actually being a podman binary which is NOT compatible (see e.g. above comment) and needs it's own codepath.

And as you found, if we didn't have this check, it would just fail later anyhow on similar incompatibilities.

I'd be interested to see someone lay out what is missing or just different between docker and nerdctl + containerd and discussing how kind might approach this (e.g. CNI networking will probably be an interesting one), at which point we can look at writing or accepting PRs to implement this (see also: https://kind.sigs.k8s.io/docs/contributing/getting-started/).

[afbjorklund]

It can probably be made to work.

Just saying that thisalias docker=podman or alias docker=nerdctl is a marketing gimmick, and also ignores the compatibility and performance implications of changing from regular containers to rootless containers... (which can be done for Docker too)

Hopefully there is some "base class" of functionality that can be shared, and then just override the specifics later ?

We have the same situation when it comes to the CRI inside the cluster, but kind opted out of that food fight. 🤡

In the end, it all has to be tested out.

[AkihiroSuda]

I'd be interested to see someone lay out what is missing or just different between docker and nerdctl + containerd and discussing how kind might approach this (e.g. CNI networking will probably be an interesting one), at which point we can look at writing or accepting PRs to implement this (see also: https://kind.sigs.k8s.io/docs/contributing/getting-started/).

Docker	nerdctl (as of v0.14.0)	Usage in kind
docker inspect --type=image	nerdctl image inspect	kind/pkg/cluster/internal/providers/docker/images.go Line 56 in 0abda0b cmd := exec.Command("docker", "inspect", "--type=image", image)
Label "com.docker.network.bridge.enable_ip_masquerade"	(N/A)	kind/pkg/cluster/internal/providers/docker/network.go Line 151 in 0abda0b "-o", "com.docker.network.bridge.enable_ip_masquerade=true",
Label "com.docker.network.driver.mtu"	(N/A)	kind/pkg/cluster/internal/providers/docker/network.go Line 154 in 0abda0b args = append(args, "-o", fmt.Sprintf("com.docker.network.driver.mtu=%d", mtu)) kind/pkg/cluster/internal/providers/docker/network.go Line 166 in 0abda0b "-f", `{{ index .Options "com.docker.network.driver.mtu" }}`)
docker network create --ipv6	(N/A)	kind/pkg/cluster/internal/providers/docker/network.go Line 157 in 0abda0b args = append(args, "--ipv6", "--subnet", ipv6Subnet)
docker network ls --filter=FILTER	(parse and filter the JSON output manually)	kind/pkg/cluster/internal/providers/docker/network.go Line 240 in 0abda0b "--filter=name=^"+regexp.QuoteMeta(name)+"$", kind/pkg/cluster/internal/providers/docker/network.go Line 256 in 0abda0b "--filter=name=^"+regexp.QuoteMeta(name)+"$",
docker ps --filter=FILTER	(parse and filter the JSON output manually)	kind/pkg/cluster/internal/providers/docker/provider.go Line 120 in 0abda0b "--filter", fmt.Sprintf("label=%s=%s", clusterLabelKey, cluster),
Label "desktop.docker.io/ports/%d/tcp", for docker run -p without explicit host port number (?)	(N/A; nerdctl run -p needs explicit host port number)	kind/pkg/cluster/internal/providers/docker/provider.go Line 179 in 0abda0b "{{ index .Config.Labels \"desktop.docker.io/ports/%d/tcp\" }}", common.APIServerInternalPort,
docker run --restart=on-failure:1	(N/A; workaround would be to use --restart=always)	kind/pkg/cluster/internal/providers/docker/provision.go Line 166 in 0abda0b "--restart=on-failure:1",

[BenTheElder]

This came up again in #kind slack.

---

Re: #2317 (comment)

Thanks, seems similar to podman in that mostly filter commands would require something else entirely.

Lack of explicit host port makes remote runtime slightly more problematic, and restart=always isn't exactly ideal ... (see code comments on why it is failure:1)

Seems like it would require detecting nerdctl and having a seperate codepath, like podman, which is doable, if someone wants to propose it (and also what CI might look like ...)

Some discussion probably needed around networking as well (which is moreso what I was looking for, along with things like the restart limitation).

It would certainly be neat if they became drop-in compatible, but that seems like an awful lot of work on the nerdctl end, and potentially prone to divergence. With podman we've had much more luck doing an explicit provider, but, it's been more time consuming to support than hoped to support even then.

[James-Riordan]

I could be off base here, you all seem to be quite knowledgeable on the history of these OCI runtimes, but I think support for Containerd instead of Docker by default would be a worthwhile step- assuming you can get rid of the need for a running Daemon such as Docker. Docker is like the tiller of Helm2.0. There's root access granted. And as someone whose used kind to build a near-prod identical setup (with the exception of just raw kubeadm aka no kind in live clusters), I would find it extremely valuable to be able to lock down and virtualize the last bit of local kubernetes development by spinning kind up with containerd instead of Docker

[afbjorklund]

Running rootless should be orthogonal to trying to run without dockerd (which also happens to run containerd, internally) ?

it's been more time consuming to support than hoped to support even then.

We have the same experience in minikube, supporting podman and cri-o in addition to docker and containerd is taking a lot of effort and doesn't add very much for the user. The difference between a container node and a real node is hard enough, I think. Then having to write everything three times to cater for the differences in syntax and then testing the different configurations...

It would be interesting to hear more about the perceived benefits of containerd-in-containerd, over containerd-in docker ?

Previously it was more that people wanted to avoid Docker Desktop, for licensing reasons (both Open Source and cost)

[BenTheElder]

I could be off base here, you all seem to be quite knowledgeable on the history of these OCI runtimes, but I think support for Containerd instead of Docker by default would be a worthwhile step- assuming you can get rid of the need for a running Daemon such as Docker.

containerd still uses a daemon, Kubernetes uses multiple daemons. daemons are useful.

you can use kind with podman to avoid running a container engine daemon on the host, but the support is not fully baked and the capabilities are not the same, notably kind clusters backed by podman will not have reboot support in part due to the lack of a daemon.

Docker is like the tiller of Helm2.0. There's root access granted.

Docker supports running "rootless" about as well as any other container engine, FWIW.

You can also not grant users other than root access to the docker socket.

And as someone whose used kind to build a near-prod identical setup (with the exception of just raw kubeadm aka no kind in live clusters), I would find it extremely valuable to be able to lock down and virtualize the last bit of local kubernetes development by spinning kind up with containerd instead of Docker

kind runs containerd within the nodes for the Kubernetes container runtime, but I'm not sure you'll actually find any user experience improvements versus using docker on the host. containerd is great for production runtime, but docker is more suitable as a developer tool.

[zgfh]

According to the description of https://github.com/containerd/nerdctl#motivation
I think nerdctl will be more and more different from docker in the future
I think add a provider for nerdctl maybe maybe is a beteer way

[BenTheElder]

I agree that if nerdctl is supported it's probably best done with an explicit provider instead of expecting it to 100% match docker.

I don't think this something we're currently planning. We've explicitly left room for this sort of thing to be possible, however at this time we're focused on bugfixes and maintenance, not features.

Podman support has already been .... rather time consuming, and fragmented capabilities. The return on investment here is questionable and not something we can likely afford to focus on in the near future. While nerdctl provides interesting opportunities for experimenting with containers and containerd, anywhere you can run containerd you can run dockerd (including lima etc) and I'd still recommend dockerd for development containers.

Kubernetes workloads (or simple containers, KIND is ... not so much) should be portable to other runtimes, of course.

[afbjorklund]

I don't think this something we're currently planning. We've explicitly left room for this sort of thing to be possible, however at this time we're focused on bugfixes and maintenance, not features.

I think I came to a similar conclusion with the singularity provider, while technically possible it didn't offer so many benefits to the user - that mostly wanted to run with Docker Desktop (or Podman Desktop).

[zvonkok]

/cc @zvonkok

[BenTheElder]

containerd/nerdctl#349 (comment)

[yankay]

HI dears.

This is the proposal to support nerdctl in the kind. Would you please review it and give comments :-)

The Goal

• support nerdctl in the kind
• less code as possible to reduce the maintain work like podman (reuse the docker provider)

The work items

• Fix the nerdctl issues , to make kind can create the cluster with nerdctl. Almost Done.
  • nerdctl doesn't work with kind containerd/nerdctl#349 (comment)
  • v1.7 planning containerd/nerdctl#2600
• minimal CI up : [WIP] add nerdctl ci minimal support #3408
• add a nerdctl provider just swap docker => nerdctl, follow by [WIP]Add ipv6 error message check to support the nerdctl #3396 (comment)
• Rootless mode
• More test , fix the other bugs
  • nerdctl ps -a --filter label=io.x-k8s.kind.cluster --format '{{.Label "io.x-k8s.kind.cluster"}}' behaviour is inconsistent with docker containerd/nerdctl#2598
  • Cannot load docker-image with Nerdctl #3427
• Docs: add experimetal support for nerdctl

Relate work/disscussion before: #3396 (comment)

And I will try to fix the bugs ( nerdctl and kind ) in the next.