Unable to load docker desktop containerd managed images to cluster #3795
Comments
iamvinov-atlassian
added
the
kind/bug
|
More information. Unticking: "Use containerd for pulling and storing images" in Docker Desktop actually resolves this. |
|
That error message is coming from ctr when we ask it to import the image saved from docker. Unfortunately I can't run docker desktop at work, will have to find another way to reproduce this. Can you look at the same image exported the way kind does with docker save, or provide a tarball from that somewhere? That would speed things up (can replicate the rest with kind load image-archive) this sounds like a containerd/docker bug but we need to confirm how before contacting them. The part kind is doing could be bugged but is pretty straightforward once we decide we need to load the image because it's not already available |
BenTheElder
changed the title
kind cluster
|
Could be a small sample image like busybox if you can confirm the bug still applies to that image and share the containerd vs dockerd mode versions that would speed things up. Otherwise it may be difficult to reproduce due to the licensing of the application and/or my employer's policies, I'll have to see if this is something I can replicate in some other way. |
|
I'm hitting what seems to be the same issue in kuttl's integration tests. Kuttl embeds kind, currently v0.25.0. Interestingly this works on CI (GHA, ubuntu 20.04 runner) but on my desktop this fails with the same message as for @iamvinov-atlassian What I've been able to figure out using
So it seems like from the PoV of I'm running: |
|
@BenTheElder you might be able to reproduce this with: Unfortunately the actual error is hidden as |
|
We came across similar in our environment, we're not using docker desktop but have configured Docker to use containerd as the image store per these docs. I'm not sure if this is exactly the same as what @porridge reports, as it doesn't involve an attestation blob. However I can simply reproduce this with: Versions:Note this does need Docker that has multi-platform support, so I don't see this on Docker 24 on another system. I've not checked all versions but definitely it happens with Docker 27 on the host. For the nginx image this happens because there's a reference to something for platform 386, which in some cases matches amd64 (containerd's platform matching code). Therefore a workaround is: Obviously that's not ideal, because #2957 wanted So that means |
|
Do we know why this works? Is it potentially fetching references from a remote (and therefore broken airgapped)? |
|
@BenTheElder It's not fetching from a remote cc @AkihiroSuda -- could you help decide if this is a containerd bug or a docker issue? i.e. is docker generating a bad OCI image or is containerd mishandling it? In order to reproduce this all that is needed is a Docker instance configured to use containerd (CE edition is fine), then run There are three potential issues here:
(2 is why |
Seems to be a bug of containerd. |
|
We should circle back to see if it's still a bug after upgrading, I am working on shipping v1.7.24 (current 1.7.x) |
|
At HEAD the default node image is on Kubernetes 1.32.0 + containerd 1.7.24, there are a lot of changes in containerd and I wonder if any of them fixed the bug. #3768 tracks containerd 2.0, that one is a bigger can of worms for our users and might be a bit, but also AIUI the fix isn't necessarily in 2.0 if not in 1.7, it's just that this particular issue can be avoided by using the transfer service, but we should also report a bug to containerd if it's still present in current releases without using the transfer service. |
@BenTheElder do I need to upgrade anything on my workstation to test whether the issue is fixed in my case? Or should it be enough to bump the version of kind to current HEAD? 🤔 Because just using the current kind snapshot is not helping: |
It's the node image at HEAD, so if you use the default image then bumping to HEAD would test it, but if you're setting the node image to use then you'd have to change that. Thanks for testing. |
|
xref #3828 (comment) |
|
FWIW we will be moving to the transfer API hopefully (since we prefer to use defaults and plan to upgrade to containerd 2.0), pending a fix to We should probably still report a bug with details to containerd. #3795 (comment) I haven't had time to locally reproduce and I'd appreciate if one of you would file with containerd, thanks! |
FTR Unfortunately I don't really understand the pieces involved so I'm not able to formulate a report in containerd terms. And I'm a bit swamped with other work so no time to learn this. |
|
We're working on upgrading to containerd 2.0.2 (after a 2.x fix related to image import landed), which uses the transfer service by default and I think may fix this. |
|
In the meantime if anyone hits this: recommend opting out of containerd managed images at least until the containerd 2.x upgrade is available, though you could instead try #3805 (which should be obviated by the 2.x upgrade) |
|
At HEAD / the latest commit we're using containers 2.x with the default import transport. Can someone with this issue test it? You can checkout the repo and run make build or make install to get a binary. |
@BenTheElder this didn't seem to help.
|
Yes, the default node image was updated at HEAD, so as long as you're using the default it should have containerd 2.x (this should also be visible in the node info with something like That's unfortunate, FWIW when we import an image, aside from detecting if the image needs importing, we're just asking docker to save it, and containerd to import the saved image. So the implementations are primarily in those projects. I would expect |
|
NOTE: I cannot directly repro myself due to the licensing changes https://www.theregister.com/2021/08/31/docker_desktop_no_longer_free/ (not permitted at my employer) So I will need help identifying a fix. That said, I think we'll most likely require a patch in containerd or docker desktop, and at most just updating containerd in kind to a patched release. |
|
@BenTheElder I don't think you need docker desktop to reproduce this. 🤔 I'm facing it using the moby engine provided by Fedora 41, which I think is rather strict w.r.t. licensing: Should be possible to do it in a VM... |
|
ACK, I'll try to get to this ... In the meantime if anyone has time, we can ignore kind and focus on |
|
@BenTheElder I tried the following and I think I succeeded with reproducing it in the end: |
|
Thank you! ... subscribing |
|
I'm having a similar issue (I think) since Everything works fine in |
|
@lucacome that's something else, you're using an unsupported image, please see the docs about images (and the release notes) as well as the recent issue #3853 (basically we're mid transition to containerd 2.0, there's some issues with it in testing kubernetes discovered since that are pending a patch upstream, and I have limited bandwidth at the moment so we haven't actually released |
|
Sorry, I didn't think to search the closed issues. I actually didn't know there could be compatibility issues with images. We've always updated to the latest image available for years, and fortunately this is the first time we run into any problems, but we probably have to rethink this. Thanks! |
We try very hard not to break things, but sometimes we have to deal with ecosystem changes (in this case containerd 1.x => 2.x has some changes) and it becomes unreasonable to guarantee no issues, we might need to reiterate this more strongly in https://kind.sigs.k8s.io/docs/user/quick-start/#creating-a-cluster:~:text=This%20will%20bootstrap,a%20kind%20release. in addition to the release notes warnings. (Similarly, Kubernetes makes breaking changes to bootstrapping, so while kind tries to isolate from that, we can't always guarantee that a new image with a version that didn't exist at the time of the kind release still works without changes in kind, it's just been unusually stable for the past year or two, there are more breaking changes from the ecosystem coming like #3847 ...) |
|
Hi, Docker Engine maintainer here, let me give some context on how it works on the Docker side. With the containerd image store integration enabled, Docker uses the digest of the image manifest index as the image's ID. If you pull an image with Docker, by default it pulls only the native platform blobs, but still persist the whole image index to persist the image identity (to keep stable image IDs). However, since in most cases you don't pull all possible platforms for the image, a tarball from On containerd side, such image can still be loaded, but it requires passing an extra However, there's a way out of this on the Docker side too. |
|
Thanks @vvoland! Will try to follow up more soon ... KEP season. Quickly:
I think we'd actually like to preserve the ID if possible too, though I think that's currently not working either (it might after this change?) #2394
WithSkipMissing seems like what we'd like. If we can't get it in ctr we could start using our own client, but then we have to break things a lot more, and I had hoped our relatively minimal exposure to ctr would be reasonable.
That makes sense, at least for a quick workaround users can |
What happened:
I am attempting to load Docker images into
kindcluster:kind load docker-image busybox -n nebulaebut getting the following error:❯ kind load docker-image busybox -n nebulae Image: "busybox" with ID "sha256:5b0f33c83a97f5f7d12698df6732098b0cdb860d377f6307b68efe2c6821296f" not yet present on node "nebulae-control-plane", loading... ERROR: failed to load image: command "docker exec --privileged -i nebulae-control-plane ctr --namespace=k8s.io images import --all-platforms --digests --snapshotter=overlayfs -" failed with error: exit status 1 Command Output: ctr: content digest sha256:83e82a8dd385e27d95f2118c1332d414684aa665552f7f837f86da33674308c4: not foundWhat you expected to happen:
I expected the image to load successfully. I already have this image pulled locally using
docker pull busybox. Upon further investigation, it seems to me thatkindor (containerd) expects the image for all platforms to be present on the host for theloadcommand to succeed.How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?:
From looking at other answers on the internet, it seems generally this error occurs when the image arch doesn't match the host arch. But this is not the case. I did perform
docker images --treeand made sure the images match my host (M3 MacBook Pro) OS.Environment:
kind v0.25.0 go1.23.3 darwin/arm64docker info,podman infoornerdctl info):MacOS Sonoma v14.7kubectl version):The text was updated successfully, but these errors were encountered: