Added self-signed cert CA, still failing to pull images

[madeddie]

I used the functionality created here #1654, the test of running
ctr -a /run/host-containerd/containerd.sock image pull ourreg/image:tag in a sheltie shell works fine, but k8s itself still can't pull the image and still logs the same complaint.

Turning off trust for the bundle breaks the ctr image pull in the sheltie session, so it seems those 2 containerd's are using different CA paths or something?

Here I start with the bundle not trusted to show the pull failing, then setting it trusted and it succeeds

bash-5.0# ctr -a /run/host-containerd/containerd.sock i pull artifactory.service.internal/some-image:bla
INFO[0000] trying next host                              error="failed to do request: Head \"https://artifactory.service.internal/v2/some-image/manifests/bla\": x509: certificate signed by unknown authority" host=artifactory.service.internal
ctr: failed to resolve reference "artifactory.service.internal/some-image:bla": failed to do request: Head "https://artifactory.service.internal/v2/some-image/manifests/bla": x509: certificate signed by unknown authority
bash-5.0# apiclient set pki.my-trusted-bundle.trusted=true                                                                     
bash-5.0# ctr -a /run/host-containerd/containerd.sock i pull artifactory.service.internal/some-image:bla
INFO[0000] trying next host - response was http.StatusNotFound  host=artifactory.service.internal
ctr: failed to resolve reference "artifactory.service.internal/some-image:bla": artifactory.service.internal/some-image:bla: not found
bash-5.0# 

after trying to deploy again on this same host:

Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Normal   Scheduled  11s   default-scheduler  Successfully assigned test/test-0 to ip-x-x-x-x.ec2.internal
  Normal   Pulling    0s    kubelet            Pulling image "artifactory.service.internal/some-image:realtag"
  Warning  Failed     0s    kubelet            Failed to pull image "artifactory.service.internal/some-image:realtag": rpc error: code = Unknown desc = failed to pull and unpack image "artifactory.service.internal/some-image:realtag": failed to resolve reference "artifactory.service.internal/some-image:realtag": failed to do request: Head "https://artifactory.service.internal/v2/some-image/manifests/realtag": x509: certificate signed by unknown authority
  Warning  Failed     0s    kubelet            Error: ErrImagePull
  Normal   BackOff    0s    kubelet            Back-off pulling image "artifactory.service.internal/some-image:realtag"
  Warning  Failed     0s    kubelet            Error: ImagePullBackOff

[etungsten]

Hi @madeddie , thanks for bringing this up. We'll take a closer look into this.

[arnaldo2792]

Hey @madeddie, I was able to replicate what you experiencing, but could you please confirm that you updated the certificates store from a host container? I replicated the problem by updating the certificates store from the admin container, after kubelet and containerd started.

[madeddie]

I used the admin container and I tried in a sheltie

[arnaldo2792]

Thanks for confirming! So we have a bug (:cockroach:), containerd is caching the certificates store and it doesn't reload it if the store changes. There is a work around though, if you don't have a strong requirement to use host containers to update your certificates store, you could use a bootstrap container (as described in #2006) to add certificates to the host's store.

[arnaldo2792]

I opened #2021 to keep track of this.