DNS issue when deploying a Knative service from the insecure in-cluster registry

[metacoma]

Ask your question here:

Is it possible to use in-cluster DNS names in the .spec.containers[].image field?

I have an insecure registry deployed inside the cluster. The full registry URL is
zot-int.zot.svc.cluster.local:5000

The following Knative Serving manifest:

   apiVersion: serving.knative.dev/v1
    kind: Service
    metadata:
      name: knative-function-test
    spec:
      template:
        spec:
          containers:
            - image: zot-int.zot.svc.cluster.local:5000/test3:latest

leads to the following deployment error:

  Warning  Failed     5s (x2 over 18s)   kubelet            Failed to pull image "zot-int.zot.svc.cluster.local:5000/test3:latest": Error 
response from daemon: Get "http://zot-int.zot.svc.cluster.local:5000/v2/": dial tcp: lookup zot-int.zot.svc.cluster.local: 
Temporary failure in name resolution

However, the FQDN zot-int.zot.svc.cluster.local resolves correctly:

zot-int.zot.svc.cluster.local has address 10.43.189.152

Additionally, this registry is listed in registries-skipping-tag-resolving within the config-deployment ConfigMap:

registries-skipping-tag-resolving: 10.43.189.152:5000,zot-int.zot:5000,zot-int.zot.svc.cluster.local:5000

The host's /etc/docker/daemon.json also includes the registry in the insecure-registries list:

{
  "insecure-registries": [
    "10.43.189.152:5000",
    "zot-int.zot:5000",
    "zot-int.zot.svc.cluster.local:5000"
  ]
}

ugly workaround

If I change the DNS host in registry_url to the IP address in image service definition, like

   apiVersion: serving.knative.dev/v1
    kind: Service
    metadata:
      name: knative-function-test
    spec:
      template:
        spec:
          containers:
            - image: 10.43.189.152:5000/test3:latest

Everything works as expected

NAME                                                      READY   STATUS    RESTARTS   AGE                                                    
knative-function-test-00001-deployment-586cc9fff8-224sp   2/2     Running   0          12s             
...
  Normal  Pulling    18s   kubelet            Pulling image "10.43.189.152:5000/test3:latest"
  Normal  Pulled     18s   kubelet            Successfully pulled image "10.43.189.152:5000/test3:latest" in 37ms (37ms including waiting). Image size: 500156127 bytes.

The HostAliases solution (as described here knative-extensions/kn-plugin-quickstart#429) doesn't work for me because the registry IP address may change

env:

# single-node cluster deployed by k3s
Server Version: v1.32.0+k3s1
knative-operator 1.16.0
knative-serving   1.16.0    True    
knative-eventing   1.16.0    True    

[skonto]

Hi @metacoma,

Warning Failed 5s (x2 over 18s) kubelet Failed to pull image "zot-int.zot.svc.cluster.local:5000/test3:latest": Error

The above error indicates that kubelet cannot have access to that DNS name. Kubelet needs to be able to resolve that so it can pull the image.

[metacoma]

hey @skonto thank you for help,

Kubelet needs to be able to resolve that so it can pull the image

Yes, you are right. It was my misunderstanding that Docker resolves registry address from the "host" system.

For lurkers, here are short snippets on how to fix it in a single-node k3s cluster. I tested it on Ubuntu 24.04.

kubectl apply -f

---
apiVersion: operator.knative.dev/v1beta1
kind: KnativeServing
metadata:
  annotations:
  name: knative-serving
  namespace: knative-serving
spec:
  config:
    deployment:
      registries-skipping-tag-resolving: zot-int.zot.svc.cluster.local:5000,zot-int.zot:5000,127.0.0.1:30001
---
apiVersion: v1
kind: Service
metadata:
  name: dns-expose
  namespace: kube-system
spec:
  type: NodePort
  ports:
    - name: dns-tcp
      port: 53
      targetPort: 53
      protocol: TCP
      nodePort: 30002  
    - name: dns-udp
      port: 53
      targetPort: 53
      protocol: UDP
      nodePort: 30002  
  selector:
    k8s-app: kube-dns

.ONESHELL: docker_insecure_registry
docker_insecure_registry:
        INSECURE_REGISTRY="zot-int.zot.svc.cluster.local:5000"
        DOCKER_CONFIG="/etc/docker/daemon.json"

        if [[ ! -f "$$DOCKER_CONFIG" ]]; then
                        echo "{}" | sudo tee "$$DOCKER_CONFIG" > /dev/null
        fi
        if ! jq -e ".\"insecure-registries\" | index(\"$$INSECURE_REGISTRY\")" "$$DOCKER_CONFIG" > /dev/null; then
                        TMP_FILE=/tmp/daemon.json
                        jq --arg reg "$$INSECURE_REGISTRY" '
                                        .["insecure-registries"] += [$$reg] // { "insecure-registries": [$$reg] }
                        ' "$$DOCKER_CONFIG" > "$$TMP_FILE" && sudo mv "$$TMP_FILE" "$$DOCKER_CONFIG"
                        sudo systemctl restart docker
        fi
.ONESHELL: forward_dns_cluster_local
forward_dns_cluster_local:
        (test -d /etc/systemd/resolved.conf.d || sudo mkdir -p /etc/systemd/resolved.conf.d)
        cat<<EOF | sudo tee /etc/systemd/resolved.conf.d/k8s-dns.conf
        [Resolve]
        DNS=127.0.0.1:30002
        Domains=~svc.cluster.local
        EOF
        sudo systemctl restart systemd-resolved

[skonto]

Nice! On my side I used a private registry on Minikube on my linux box like this:

# Created the certs
 openssl req \
  -newkey rsa:4096 -nodes -sha256 -keyout certs/registry.key \
  -addext "subjectAltName = DNS:registry.test" \
  -x509 -days 3650 -out certs/registry.crt

kubectl create secret tls registry-cert \
    --cert=certs/registry.crt \
    --key=certs/registry.key \
    -n test

Then deployed the private registry deployment and the following svc for that registry (in test namespace):

k get svc -n test
NAME       TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)          AGE
registry   LoadBalancer   10.100.15.191   10.100.15.191   5000:32755/TCP   47s

 curl  --cacert certs/registry.crt --resolve registry.test:5000:10.100.15.191  https://registry.test:5000/v2/_catalog
{"repositories":[]}


# add an entry in /etc/hosts of my linux box for registry.test to resolve to 10.100.15.191

mkdir -p /etc/docker/certs.d/registry.test:5000

# Copy your crt in  /etc/docker/certs.d/registry.test:5000 and restart docker daemon

docker push registry.test:5000/helloworld-go 

 curl  --cacert certs/registry.crt --resolve registry.test:5000:10.100.15.191  https://registry.test:5000/v2/_catalog
{"repositories":["helloworld-go"]}

# add  registries-skipping-tag-resolving: registry.test:5000 in config-deployement
# Probably we can skip this if we set username/password for the registry (I only set tls) and then follow https://knative.dev/docs/serving/deploying-from-private-registry/#procedure

# Then used the following image in a ksvc
     containers:
      - image: registry.test:5000/helloworld-go

[dprotaso]

For kind you can see how to patch the containerd setup here

https://github.com/chainguard-dev/actions/blob/300b446c0eb21e051cc749e5ed117e3a39ed4dbe/setup-kind/action.yaml#L161-L162

[metacoma]

@skonto, thank you for your snippets — they’re very useful!