Can Zot work as a pull through cache registry?

[ossfellow]

Hello,
Zot's streamlined design and single binary implementation makes it a great registry choice; so thank you for creating it.

I'd like to use it; however I can't tell from the documentation if Zot can be used as a pull through cache registry if the onDemand attribute of the Sync extension is set to true, or not; or even some other configuration could turn it into such a registry.

I appreciate your insight and clarification.

[rchincha]

@masoudbahar I suppose you are referring to https://zotregistry.io/latest/articles/mirroring/

[ossfellow]

@rchincha, indeed not, as I’ve seen the mirroring feature, which is quite useful.
A pull through cache registry only pulls down an image that is being requested, so, the subsequent pulls will use the local cache, speeding up the operations; it doesn’t pull all the images down, as is the case with a mirror.

Have I misunderstood the Zot’s mirroring capability?

[rchincha]

@masoudbahar, when the sync extension is enabled, zot registry does keep a local copy in its cache and serves future requests from this cache. The trigger to pull from a remote/upstream OCI registry is either poll-driven or ondemand which should only happen once per image.

[ossfellow]

Awesome; so, to configure this all I need to do is to enable Sync extension, and set the onDemand attribute to true, and it’ll only pull down the images requested, not every image stored in the upstream registry; right? 🙂

[rchincha]

Correct. Pls do try and let us know.

[eusebiu-constantin-petu-dbk]

That's correct.

[ossfellow]

Awesome; thank you for explaining how it works.

[ralgozino]

Hi! sorry for writing on an already answered topic, but my question is related to this use case.

I have a zot instance running and configured as you described to act as a pull-through cache, I can see in the logs that the images are being pulled by zot and it is all transparent for the client pulling the image from zot.

The pulled images don't show up neither in the UI nor with the zli image list command. Is this expected? I would like them to show up in there, they are available images just like the others, aren't they? This would be useful to know which images are being used from the remote registries also :)

Thanks!

[rchincha]

If you don't mind, could you also share your (anonymized) configuration so we can try it out.

[ralgozino]

sure thing, here it is:

{
  "distSpecVersion": "1.0.1",
  "storage": {
    "rootDirectory": "redacted"
  },
  "http": {
    "address": "0.0.0.0",
    "port": "8080",
    "tls": {
      "cert": "redacted",
      "key": "redacted"
    }
  },
  "log": {
    "level": "debug"
  },
  "extensions": {
    "ui": {
      "enable": true
    },
    "search": {
      "enable": true,
      "cve": {
        "updateInterval": "2h"
      }
    },
    "sync": {
      "enable": true,
      "registries": [
        {
          "urls": [
            "https://k8s.gcr.io",
            "https://registry.k8s.io"
          ],
          "content": [
            {
              "prefix": "**",
              "destination": "/k8s"
            }
          ],
          "onDemand": true,
          "tlsVerify": true
        },
        {
          "urls": [
            "https://ghcr.io"
          ],
          "content": [
            {
              "prefix": "**",
              "destination": "/ghcr"
            }
          ],
          "onDemand": true,
          "tlsVerify": true
        },
        {
          "urls": [
            "https://quay.io"
          ],
          "content": [
            {
              "prefix": "**",
              "destination": "/quay"
            }
          ],
          "onDemand": true,
          "tlsVerify": true
        },
        {
          "urls": [
            "https://docker.io/library"
          ],
          "content": [
            {
              "prefix": "**",
              "destination": "/docker"
            }
          ],
          "onDemand": true,
          "tlsVerify": true
        }
      ]
    }
  }
}

[ralgozino]

@rchincha there may be something else going on here...

If I do docker pull <my zot>/docker/<some image> the image gets pulled through and I now see it in the UI.

My previous tests were in a Kubernetes cluster, setting the mirror option of containerd, like this:

version = 2
root = "/var/lib/containerd"
state = "/run/containerd"
oom_score = 0

[grpc]
  max_recv_message_size = 16777216
  max_send_message_size = 16777216

[debug]
  level = "info"

[metrics]
  address = ""
  grpc_histogram = false

[plugins]
  [plugins."io.containerd.grpc.v1.cri"]
    sandbox_image = "k8s.gcr.io/pause:3.2"
    max_container_log_line_size = 16384
    [plugins."io.containerd.grpc.v1.cri".containerd]
      default_runtime_name = "runc"
      snapshotter = "overlayfs"
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
         [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
          runtime_type = "io.containerd.runc.v2"
          runtime_engine = ""
          runtime_root = ""
          privileged_without_host_devices = false
          base_runtime_spec = ""
          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
            SystemdCgroup = true
    [plugins."io.containerd.grpc.v1.cri".registry]
    [plugins."io.containerd.grpc.v1.cri".registry.configs]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."ghcr.io"]
          endpoint = ['https://<my zot instance>:8080/ghcr']
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."quay.io"]
          endpoint = ['https://<my zot instance>:8080/quay']
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."registry.k8s.io"]
          endpoint = ['https://<my zot instance>:8080/k8s']
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."k8s.gcr.io"]
          endpoint = ['https://<my zot instance>:8080/k8s']
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
          endpoint = ['https://<my zot instance>:8080/docker']

I don't know what containerd does differently when it pulls from a "mirror" that makes it not show up on Zot's UI, I can see in Zot's logs that containerd is fetching from it. For example, if I run a pod that uses the image docker.io/ralgozino/echo I then see this in zots' logs (and other similar lines):

{"level":"info","module":"http","component":"session","clientIP":"192.168.65.1:50737","method":"GET","path":"/v2/docker/ralgozino/echo/blobs/sha256:b57d7311d892434ee35357cac35e02188c22a707893d636961802d8dc9aaa889","statusCode":200,"latency":"0s","bodySize":1013,"headers":{"Accept-Encoding":["gzip"],"Connection":["close"],"User-Agent":["docker/25.0.3 go/go1.21.6 git-commit/f417435 kernel/6.6.16-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/25.0.3 \\(darwin\\))"]},"goroutine":12496,"caller":"zotregistry.dev/zot/pkg/api/session.go:132","time":"2024-04-05T19:16:46.774172345+02:00","message":"HTTP API"}

[rchincha]

@ralgozino you should look for this log msg instead.
pkg/extensions/sync/destination.go:184: registry.log.Info().Str("image", fmt.Sprintf("%s:%s", repo, reference)).Msg("successfully synced image")

If this is not seen, then the sync didn't happen (check for errors in log)

[ralgozino]

Hi @rchincha , thanks for the tips, it seems that containerd is not working as I thought and it may even not support using a subpath in the mirror address (i.e. ///). I'll need to dig this deeper.

Zot mirroring is working fine 🎉

Thank you for your quick feedback and sorry for wasting your time.

Have a great week!