writing blob: Size mismatch

[edsantiago]

Possibly a broken image or manifest? And possibly because I ^Ced the first time I pushed it, then rebuilt it and pushed a completely different one? Anyhow I don't know the root cause nor how to investigate further nor even where to report this or if it's even a bug.

$ podman pull -q quay.io/libpod/testimage:20241009
138196c38b3baeb5bef83b1513d192fe790c856124cd3c6655b856ddfa95d867
$ skopeo copy containers-storage:quay.io/libpod/testimage:20241009 dir://tmp/sdfsdf
Getting image source signatures
Copying blob 5f70bf18a086 done   | 
Copying blob 160fb30b1281 done   | 
FATA[0000] writing blob: Size mismatch when copying sha256:160fb30b1281fc4d8c9c0bb68e63ab1b55a78be718d50f9657c0a78a030efb32, expected 11801184, got 12175360 

skopeo copy docker://quay.io/libpod/testimage:20241009 works just fine.

[edsantiago]

Same thing with 20241010, which I did not ^C

[nalind]

This doesn't seem to be specific to skopeo, as I can reproduce it with podman push quay.io/libpod/testimage:20241009 dir:/tmp/sdfsdf, so I suspect it's in either containers/image or containers/storage. Adding --dest-compress to the skopeo copy command to force recompression seems to work around it.

[Luap99]

The manifest contains

"annotations": {
    "io.github.containers.compression.zstd": "true"
},

The old (currently used) testimage 20240123 does not so I assume this might have something to do with zstd compression

[mtrmac]

Notes:

• The architecture-specific instance is quay.io/libpod/testimage@sha256:d58f79a56a477ac8ed3c07aa2c8f68cd05e2afdd754d40463dbf888798590aae
• The layer is zstd:chunked.
• 12175360 is the correct size of the payload
• The value comes from c/storage.Layer.UncompressedSize

[mtrmac]

… and that size seems to come from adding the size of file contents ( chunkedDiffer.mergeTocEntries ). Yes, that’s not going to work.

[mtrmac]

I think I know how to fix this, working on it now.

---

If that hypothesis is correct, this means that re-pushes of zstd:chunked layers have been broken for about 1.5 years — i.e. we have a testing gap.

[edsantiago]

We have many zstd testing gaps. We've never actually done any real testing anywhere in CI, only very-halfhearted efforts.

I wonder if this is the bug that keeps blocking my buildah parallel tests PR?

[mtrmac]

containers/storage#2130 might be a fix (absolutely untested!) to unblock, we should have a better one.

---

Do we have a generic: "push an image (using our default settings); pull it (without reusing any layers); push it again" test? That’s essentially what broke here.

[mtrmac]

Confirmed that (an updated version of) containers/storage#2130 fixes this, the metadata now contains the correct size.