store: expose blob API for allowing additional layer store to export lazy layers

[ktock]

This is eStargz implementation of containers/storage#902.

Currently, layers acquired from the stargz store cannot be exported (e.g. podman save and podman push fail) as mentioned in containers/storage#795 (comment). For solving this issue, a new API blob is proposed to the additional layer store (containers/storage#902). This file exposes the raw contents of that layer.

This commit implements this API. For more details about blob API, please refer to containers/storage#902.

This also adds the following annotation to the image created from nativeconverter.

• io.containers.estargz.uncompressed-size : contains the uncompressed size of the layer.

For exporting a layer, the additional layer store (stargz store) needs to expose that layer's uncompressed size through info file. However, the current OCI image doesn't contain this information by default. So we need this special annotation.

demo

(stargz-store is required)

Lazily pulled image can run:

# podman pull ghcr.io/ktock/python:3.9-esgz-storage
Trying to pull ghcr.io/ktock/python:3.9-esgz-storage...
Getting image source signatures
Copying blob 0180433592e1 skipped: already exists
... (performs lazy pull) ...
# podman run --rm -it ghcr.io/ktock/python:3.9-esgz-storage echo hello
hello

Lazily pulled image can save and push:

# podman save ghcr.io/ktock/python:3.9-esgz-storage > /tmp/python.tar
# cat /tmp/python.tar | tar --list
6c853b792e30742d9696a6817feb2229ab0f46cf54f3a3b663811252c2f73502.tar
...
# podman tag ghcr.io/ktock/python:3.9-esgz-storage ghcr.io/ktock/python:3.9-esgz-storage-repush
# podman push ghcr.io/ktock/python:3.9-esgz-storage-repush

Note that re-pushed image isn't eStargz anymore because podman re-compresses the blob in the non-eStargz way.
TOC digest annotations are also eliminated.

# podman pull ghcr.io/ktock/python:3.9-esgz-storage-repush
{"error":"failed to resolve layer: failed to resolve layer \"ghcr.io/ktock/python:3.9-esgz-storage-repush\" / \"sha256:35ffa438f5d9b28c29e6519690c7ee1bbd395cc873e23bc222314a9c069af60d\": failed to read layer: failed to parse stargz: error parsing footer: 2 error(s) occurred:\n\t* gzip: invalid header\n\t* legacy: failed to get footer gzip reader: gzip: invalid header","layerdigest":"sha256:35ffa438f5d9b28c29e6519690c7ee1bbd395cc873e23bc222314a9c069af60d","level":"debug","msg":"error resolving layer (context error: \u003cnil\u003e)","remote-snapshot-prepared":"false","time":"2021-05-27T01:36:46.405188322Z"}
... (performs non-lazy pull) ...

[ktock]

@AkihiroSuda PTAL 🙏

[AkihiroSuda]

Is this a typo of "containerd" or intentional?

[ktock]

This label isn't used by containerd but used by podman/CRI-O so it's intentional.