Integrate the PuzzleFS image into the OCI image specification

[ariel-miculas]

Previously, the OCI Image Index contained a list of manifests which were referencing the PuzzleFS rootfs image, i.e. the metadata of the PuzzleFS image in Capnproto format. Now the Image Index [1] references an Image Manifest [2] and the PuzzleFS image (the PuzzleFS rootfs image together with the file chunks) is embedded into the layers field of the Image Manifest.

Where PuzzleFS diverges from the Image Manifest spec is in the layers definition: our layers are not self contained images and thus they do not stack. Instead, we have a rootfs layer which stores the PuzzleFS image rootfs and multiple file chunks which contain the actual data of the filesystem. No extraction step is performed. Instead, when mounting a PuzzleFS image, the filesystem is reconstructed from the PuzzleFS metadata and the file chunks, not unlike how squashfs/erofs archives are mounted directly.
See the "Inspecting a puzzlefs image" section from the README for more details about the format.

The image config is an empty descriptor [3] for now, but we don't store it in blobs/sha256, which causes skopeo copy to fail because it doesn't find the blob referenced by the empty descriptor in the data store. This will be addressed in a subsequent commit.

See #55 for more context.

[1] https://github.com/opencontainers/image-spec/blob/main/image-index.md
[2] https://github.com/opencontainers/image-spec/blob/main/manifest.md
[3] https://github.com/opencontainers/image-spec/blob/main/manifest.md#guidance-for-an-empty-descriptor

[ariel-miculas]

@hallyn @tych0 I'm curious what you think

[hallyn]

Thanks, I had to reread the description a few times over two days, but I think this is exactly what I had wanted :)

[hallyn]

(Waiting to see what @tych0 thinks before merging)

@ariel-miculas did you notice any performance impact?

[ariel-miculas]

I didn't do any performance tests, but I don't expect performance changes because PuzzleFS still gets the data chunks from the PuzzleFS rootfs image, i.e. the Capnproto metadata file. The descriptors in the image manifests that reference the data chunks are only there so that other tools can transfer the PuzzleFS image between directories/registries etc. The only thing that adds overhead is that we now have to get the PuzzleFS rootfs image from the Image Manifest instead of directly from the Image Index, but that shouldn't be of much concern. And we'll probably need a mount helper for the kernel driver so that it doesn't have to deal with the OCI format.

[tych0]

Yeah, this looks reasonable to me, thanks.

[ariel-miculas]

Great, thanks for the feedback!

[tych0]

I guess one point of note just in your commit message. Maybe a more philosophical point, but:

Where PuzzleFS diverges from the Image Manifest spec is in the layers definition: our layers are not self contained images and thus they do not stack.

https://github.com/opencontainers/image-spec/blob/367a53cec838b502c193399faeae9ce7ace65c50/manifest.md?plain=1#L60-L72

It's a little bit ambiguous to me, we certainly don't do anything "in stack order" since you're right we don't have a stack order, but the result is a fs applied to an empty directory, so the end result is mostly the same.

I wonder if it's worthwhile to define another format for puzzlefs images, sort of like squashfs, where everything is catted together, so that copying images around for testing etc. is easier without all the OCI stuff installed? In a production deployment, you'd still want the OCI format (or something like it) to promote sharing of data across images, but maybe it'll help you not fight the tooling while getting off the ground to have a self-contained format?

Anyway... looking forward to meeting up this week!