Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pkg/compression: new zstd variant zstd:chunked #1084

Merged
merged 12 commits into from
Jul 7, 2021
Merged

pkg/compression: new zstd variant zstd:chunked #1084

merged 12 commits into from
Jul 7, 2021

Commits on Jul 2, 2021

  1. compression: change return type for zstdWriterWithLevel 

    Signed-off-by: Giuseppe Scrivano <[email protected]>
    @giuseppe
    giuseppe committed Jul 2, 2021
    Configuration menu
    Copy the full SHA
    48f39f6 View commit details
    Browse the repository at this point in the history

  2. compression: support generating compressor metadata 

    Signed-off-by: Giuseppe Scrivano <[email protected]>
    @giuseppe
    giuseppe committed Jul 2, 2021
    Configuration menu
    Copy the full SHA
    b167b4b View commit details
    Browse the repository at this point in the history

  3. compression: let algorithms register a MIME type 

    Signed-off-by: Giuseppe Scrivano <[email protected]>
    @giuseppe
    giuseppe committed Jul 2, 2021
    Configuration menu
    Copy the full SHA
    7421a48 View commit details
    Browse the repository at this point in the history

  4. pkg/compression: new zstd variant zstd:chunked 

    add a new custom variant of the zstd compression that permits to
retrieve each file separately.

The idea is based on CRFS and its stargz format for having seekable
and indexable tarballs.

One disadvantage of the stargz format is that a custom file is added
to the tarball to store the layer metadata, and the metadata file is
part of the image itself.  Clients that are not aware of the stargz
format will propagate the metadata file inside of the containers.

The zstd compression supports embeddeding additional data as part of
the stream that the zstd decompressor will ignore (skippable frame),
so the issue above with CRFS can be solved directly within the zstd
compression format.

Beside this minor advantage, zstd is much faster and compresses better
than gzip, so take this opportunity to push the zstd format further.

The zstd compression is supported by the OCI image specs since August
2019: opencontainers/image-spec#788 and has
been supported by containers/image since then.

Clients that are not aware of the zstd:chunked format, won't notice
any difference when handling a blob that uses the variant.

Signed-off-by: Giuseppe Scrivano <[email protected]>
    @giuseppe
    giuseppe committed Jul 2, 2021
    Configuration menu
    Copy the full SHA
    40848f4 View commit details
    Browse the repository at this point in the history

  5. types: add interface for partial blob retrieval 

    Signed-off-by: Giuseppe Scrivano <[email protected]>
    @giuseppe
    giuseppe committed Jul 2, 2021
    Configuration menu
    Copy the full SHA
    e8ce5b8 View commit details
    Browse the repository at this point in the history

  6. types: add interface for storing partial blob 

    Signed-off-by: Giuseppe Scrivano <[email protected]>
    @giuseppe
    giuseppe committed Jul 2, 2021
    Configuration menu
    Copy the full SHA
    05f70a3 View commit details
    Browse the repository at this point in the history

  7. docker: support partial blob retrieval 

    Signed-off-by: Giuseppe Scrivano <[email protected]>
    @giuseppe
    giuseppe committed Jul 2, 2021
    Configuration menu
    Copy the full SHA
    93477e6 View commit details
    Browse the repository at this point in the history

  8. storage: support partial storage with zstd:chunked 

    Signed-off-by: Giuseppe Scrivano <[email protected]>
    @giuseppe
    giuseppe committed Jul 2, 2021
    Configuration menu
    Copy the full SHA
    67a4c99 View commit details
    Browse the repository at this point in the history

  9. copy: do not ignore errors on Close 

    Signed-off-by: Giuseppe Scrivano <[email protected]>
    @giuseppe
    giuseppe committed Jul 2, 2021
    Configuration menu
    Copy the full SHA
    3250f2d View commit details
    Browse the repository at this point in the history

  10. copy: use partial blob retrieval if possible 

    Signed-off-by: Giuseppe Scrivano <[email protected]>
    @giuseppe
    giuseppe committed Jul 2, 2021
    Configuration menu
    Copy the full SHA
    138ab62 View commit details
    Browse the repository at this point in the history

  11. copy: provide progress bar for partial blobs 

    Signed-off-by: Giuseppe Scrivano <[email protected]>
    @giuseppe
    giuseppe committed Jul 2, 2021
    Configuration menu
    Copy the full SHA
    ee5014c View commit details
    Browse the repository at this point in the history

  12. copy: make using partial blobs configurable 

    Signed-off-by: Giuseppe Scrivano <[email protected]>
    @giuseppe
    giuseppe committed Jul 2, 2021
    Configuration menu
    Copy the full SHA
    be50ba7 View commit details
    Browse the repository at this point in the history