How should we handle #376 #379 and #283

[Danielku15]

@baronfel

I would like to move the discussion out of the issues #376 #379 and #283 and PR dotnet/sdk#32132 over here to this place to discuss how the implementation ultimately should look like.

I (we from our company/team) have a strong interest in getting those features and I'd be willing to bring this into the main .net SDK instead of maintaining a custom internal fork if we can agree to the approaches.

Neglecting for now backwards compatibility, I would like to understand how we wanna deal the different usecases described here: dotnet/sdk#32132 (comment)

And conclude maybe some steps forward in supporting those features. This is the information I gathered so far:

Use cases

1. Pushing a container image with specific tags to a remote registry
2. Producing a container image with specific tags into a tar.gz file (while
3. Pushing a container image with specific tags to a local daemon (could internally use 2. but then trigger a load command specific for the daemon)

The following usecase would be also in our interest:

4. We wanna use the same toolchain to create and push images from any given input folder with the related parameters for the image. This allows to build+push images in C++ applications we are creating using existing build outputs. If related tasks are exposed (similar to today) this could be fulfilled. Currently the toolchain doesn't care that the output is a .net application.

Authors have an interest in

• Building the application once for the required plaforms (building the software can be costly).
• Pushing the resulting container images to multiple destinations (as mentioned above)
• Having deterministic output hashes:
  • The layer hash should be the same for all scenarios
  • The container image hash should be the same if the tags are the same across registries.
• Giving the container images different names in the registries (as required by some registries).
• Using different credentials to authenticate for each registry.

Aspects to consider for the future:

• Authors want to publish multi-platform images (linux/windows, x64/arm). The workflow should be extensible towards this.

Possible approaches that were raised until now:

a) Use an approach like docker and define MSBuild properties and items which allows the Authors/SDK to:

• Authenticate against the registries in the push workflows (docker login)
• Define all registries and repositories (name+tags) for the image. (docker image tag)
• Define to which registries a push should happen. (docker image push)

b) Like a) but using an externally provided configuration (yaml or json file?).

c) We trigger each build/push workflow individually but ensure the output of the deterministic regarding hashes. Authors would trigger multiple publish workflows (via msbuild tasks or dotnet cli)

I hope that if we identify the final goal we can tackle the required features step-by-step.

[Danielku15]

Update:

deterministic layers
I looked a bit into the possibilities of having a deterministic output but it seems that this is not really possible. My findings:

• Filling the mtime, ctime, atime headers into the tar doesn't result in same hashes.
• gzip via zlib in .net 8.0 shouldn't have a mtime header set resulting on the same OS on the same header.
• gzip can simply be randome across executions on the same machine with same inputs.

This means producting the same layer or container images in multiple runs. Hence option c) is rather unrealistic.

deterministic images
Assuming we would still be able to have deterministic layers there are some problems in the full container image. The container manifest uses "UtcNow" as created timestamp and therefore we have different images on multiple runs even if the layers, tags etc. are the same. This would need to be changed to use an external timestamp.

Next steps: I will likely make some tests with actual docker builds to see how multiple subsequent builds and tag combinations on push/pull result in hashes. I am interest into how the image and layer hashes change if we change tags and push/pull them across registries. If docker doesn't manage to persist hashes and identifiers, it is also unrealistic for the .net SDK. That could challenge the requirement of "Having deterministic output hashes".

[Danielku15]

I have some findings which might guide the next steps:

1. The "Image ID" in our case is a hash (SHA256) of the serialized image configuration JSON. In this image configuration there are no tags or image names making it reusable for pushes to any registry or daemon with desired names and tags.
2. The "tar.gz" for a "docker load" has the tags embedded in the file (special manifest.json).
3. When pushing the image to the registries via the Docker Registry HTTP API V2, a image manifest is generated which is also fully independent from names and tags.
4. When pushing the image to the registries via the Docker Registry HTTP API V2 we need to
   • Upload all blobs (layers, image config) specifically to the image name (aka repository) it should have. (/v2/<name>/blobs/uploads/). But: Registries might optimize this internally and identify blobs based on digests so that uploads are not needed.
   • Upload the same image manifest for all digests and tags under which the image should be available. (/v2/<name>/manifests/<reference>).

So what is the common foundation for all our push targets:

• The layer blobs are the same regardless of where we push.
• The image ID is the same regardless of where we push.
• The image configuration JSON is the same regardless of where we push
• The image manifest is different for a docker load than for a registry HTTP push.
  • Docker Load: The manifest.json for holds paths relative inside the image tar to the image configuration and layer blobs + the RepoTags key and defines the layer files pointing to the layer tar in the image tar (relative paths)
  • Registry Push: The image manifest holds the blob digests of the config and the layers and their respective format. The tags are decided with the upload of this manifest to the related API point.

My current idea to support multiple registries is:

1. For single registry scenarios keep the direct Docker or Registry push scenarios.
2. Add the feature which writes a docker style tar.gz to disk.
3. Check if we can additionally write the OCI (v2) manifest into the tar.gz of docker without causing troubles. We use the annotations to encode additional data we need for the push workflows.
4. Offer a new custom task which developers can call to push a tar.gz (from step 3.) either into docker or into a registry with custom image names and tags.
   • We load the information from the OCI image manifest.
   • We create the right manifests and formats for the registry or docker push with the given information + layers.
   • We perform the push workflow like usual.

Developers would then hook via MS build targets after the tar.gz creation, and use it as input for push calls.

[Danielku15]

Update idea:

Support of multiple remote registries:

1. We add support for writing OCI Image Archives (https://github.com/opencontainers/image-spec/blob/main/image-layout.md)
2. We add some custom annotations to be able convert the OCI Image Archive.
3. We add a new task like PushContainerImage which accepts the OCI Image Archive the following input parameters and the task will then push the image to the remote registry.
   • The input OCI Image archive (required)
   • The remote registry URL (required)
   • Image name (optional, if provided used, if not provided org.opencontainers.image.base.name annotation is used, if missing -> error)
   • Image tags (optional, if provided used, if not provided index.json with application/vnd.oci.image.manifest.v1+json + org.opencontainers.image.ref.name will be respected)
   • Username and password for authentication (optional)