Skopeo failing to copy rock to Docker

[misohu]

Bug Description

CI fails to copy rock to Docker during skopeo executions.

I believe we are running the preinstalled versions for both skopeo and Docker client on the runners. The log bellow indicates that we need higher Docker api version in order to use skopeo (at least that version).

To Reproduce

1. You can run ci in these PRs Update agent to v0.13.0 kserve-rocks#82, Update controller to v0.13.0 kserve-rocks#83

Environment

action canonical/charmed-kubeflow-workflows/.github/workflows/get-rocks-modified-and-build-scan-test-publish.yaml@main on ubuntu:22.04

Relevant Log Output

Run sudo skopeo --insecure-policy copy oci-archive:kserve-agent_0.13.0_amd64.rock docker-daemon:kserve-agent:0.13.0
Getting image source signatures
Copying blob sha2[5](https://github.com/canonical/kserve-rocks/actions/runs/9480968913/job/26123730171?pr=82#step:4:6)6:7646c8da332499ae416b15479ce832db32e39a501c[6](https://github.com/canonical/kserve-rocks/actions/runs/9480968913/job/26123730171?pr=82#step:4:7)62e24324f595509a0d3db
Copying blob sha256:c197fee565ab2c9ab8dc17f1cde1331f4d0a24e845af008840aff5a0c85455e9
Copying blob sha256:dc0f6188653a3[7](https://github.com/canonical/kserve-rocks/actions/runs/9480968913/job/26123730171?pr=82#step:4:8)90eed5cc22d6fc6003b425e2ffdc77f3af9b7332eb6a22452d
Copying blob sha256:582d5fcd551c5c4c4d9edc622b6[8](https://github.com/canonical/kserve-rocks/actions/runs/9480968913/job/26123730171?pr=82#step:4:9)5fddd47b2bfed0b3a347aa30b3e259255aa5
Copying blob sha256:e0d20462638c0c260db9cca2534a238ce0ddc060b3704b51d765101c3e466c1b
time="2024-06-12T10:53:00Z" level=fatal msg="writing blob: saving image to docker engine: Error response from daemon: {\"message\":\"client version 1.22 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version\"}"

Additional Context

No response

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5866.

This message was autogenerated

[misohu]

The problem is that the preinstalled skopeo in the runners image ubuntu-20.04 is too old compared to the version of Docker API preinstalled in the image.

Version of skopeo skopeo version 1.4.1.

Version of Docker

In order to fix the problem we need to:

• Downgrade the Docker version
• or Upgrade the skopeo version

Downgrading the Docker version is more time consuming as this downgrade needs to happen on every job run. We went with the second option of upgrading skopeo. Because of this we will use the skopeo preinstalled with rockcraft (this was also recommended solution by rockcraft team). Skopeo version in rockcraft skopeo version 1.14.2.

I have spent some time debugging why the scopeo in rockcraft is not working. The skopeo in rockcraft is right now only available in the edge channel!!!!