podman on ppc64le: Error: incompatible image architecture "amd64" on "ppc64le" host

[runlevel5]

[root@orion tle]# minikube delete
🙄  "minikube" profile does not exist, trying anyways.
💀  Removed all traces of the "minikube" cluster.
[root@orion tle]# minikube delete
🙄  "minikube" profile does not exist, trying anyways.
💀  Removed all traces of the "minikube" cluster.
[root@orion tle]# rm -rf /tmp/minikube*
[root@orion tle]# rm -rf /tmp/juju*
[root@orion tle]# /usr/local/bin/minikube --alsologtostderr start --driver=podman
I0317 22:40:42.091465   10546 notify.go:125] Checking for updates...
I0317 22:40:42.369365   10546 start.go:252] hostinfo: {"hostname":"orion.dev","uptime":39818,"bootTime":1584405424,"procs":443,"os":"linux","platform":"fedora","platformFamily":"fedora","platformVersion":"32","kernelVersion":"5.6.0-0.rc5.git0.2.fc32.ppc64le","virtualizationSystem":"","virtualizationRole":"","hostid":"5632f077-29a6-48c4-9d05-933910ac9490"}
I0317 22:40:42.381682   10546 start.go:262] virtualization:
😄  minikube v1.8.2 on Fedora 32 (ppc64le)
I0317 22:40:42.381757   10546 driver.go:226] Setting default libvirt URI to qemu:///system
✨  Using the podman (experimental) driver based on user configuration
I0317 22:40:42.570332   10546 start.go:299] selected driver: podman
I0317 22:40:42.570356   10546 start.go:488] validating driver "podman" against <nil>
I0317 22:40:42.570382   10546 start.go:494] status for podman: {Installed:true Healthy:true Error:<nil> Fix: Doc:}
W0317 22:40:42.700205   10546 start.go:852] Unable to query memory limits: get podman system info: exit status 125
I0317 22:40:42.700471   10546 start.go:863] Using suggested 2200MB memory alloc based on sys=-1MB, container=-1MB
I0317 22:40:42.700663   10546 cache.go:97] Beginning downloading kic artifacts
I0317 22:40:42.700707   10546 preload.go:84] Found local preload: /root/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v1-v1.17.3-docker-overlay2.tar.lz4
I0317 22:40:42.700720   10546 cache.go:41] Caching tarball of preloaded images
I0317 22:40:42.700747   10546 preload.go:113] Found /root/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v1-v1.17.3-docker-overlay2.tar.lz4 in cache, skipping download
I0317 22:40:42.700750   10546 cache.go:99] Downloading gcr.io/k8s-minikube/kicbase:v0.0.7@sha256:a6f288de0e5863cdeab711fa6bafa38ee7d8d285ca14216ecf84fcfb07c7d176 to local daemon
I0317 22:40:42.700792   10546 image.go:84] Writing gcr.io/k8s-minikube/kicbase:v0.0.7@sha256:a6f288de0e5863cdeab711fa6bafa38ee7d8d285ca14216ecf84fcfb07c7d176 to local daemon
I0317 22:40:42.700764   10546 cache.go:44] Finished downloading the preloaded tar for v1.17.3 on docker
I0317 22:40:42.701503   10546 profile.go:121] Saving config to /root/.minikube/profiles/minikube/config.json ...
I0317 22:40:42.701926   10546 lock.go:35] WriteFile acquiring /root/.minikube/profiles/minikube/config.json: {Name:mk270d1b5db5965f2dc9e9e25770a63417031943 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
E0317 22:40:44.130208   10546 cache.go:106] Error downloading kic artifacts:  error loading image: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
I0317 22:40:44.130429   10546 start.go:245] acquiring machines lock for minikube: {Name:mka00e65579c2b557a802898fd1cf03ec4ab30a1 Clock:{} Delay:500ms Timeout:15m0s Cancel:<nil>}
I0317 22:40:44.130652   10546 start.go:249] acquired machines lock for "minikube" in 200.14µs
I0317 22:40:44.130674   10546 start.go:82] Provisioning new machine with config: {Name:minikube KeepContext:false EmbedCerts:false MinikubeISO: Memory:2200 CPUs:2 DiskSize:20000 Driver:podman HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.99.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio KubernetesConfig:{KubernetesVersion:v1.17.3 ClusterName:minikube APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false NodeIP: NodePort:0 NodeName:} Nodes:[{Name:m01 IP: Port:8443 KubernetesVersion:v1.17.3 ControlPlane:true Worker:true}] Addons:map[]}
I0317 22:40:44.130758   10546 start.go:101] createHost starting for "minikube" (driver="podman")
I0317 22:40:44.290506   10546 start.go:137] libmachine.API.Create for "minikube" (driver="podman")
I0317 22:40:44.290596   10546 main.go:110] libmachine: Reading certificate data from /root/.minikube/certs/ca.pem
I0317 22:40:44.290649   10546 main.go:110] libmachine: Decoding PEM data...
I0317 22:40:44.290680   10546 main.go:110] libmachine: Parsing certificate...
I0317 22:40:44.290847   10546 main.go:110] libmachine: Reading certificate data from /root/.minikube/certs/cert.pem
I0317 22:40:44.290885   10546 main.go:110] libmachine: Decoding PEM data...
I0317 22:40:44.290909   10546 main.go:110] libmachine: Parsing certificate...
I0317 22:42:06.541618   10546 start.go:104] createHost completed in 1m22.410840273s
I0317 22:42:06.541650   10546 start.go:73] releasing machines lock for "minikube", held for 1m22.41098338s
W0317 22:42:06.543435   10546 exit.go:101] Unable to start VM. Please investigate and run 'minikube delete' if possible: creating host: create: creating: create kic node: create container: failed args: [run --cgroup-manager cgroupfs -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --volume /root/.minikube/machines/minikube/var:/var:exec --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 gcr.io/k8s-minikube/kicbase:v0.0.7] output: Trying to pull gcr.io/k8s-minikube/kicbase:v0.0.7...
Getting image source signatures
Copying blob sha256:15fa850485767a7b5ef5a59fcd74b158b9a7faffef6254d4cf0c8d86d347e1ec
Copying blob sha256:6bd890da28be3b501e0dc75627755a536831416ed3268f47c2a517065f4edd5b
Copying blob sha256:8aa40341c4fa46ae7bf3b1915da155fd05e53bcc06cfea0606a055b781dd14cc
Copying blob sha256:46bddb84d1c5e5324fcc5efc270c7e081cc9966c125ac56c900dbca6b7a2fe06
Copying blob sha256:922219304a679d5e50e5d0428842216835a792da80fde4f4aaead52cb3baf824
Copying blob sha256:93956c6f8d9ea3e705e03e79d66a5c2acc9a4ac10ed6fb5c6cf654ba9c297df5
Copying blob sha256:a58bb48ed799948df93d5a6bde4feea06b5bb7e4b61d2c3cdca4f71050b113d7
Copying blob sha256:0dc2bee53732b00357eab324b5923de8410b75dcdbdeddf5dda80c378ff6f8d7
Copying blob sha256:998bda65c4e623ae5513998c4a0491ca9c5538ff601587b30fe3fb0b8c9c10c6
Copying blob sha256:54894fbd4ed0f89a552938808f56f8728ce07c0b2b236726b6c89102080a7af4
Copying blob sha256:33b335b88b45f8eaf38714537ca6a60692251cfd644a050bfef81e5e6e580298
Copying blob sha256:9deea2def60c3a520e2a9b36c76e9592124a1efeddb174334c9f5487660e2f60
Copying blob sha256:3e8f76e34c5db014fe4497b76025ddebdc6a4ff4a54059c757b820aa02215916
Copying blob sha256:359ec970e234324726fe5e8e70f8fc3a88743cd6976b3078e0334cf62af9b6fe
Copying blob sha256:eb43b9231aa1f8cb5718927bb7b0662e9b40f5a763374f0b817ea37b18cb5c35
Copying blob sha256:bb9df884c801ffcb4aa35216ec0b0f32b4366815749550a438a773412f356f3f
Copying blob sha256:455a40adb4259c8dda78c35d5cfc938305fa97a4f69daa233b4de9dfa41c19ec
Copying blob sha256:8318dc076aee62be32bac0c305c52be1e2685789930ed9b9fb2fca3643af0b7e
Copying blob sha256:53f24a064b6afcd3a71c87af9b479af1a21f01d2046a3cb51e34f3420fcc88ac
Copying blob sha256:277bbe363ef3c4d87bc8cf7291880a3fbbf701319bb1a64f4566fac04e183502
Copying blob sha256:cd4b881d58cc58a1bd39a900a754e74fda0916440c82ceaedf7a2b7336e918da
Copying config sha256:7980bce73693acb180efd04278d5f9fe4a14f75fb8a54f6e119578bdf1e49a72
Writing manifest to image destination
Storing signatures
Error: incompatible image architecture "amd64" on "ppc64le" host
: exit status 125

💣  Unable to start VM. Please investigate and run 'minikube delete' if possible: creating host: create: creating: create kic node: create container: failed args: [run --cgroup-manager cgroupfs -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --volume /root/.minikube/machines/minikube/var:/var:exec --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 gcr.io/k8s-minikube/kicbase:v0.0.7] output: Trying to pull gcr.io/k8s-minikube/kicbase:v0.0.7...
Getting image source signatures
Copying blob sha256:15fa850485767a7b5ef5a59fcd74b158b9a7faffef6254d4cf0c8d86d347e1ec
Copying blob sha256:6bd890da28be3b501e0dc75627755a536831416ed3268f47c2a517065f4edd5b
Copying blob sha256:8aa40341c4fa46ae7bf3b1915da155fd05e53bcc06cfea0606a055b781dd14cc
Copying blob sha256:46bddb84d1c5e5324fcc5efc270c7e081cc9966c125ac56c900dbca6b7a2fe06
Copying blob sha256:922219304a679d5e50e5d0428842216835a792da80fde4f4aaead52cb3baf824
Copying blob sha256:93956c6f8d9ea3e705e03e79d66a5c2acc9a4ac10ed6fb5c6cf654ba9c297df5
Copying blob sha256:a58bb48ed799948df93d5a6bde4feea06b5bb7e4b61d2c3cdca4f71050b113d7
Copying blob sha256:0dc2bee53732b00357eab324b5923de8410b75dcdbdeddf5dda80c378ff6f8d7
Copying blob sha256:998bda65c4e623ae5513998c4a0491ca9c5538ff601587b30fe3fb0b8c9c10c6
Copying blob sha256:54894fbd4ed0f89a552938808f56f8728ce07c0b2b236726b6c89102080a7af4
Copying blob sha256:33b335b88b45f8eaf38714537ca6a60692251cfd644a050bfef81e5e6e580298
Copying blob sha256:9deea2def60c3a520e2a9b36c76e9592124a1efeddb174334c9f5487660e2f60
Copying blob sha256:3e8f76e34c5db014fe4497b76025ddebdc6a4ff4a54059c757b820aa02215916
Copying blob sha256:359ec970e234324726fe5e8e70f8fc3a88743cd6976b3078e0334cf62af9b6fe
Copying blob sha256:eb43b9231aa1f8cb5718927bb7b0662e9b40f5a763374f0b817ea37b18cb5c35
Copying blob sha256:bb9df884c801ffcb4aa35216ec0b0f32b4366815749550a438a773412f356f3f
Copying blob sha256:455a40adb4259c8dda78c35d5cfc938305fa97a4f69daa233b4de9dfa41c19ec
Copying blob sha256:8318dc076aee62be32bac0c305c52be1e2685789930ed9b9fb2fca3643af0b7e
Copying blob sha256:53f24a064b6afcd3a71c87af9b479af1a21f01d2046a3cb51e34f3420fcc88ac
Copying blob sha256:277bbe363ef3c4d87bc8cf7291880a3fbbf701319bb1a64f4566fac04e183502
Copying blob sha256:cd4b881d58cc58a1bd39a900a754e74fda0916440c82ceaedf7a2b7336e918da
Copying config sha256:7980bce73693acb180efd04278d5f9fe4a14f75fb8a54f6e119578bdf1e49a72
Writing manifest to image destination
Storing signatures
Error: incompatible image architecture "amd64" on "ppc64le" host
: exit status 125

😿  minikube is exiting due to an error. If the above message is not useful, open an issue:
👉  https://github.com/kubernetes/minikube/issues/new/choose

It's seems to me the gcr.io/k8s-minikube/kicbase:v0.0.7 image does not come in ppc64le variant.

[afbjorklund]

KIC only supports amd64 for now, same as the minikube ISO

[runlevel5]

@afbjorklund please do not hesitate to let me know if I could be any help.

AFAIK TravisCI does support ppc64

[afbjorklund]

What I mean is that our images only support amd64 architecture, and not s390x/ppc64le

It could be added as a future feature, but we are lacking test equipment (as noted elsewhere)
It's nice that Travis offers limited LXD support for unit tests, but not enough for regression testing.

Feel free to add some background to #6798 as to what this would be used for (and where)

[tstromberg]

There is no technical reason why this shouldn't work, but it will need someone willing to provide elbow grease in order to get the Docker images into the right shape.

Help wanted!

[afbjorklund]

I was just thinking that it should give an error, if you try to run anything but none on non-amd64 ?

[runlevel5]

@tstromberg @afbjorklund Love to help if you two could give me guidance on where to start.

[afbjorklund]

I think @medyagh is our resident expert on all things KIC, and the docker image used for it

https://github.com/kubernetes/minikube/blob/master/hack/images/kicbase.Dockerfile

[runlevel5]

I think @medyagh is our resident expert on all things KIC, and the docker image used for it
https://github.com/kubernetes/minikube/blob/master/hack/images/kicbase.Dockerfile

The kindest/base image should support ppc64le first. I've lodged an Issue upstream (link: kubernetes-sigs/kind#1444) however it's seems to me the author is not so keen to support ppc64le.

[afbjorklund]

The kindest/base image should support ppc64le first.

I don't think that is needed, as long as ubuntu:19.10 and all software is available for ppc64le ?

And it seems it is https://hub.docker.com/r/ppc64le/ubuntu/

At least Server: https://ubuntu.com/download/server/power

You should be able to build kindest/base yourself, using the Dockerfile provided by the project...

[runlevel5]

@afbjorklund yes I could build the kindest/base myself. It is just a bit inconvenient though. I am hopeful for a 1st-class support though...

[afbjorklund]

It will always be weird though, like it has been ever since Yellow Dog Linux. But the PowerPC CPU architeture was so much nicer than x86.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[runlevel5]

/remove-lifecycle rotten

[priyawadhwa]

Hey @runlevel5 the kic base image now uses ubuntu instead of the kind base image. It may be easier to build the image for your architecture now.

minikube/deploy/kicbase/Dockerfile

Line 22 in 717b738

FROM ubuntu:focal-20200423

[runlevel5]

@priyawadhwa thanks for letting me know