failed to generate spec: path "/tmp" is mounted on "/" but it is not a shared mount

[tekumara]

Error: failed to generate container "73c7a50781eaf5de74ab1f95568a7bc7e26016fd4a06aa28bf8ea2a79be3f9dd" spec: failed to generate spec: path "/tmp" is mounted on "/" but it is not a shared mount

Expand below to see more.

kubectl describe pods -n kube-image builder-btls

Name:         builder-btlsg
Namespace:    kube-image
Priority:     0
Node:         k3d-kim-server-0/172.22.0.2
Start Time:   Sun, 27 Mar 2022 10:49:58 +1100
Labels:       app=kim
              app.kubernetes.io/component=builder
              app.kubernetes.io/managed-by=kim
              app.kubernetes.io/name=kim
              component=builder
              controller-revision-hash=7bb6779b98
              pod-template-generation=1
Annotations:  <none>
Status:       Pending
IP:           172.22.0.2
IPs:
  IP:           172.22.0.2
Controlled By:  DaemonSet/builder
Init Containers:
  rshared-tmp:
    Container ID:  containerd://949bd0c0307b7e9bd307fe6fdc154baac68c2807843aef74914294af5c622087
    Image:         docker.io/moby/buildkit:v0.8.3
    Image ID:      docker.io/moby/buildkit@sha256:171689e43026533b48701ab6566b72659dd1839488d715c73ef3fe387fab9a80
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
    Args:
      (if mountpoint $_DIR; then set -x; nsenter -m -p -t 1 -- env PATH=$_PATH sh -c 'mount --make-rshared $_DIR'; fi) || true
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sun, 27 Mar 2022 10:50:16 +1100
      Finished:     Sun, 27 Mar 2022 10:50:16 +1100
    Ready:          True
    Restart Count:  0
    Environment:
      _DIR:   /tmp
      _PATH:  /usr/sbin:/usr/bin:/sbin:/bin:/bin/aux
    Mounts:
      /tmp from host-tmp (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-f4cz4 (ro)
  rshared-buildkit:
    Container ID:  containerd://d61fd8440158f9e282cbcd0fcf77fa2c24e1e5826c3644e97b7bbe6cf82eb944
    Image:         docker.io/moby/buildkit:v0.8.3
    Image ID:      docker.io/moby/buildkit@sha256:171689e43026533b48701ab6566b72659dd1839488d715c73ef3fe387fab9a80
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
    Args:
      (if mountpoint $_DIR; then set -x; nsenter -m -p -t 1 -- env PATH=$_PATH sh -c 'mount --make-rshared $_DIR'; fi) || true
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sun, 27 Mar 2022 10:50:16 +1100
      Finished:     Sun, 27 Mar 2022 10:50:16 +1100
    Ready:          True
    Restart Count:  0
    Environment:
      _DIR:   /var/lib/buildkit
      _PATH:  /usr/sbin:/usr/bin:/sbin:/bin:/bin/aux
    Mounts:
      /var/lib/buildkit from host-var-lib-buildkit (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-f4cz4 (ro)
  rshared-containerd:
    Container ID:  containerd://49825e4b0231bfbd98f166208902e31a07c80f47ee2b754a17f3c9cdcef93a5c
    Image:         docker.io/moby/buildkit:v0.8.3
    Image ID:      docker.io/moby/buildkit@sha256:171689e43026533b48701ab6566b72659dd1839488d715c73ef3fe387fab9a80
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
    Args:
      (if mountpoint $_DIR; then set -x; nsenter -m -p -t 1 -- env PATH=$_PATH sh -c 'mount --make-rshared $_DIR'; fi) || true
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sun, 27 Mar 2022 10:50:17 +1100
      Finished:     Sun, 27 Mar 2022 10:50:17 +1100
    Ready:          True
    Restart Count:  0
    Environment:
      _DIR:   /var/lib/rancher
      _PATH:  /usr/sbin:/usr/bin:/sbin:/bin:/bin/aux
    Mounts:
      /var/lib/rancher from host-containerd (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-f4cz4 (ro)
Containers:
  buildkit:
    Container ID:  
    Image:         docker.io/moby/buildkit:v0.8.3
    Image ID:      
    Port:          1234/TCP
    Host Port:     1234/TCP
    Args:
      --addr=unix:///run/buildkit/buildkitd.sock
      --addr=tcp://0.0.0.0:1234
      --containerd-worker=true
      --containerd-worker-addr=/run/k3s/containerd/containerd.sock
      --containerd-worker-gc
      --oci-worker=false
      --tlscacert=/certs/ca/tls.crt
      --tlscert=/certs/server/tls.crt
      --tlskey=/certs/server/tls.key
    State:          Waiting
      Reason:       CreateContainerError
    Ready:          False
    Restart Count:  0
    Liveness:       exec [buildctl debug workers] delay=5s timeout=1s period=20s #success=1 #failure=3
    Readiness:      exec [buildctl debug workers] delay=5s timeout=1s period=20s #success=1 #failure=3
    Environment:    <none>
    Mounts:
      /certs/ca from certs-ca (ro)
      /certs/server from certs-server (ro)
      /run from host-run (rw)
      /sys/fs/cgroup from host-ctl (rw)
      /tmp from host-tmp (rw)
      /var/lib/buildkit from host-var-lib-buildkit (rw)
      /var/lib/rancher from host-containerd (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-f4cz4 (ro)
  agent:
    Container ID:  
    Image:         rancher/kim:v0.1.0-beta.7
    Image ID:      
    Port:          1233/TCP
    Host Port:     1233/TCP
    Command:
      kim
      --debug
      agent
    Args:
      --agent-port=1233
      --buildkit-socket=unix:///run/buildkit/buildkitd.sock
      --buildkit-port=1234
      --containerd-socket=/run/k3s/containerd/containerd.sock
      --tlscacert=/certs/ca/tls.crt
      --tlscert=/certs/server/tls.crt
      --tlskey=/certs/server/tls.key
    State:          Waiting
      Reason:       CreateContainerError
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /certs/ca from certs-ca (ro)
      /certs/server from certs-server (ro)
      /etc/pki from host-etc-pki (ro)
      /etc/ssl from host-etc-ssl (ro)
      /run from host-run (rw)
      /sys/fs/cgroup from host-ctl (rw)
      /var/lib/buildkit from host-var-lib-buildkit (rw)
      /var/lib/rancher from host-containerd (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-f4cz4 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  host-ctl:
    Type:          HostPath (bare host directory volume)
    Path:          /sys/fs/cgroup
    HostPathType:  Directory
  host-etc-pki:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/pki
    HostPathType:  DirectoryOrCreate
  host-etc-ssl:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/ssl
    HostPathType:  DirectoryOrCreate
  host-run:
    Type:          HostPath (bare host directory volume)
    Path:          /run
    HostPathType:  Directory
  host-tmp:
    Type:          HostPath (bare host directory volume)
    Path:          /tmp
    HostPathType:  Directory
  host-var-lib-buildkit:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/buildkit
    HostPathType:  DirectoryOrCreate
  host-containerd:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/rancher
    HostPathType:  DirectoryOrCreate
  certs-ca:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kim-tls-ca
    Optional:    false
  certs-server:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kim-tls-server
    Optional:    false
  kube-api-access-f4cz4:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              node-role.kubernetes.io/builder=true
Tolerations:                 node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/network-unavailable:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  3m36s                  default-scheduler  Successfully assigned kube-image/builder-btlsg to k3d-kim-server-0
  Normal   Pulling    3m37s                  kubelet            Pulling image "docker.io/moby/buildkit:v0.8.3"
  Normal   Pulled     3m20s                  kubelet            Successfully pulled image "docker.io/moby/buildkit:v0.8.3" in 16.8479912s
  Normal   Created    3m19s                  kubelet            Created container rshared-buildkit
  Normal   Created    3m19s                  kubelet            Created container rshared-tmp
  Normal   Started    3m19s                  kubelet            Started container rshared-tmp
  Normal   Started    3m19s                  kubelet            Started container rshared-buildkit
  Normal   Pulled     3m19s                  kubelet            Container image "docker.io/moby/buildkit:v0.8.3" already present on machine
  Normal   Pulled     3m18s                  kubelet            Container image "docker.io/moby/buildkit:v0.8.3" already present on machine
  Normal   Created    3m18s                  kubelet            Created container rshared-containerd
  Normal   Started    3m18s                  kubelet            Started container rshared-containerd
  Normal   Pulling    3m17s                  kubelet            Pulling image "rancher/kim:v0.1.0-beta.7"
  Warning  Failed     3m17s                  kubelet            Error: failed to generate container "73c7a50781eaf5de74ab1f95568a7bc7e26016fd4a06aa28bf8ea2a79be3f9dd" spec: failed to generate spec: path "/tmp" is mounted on "/" but it is not a shared mount
  Normal   Pulled     3m6s                   kubelet            Successfully pulled image "rancher/kim:v0.1.0-beta.7" in 10.3655793s
  Warning  Failed     3m6s                   kubelet            Error: failed to generate container "c7913044e35bbb8ef948e2bd17848cb308888cb4dbeddc97d08bfad073d08853" spec: failed to generate spec: path "/var/lib/buildkit" is mounted on "/" but it is not a shared mount
  Warning  Failed     3m6s                   kubelet            Error: failed to generate container "4f9180c6190fedcd9601131d41b1ca48160330bbef6b40ca9b0fd2cbd0bae24c" spec: failed to generate spec: path "/tmp" is mounted on "/" but it is not a shared mount
  Warning  Failed     3m6s                   kubelet            Error: failed to generate container "68a8bf3b5550081f11d07d1c5e614eb5024f65633d60b7a4d78913c15f10d091" spec: failed to generate spec: path "/var/lib/buildkit" is mounted on "/" but it is not a shared mount
  Warning  Failed     2m55s                  kubelet            Error: failed to generate container "356515b2435fc6ec28c8f3e7be405fc22a64d10b6b423ea17342e6cf30c7b823" spec: failed to generate spec: path "/tmp" is mounted on "/" but it is not a shared mount
  Normal   Pulled     2m55s (x2 over 3m6s)   kubelet            Container image "rancher/kim:v0.1.0-beta.7" already present on machine
  Warning  Failed     2m55s                  kubelet            Error: failed to generate container "0031efc0dc317b721e73d3b500b5b006730b16d0f354538cf5c7d728daafd802" spec: failed to generate spec: path "/var/lib/buildkit" is mounted on "/" but it is not a shared mount
  Normal   Pulled     2m43s (x4 over 3m17s)  kubelet            Container image "docker.io/moby/buildkit:v0.8.3" already present on machine
  Warning  Failed     2m43s                  kubelet            Error: failed to generate container "4f60fc92c2b125fba4279390bfcd6a6255e4e0d8faec90838d49013b2c52b04a" spec: failed to generate spec: path "/tmp" is mounted on "/" but it is not a shared mount

[tekumara]

$ kubectl -n kube-image logs builder-btlsg rshared-tmp
/tmp is a mountpoint
+ nsenter -m -p -t 1 -- env 'PATH=/usr/sbin:/usr/bin:/sbin:/bin:/bin/aux' sh -c 'mount --make-rshared $_DIR'
mount: /tmp: Invalid argument

$ kubectl -n kube-image logs builder-btlsg rshared-buildkit
/var/lib/buildkit is a mountpoint
+ nsenter -m -p -t 1 -- env 'PATH=/usr/sbin:/usr/bin:/sbin:/bin:/bin/aux' sh -c 'mount --make-rshared $_DIR'
mount: /var/lib/buildkit: Invalid argument

$ kubectl -n kube-image logs builder-btlsg rshared-containerd
/var/lib/rancher is a mountpoint
+ nsenter -m -p -t 1 -- env 'PATH=/usr/sbin:/usr/bin:/sbin:/bin:/bin/aux' sh -c 'mount --make-rshared $_DIR'
mount: /var/lib/rancher: Invalid argument

[tekumara]

FYI I'm using k3s in k3d, and have the same issue on macos and ubuntu.

$ k3d --version           
k3d version v5.3.0
k3s version v1.22.6-k3s1 (default)

[ashlineldridge]

I am also seeing this issue. I'm trying to create the simplest k3d/k3s cluster and perform a kim build on it. If I switch to kind it works. I can reproduce using the following:

> k3d cluster create all-in-one -p 1233:1233@loadbalancer -p 1234:1234@loadbalancer
...

> ./bin/kim builder install --endpoint-addr 127.0.0.1
...
INFO[0074] Waiting on builder daemon availability...
INFO[0081] Waiting on builder daemon availability...
Error: timeout waiting for builder to become available

> kubectl get pods -n kube-image
NAME            READY   STATUS                 RESTARTS   AGE
builder-t6n7j   0/2     CreateContainerError   0          34m

If I look at the status of the failed pod, I see:

  containerStatuses:
  - image: rancher/kim:v0.1.0-beta.7
    imageID: ""
    lastState: {}
    name: agent
    ready: false
    restartCount: 0
    started: false
    state:
      waiting:
        message: 'failed to generate container "90cbca9c4fc190572d3e0f8488aea95c26b0ef03310f2ac9f8431ca331359413"
          spec: failed to generate spec: path "/var/lib/buildkit" is mounted on "/"
          but it is not a shared mount'
        reason: CreateContainerError
  - image: docker.io/moby/buildkit:v0.8.3
    imageID: ""
    lastState: {}
    name: buildkit
    ready: false
    restartCount: 0
    started: false
    state:
      waiting:
        message: 'failed to generate container "657848459e41fa6fa8f5d444deb509d370fa0f7163584e7a9353bf1380e213c3"
          spec: failed to generate spec: path "/tmp" is mounted on "/" but it is not
          a shared mount'
        reason: CreateContainerError

My system information is as follows:

macOS: 12.3.1
kim: v0.1.0-beta.7 (e597b9564b47213734787b3e0c540a635b250bbf)
k3d: v5.4.3
k3s: v1.23.6-k3s1 (default)

[tekumara]

oh hai @ashlineldridge! 👋

[ashlineldridge]

@tekumara Hey Oliver! I didn't even recognise your username!

[dweomer]

I last tested this with k3d v4.4.3 without issue. While it is possible that something in the k3s image has changed I think it more likely that k3d 5.x has changed how /tmp is setup for the k3s container(s). It is possible that I have cheated in making this work on my dev box by modifying / to be rshared.