Clearlinux docker images does not unpack in containerd.

[lucidprogrammer]

This issue is limited to clearlinux containers running on containerd environments. This is a serious blocker.

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: clear-deployment
  namespace: openfaas-fn
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: clear  
    spec:                 
      containers: 
      - name: clear 
        image: clearlinux/python:3.7
        command: ["python"]
        args: ["-m","http.server"]
        imagePullPolicy: Always
        ports:
        - containerPort: 8000

kubectl -n openfaas-fn describe pod -l app=clear

Name:               clear-deployment-76b4868b98-mgvtj
Namespace:          openfaas-fn
Priority:           2000001000
PriorityClassName:  system-node-critical
Node:               fargate-ip-192-168-99-171.ec2.internal/192.168.99.171
Start Time:         Mon, 17 Feb 2020 01:34:45 +0000
Labels:             app=clear
                    eks.amazonaws.com/fargate-profile=fargate-openfaas-fn
                    pod-template-hash=76b4868b98
Annotations:        kubernetes.io/psp: eks.privileged
Status:             Pending
IP:                 192.168.99.171
Controlled By:      ReplicaSet/clear-deployment-76b4868b98
Containers:
  clear:
    Container ID:  
    Image:         clearlinux/python:3.7
    Image ID:      
    Port:          8000/TCP
    Host Port:     0/TCP
    Command:
      python
    Args:
      -m
      http.server
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-c6v5b (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  default-token-c6v5b:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-c6v5b
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                From                                             Message
  ----     ------     ----               ----                                             -------
  Normal   Scheduled  <unknown>          fargate-scheduler                                Successfully assigned openfaas-fn/clear-deployment-76b4868b98-mgvtj to fargate-ip-192-168-99-171.ec2.internal
  Warning  Failed     75s                kubelet, fargate-ip-192-168-99-171.ec2.internal  Failed to pull image "clearlinux/python:3.7": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/clearlinux/python:3.7": unpack: failed to extract layer sha256:baa90903a66df167bb3f28a20e040afe5c7cde54de5824ccf74032270d200778: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount493119581: chmod /var/lib/containerd/tmpmounts/containerd-mount493119581/usr/bin/b2sum: no such file or directory: unknown
  Warning  Failed     59s                kubelet, fargate-ip-192-168-99-171.ec2.internal  Failed to pull image "clearlinux/python:3.7": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/clearlinux/python:3.7": unpack: failed to extract layer sha256:baa90903a66df167bb3f28a20e040afe5c7cde54de5824ccf74032270d200778: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount084365975: chmod /var/lib/containerd/tmpmounts/containerd-mount084365975/usr/bin/b2sum: no such file or directory: unknown
  Normal   Pulling    29s (x3 over 77s)  kubelet, fargate-ip-192-168-99-171.ec2.internal  Pulling image "clearlinux/python:3.7"
  Warning  Failed     29s (x3 over 75s)  kubelet, fargate-ip-192-168-99-171.ec2.internal  Error: ErrImagePull
  Warning  Failed     29s                kubelet, fargate-ip-192-168-99-171.ec2.internal  Failed to pull image "clearlinux/python:3.7": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/clearlinux/python:3.7": unpack: failed to extract layer sha256:baa90903a66df167bb3f28a20e040afe5c7cde54de5824ccf74032270d200778: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount214157025: chmod /var/lib/containerd/tmpmounts/containerd-mount214157025/usr/bin/b2sum: no such file or directory: unknown
  Normal   BackOff    4s (x4 over 74s)   kubelet, fargate-ip-192-168-99-171.ec2.internal  Back-off pulling image "clearlinux/python:3.7"
  Warning  Failed     4s (x4 over 74s)   kubelet, fargate-ip-192-168-99-171.ec2.internal  Error: ImagePullBackOff

[qzheng527]

I tired on my side it was working on my Clear OS. Which OS are you using? @lucidprogrammer

[ZhongbaoShi]

What's your containerd version? Could you enable debug and paste the containerd log?

ctr --debug images pull

[mythi]

We should move this to clearlinux/distribution/issues/1765