Pods with unready Containers exist on this node, we can't clean the slots yet

[rpieczon]

Describe the bug

Akri agent daemonset keeps reporting following error whenever any of pod running on a cluster is not ready.

2023-11-16T13:44:46Z TRACE agent::util::slot_reconciliation] reconcile - Pods with unready Containers exist on this node, we can't clean the slots yet

In my case failing POD doesn't use USB resources.

Output of kubectl get pods,akrii,akric -o wide

lpfe04@f1725b929a:~$ kubectl get pod,akrii,akric -n akri
NAME                                              READY   STATUS    RESTARTS   AGE
pod/akri-agent-daemonset-9gwl2                    1/1     Running   0          10m
pod/akri-controller-deployment-7c6455f79-zt779    1/1     Running   0          11m
pod/akri-udev-discovery-daemonset-2d9hp           1/1     Running   0          10m
pod/akri-webhook-configuration-7bf6656b45-mclth   1/1     Running   0          11m

NAME                                  CONFIG        SHARED   NODES            AGE
instance.akri.sh/gsm-dongle-6e977d    gsm-dongle    false    ["f1725b929a"]   10m
instance.akri.sh/wifi-dongle-254c38   wifi-dongle   false    ["f1725b929a"]   10m
instance.akri.sh/wifi-dongle-ac917e   wifi-dongle   false    ["f1725b929a"]   10m

NAME                                CAPACITY   AGE
configuration.akri.sh/gsm-dongle    1          11h
configuration.akri.sh/wifi-dongle   1          11h

Kubernetes Version: [e.g. Native Kubernetes 1.19, MicroK8s 1.19, Minikube 1.19, K3s]

kubernetes: v1.26.8+rke2r1"

Expected behavior

I would expect reconciliation process can be continue if failing pod is out of usb usage.management context.

[kate-goldenring]

@rpieczon just to clarify, are you saying if any pod (even if unassociated with Akri) is unready, it causes this slot reconciliation error? From what i remember slot reconciliation should only check pods with an expected annotation.

[bfjelds]

i lose track a little, but the annotations are on the container, not the pod i think ... and it might be that an unready pod is considered a potential place where an annotated container could eventually exist. might be worth looking at the resource requests to limit where this early exit happens.

[bfjelds]

might be hard to check for the resource though. if the pod isn't ready and the container doesn't exist, there isn't much context to check the instances against.

[rpieczon]

@rpieczon just to clarify, are you saying if any pod (even if unassociated with Akri) is unready, it causes this slot reconciliation error? From what i remember slot reconciliation should only check pods with an expected annotation.

Exactly in my case I have failing Prometheus POD which has zero requirements related with USB allocation.

[rpieczon]

Any update on it?