Deleted nodes are not removed from PVC

[HarleyB123]

What happened:

I am using cluster-autoscaler in an AKS cluster to create new nodes as scale increases. On the occasion where pods (backed by statefulsets) are being added to the cluster whilst nodes are being removed, a pod will stay stuck in "Pending" as the PVC has not binded. When I edit the PVC, the volume.kubernetes.io/selected-node annotation is using a node that has been deleted. Removing this annotation fixes the problem.

What you expected to happen:

The CSI Driver (if it is responsible for doing so) should detect if a PVC has an annotation with a node that no longer exists and remove it.

How to reproduce it:

Create loads of pods on a scalable AKS cluster, remove enough pods to scale down nodes whilst adding a few pods to the cluster at the same time.

Anything else we need to know?:

The issue here relates to this, though this comment appears to suggest that the CSI driver could implement logic that would solve this issue

Environment:

• CSI Driver version: v1.20.0
• Kubernetes version (use kubectl version): 1.21.4

[edreed]

The issue here relates to this, though this comment appears to suggest that the CSI driver could implement logic that would solve this issue

@HarleyB123 This is not my understanding of this thread, but rather it seems to suggest handling this condition in the external-provisioner. kubernetes-csi/external-provisioner#544 is tracking this issue now.

Having the CSI driver remove the annotation is problematic for several reasons:

1. CSI drivers are not required to have knowledge of or access to Kubernetes objects directly.
2. The volume.kubernetes.io/selected-node is managed by the Kubernetes PersistentVolume infrastructure. Its usage may change or be removed in the future. Updates to this annotation could cause problems in the future as its usage changes (and potentially this issue addressed holistically).
3. Each CSI driver would have to address this condition individually. It is better handled by the infrastructure so all CSI drivers can benefit.

[HarleyB123]

Ah understood, thanks for explaining the limitations in the CSI driver handling this logic!