Cannot bind volume on EKS

[alexnimo]

What happened:

Tried to install apiclarity on EKS version 1.24. Tried it on managed node group and on Fargate.
The ebs-csi controller is provisioned.
Both of the apiclarity pods are stuck on pending and return the following error:
Warning FailedScheduling 9m33s default-scheduler running PreBind plugin "VolumeBinding": binding volumes: timed out waiting for the condition

What you expected to happen:

Pods to start normally

How to reproduce it (as minimally and precisely as possible):

EKS cluster version 1.24

Are there any error messages in API Clarity logs?

Warning FailedScheduling 9m33s default-scheduler running PreBind plugin "VolumeBinding": binding volumes: timed out waiting for the condition

Anything else we need to know?:

Environment:

• Kubernetes version 1.24
• Istio version (use istioctl version):
  client version: 1.16.1
  control plane version: 1.11.4
  data plane version: 1.11.4 (1 proxies)
• APIClarity version (use kubectl -n apiclarity exec deploy/apiclarity -- ./backend version)
• Cloud provider or hardware configuration:
• AWS + EKS
• Others:

[akpsgit]

Hi @alexnimo, can this be related to https://stackoverflow.com/questions/68725070/k8s-pods-failure-error-while-running-volumebinding-prebind-plugin-for-pod-a?

[alexnimo]

Hi @akpsgit ,
Nope, it looks like something else.
The error message on the PVC is:
Normal Provisioning 3m17s (x688 over 42h) ebs.csi.aws.com_ebs-csi-controller-7bffdbb6d6-rt8td_0f667e1c-20f9-49ab-8ae9-db2b72131365 External provisioner is provisioning volume for claim "apiclarity/data-apiclarity-apiclarity-postgresql-0"
Normal ExternalProvisioning 2m37s (x10264 over 42h) persistentvolume-controller waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
OIDC was already in use and working with different pods so not sure it's related....

[pkalapat]

To resolve this issue it needs Amazon EBS CSI driver as an Amazon EKS add-on and setting up the driver with relevant IAMServiceAccount roles and policies. Post eks1.23 it is required to install add ons. Some useful references

https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html
https://docs.aws.amazon.com/eks/latest/userguide/csi-iam-role.html

First uninistall your current installation of API Clarity with helm
helm uninstall apiclarity -n apiclarity

Next, follow the instructions provided in the above links to install CSI add ons in your cluster and give it IAMServiceRole with associated trust policy

Now re-try apiclarity install

If this does not work you may have to delete your existing cluster, configure with add on and then attempt apiclarity install.