Kong unable to fetch JWT credentials

[Anupam5972]

For the Versions

• Kong: 3.4.2
• Kong-ingress-controller: 2.12.3

Deployed Using helm Charts

The credentials are simultaneously created while the kong is deployed.

I am not facing this error for Version 2.10.5 but any version above that from 2.11.0 I am facing this issue

time="2024-03-25T14:04:51Z" level=error msg="resource processing failed: credential \"kong-jwt-credential\" failure: failed to fetch secret: Secret XXXXX/kong-jwt-credential not found" GVK="configuration.konghq.com/v1, Kind=KongConsumer" name=jwt-consumer namespace=XXXXX
time="2024-03-25T14:04:54Z" level=error msg="resource processing failed: credential \"kong-jwt-credential\" failure: failed to fetch secret: Secret XXXXX/kong-jwt-credential not found" GVK="configuration.konghq.com/v1, Kind=KongConsumer" name=jwt-consumer namespace=XXXXX.```

[rainest]

It looks like you upgraded from an older version, did you update your CRDs?

Though we don't have a full understanding of why it happens, we have previously seen this issue where the controller becomes unable to fetch resources when using an outdated set of CRDs.

[Anupam5972]

I have my CRDs upto date, I rechecked it again and yeah it is

[Anupam5972]

I'm using the controller-gen.kubebuilder.io/version: v0.13.0
I used Helm to deploy it chart version "kong-2.28.0"
I want to Upgrade the KIC version from 2.10.5 to 2.12.3 but it's not fetching the credentials from 2.11.x itself

[davidmontoyago]

Hi everyone, our team is also experiencing this issue after upgrading from KIC 2.3 to 2.12.0 and 2.12.3. In our case it happens for all types of credentials. That is, JWT credentials, API keys and ACL groups. The ingress controller container just seems unable to read any k8s Secrets. We've verified the CRDs version and they match version 0.7.0 as seen below:

$ kubectl get crd kongconsumers.configuration.konghq.com -ojson | grep '"controller-gen.kubebuilder.io/version":'
"controller-gen.kubebuilder.io/version": "v0.7.0"

We've also verified that the Ingress controller's Service Account has enough permissions to fetch k8s Secrets in any namespace.

This is a long shot. Could this be related to the new way the Service Account token is now mounted with a projected volume? I'm thinking maybe the k8s client is not getting initialized with the proper SA token.

Cross-referencing Kong/kubernetes-ingress-controller#5784 and Kong/kubernetes-ingress-controller#5710 as they seem related.