Kyuubi Fails to retrieve application information 1Hour

[moelhoussein]

Hello Team, we are running kyuubi 1.9.2 on AKS 1.30.4 we are noticing that batch jobs are reported PENDING then ERROR after roughly one hour. A Pod restart clear this

In the logs we see:

2025-01-03 14:03:22.284 DEBUG org.apache.kyuubi.engine.KubernetesApplicationOperation: Getting application[label: kyuubi-unique-tag=1d398b12-5d32-41bf-915e-ada09f5519e1]'s info from Kubernetes cluster
2025-01-03 14:03:22.284 WARN org.apache.kyuubi.engine.KubernetesApplicationOperation: Waiting for driver pod with label: kyuubi-unique-tag=1d398b12-5d32-41bf-915e-ada09f5519e1 to be created, elapsed time: 0ms, return UNKNOWN status

Spark relevant configs:

  sparkDefaults: |
    spark.master=k8s://**redacted**.hcp.eastus2.azmk8s.io:443
    spark.submit.deployMode=cluster
    spark.kubernetes.file.upload.path=abfss://[email protected]/fileupload
    spark.kubernetes.namespace=kyuubi-dv1
    spark.kubernetes.authenticate.driver.serviceAccountName=kyuubi-dv1

  kyuubiDefaults: |
    kyuubi.kubernetes.master.address=https://**redacted**.hcp.eastus2.azmk8s.io:443
    kyuubi.kubernetes.namespace=kyuubi-dv1
    kyuubi.kubernetes.authenticate.driver.serviceAccountName=kyuubi-dv1
    kyuubi.kubernetes.trust.certificates=true
    kyuubi.kubernetes.application.state.source=CONTAINER
    kyuubi.kubernetes.authenticate.oauthTokenFile=/var/run/secrets/kubernetes.io/serviceaccount/token

So far this behavior is reproducible 100% of the time. Any thoughts on what might be causing this?

[pan3793]

It's likely caused by the Kyuubi Pod token expires after 1 hour

kyuubi.kubernetes.authenticate.oauthTokenFile=/var/run/secrets/kubernetes.io/serviceaccount/token

[moelhoussein]

I was able to do some tracing, and I see this indeed due to token expiry. Doesn't fabric8 has a token refresh mechanism since 6.1.x?

[pan3793]

I guess this is a Kyuubi implementation issue, we just read the content from the kyuubi.kubernetes.authenticate.oauthTokenFile and call ConfigBuilder.withOauthToken, I guess this approach does not support token refresh...

[pan3793]

@moelhoussein the code at org.apache.kyuubi.util.KubernetesUtils, I guess replacing the withOauthToken with withOauthTokenProvider should fix your issue

[turboFei]

Raised #6883 to address it.