Kubernetes Authentication failure 61 minutes after starting.

[cpressland]

Which project are you reporting a bug for?

kr8s

What happened?

When using kr8s as part of a FastAPI application authentication begins failing 61 minutes after starting.

Sample code:

from pathlib import Path

from fastapi import FastAPI, status
from fastapi.responses import JSONResponse, Response
from kr8s.objects import ConfigMap

app = FastAPI()

def get_current_namespace() -> str:
    """Get the current namespace."""
    return Path("/var/run/secrets/kubernetes.io/serviceaccount/namespace").read_text().strip()


configmap = ConfigMap("example", namespace=get_current_namespace())
if not configmap.exists():
    configmap.create()

@app.get("/health/readiness")
def readiness() -> Response:
    """Readiness check endpoint, returns a JSON response with status 'ok'."""
    configmap.refresh()
    if configmap.exists():
        return JSONResponse({"status": "ok"}, status_code=status.HTTP_200_OK)
    return JSONResponse({"status": "error"}, status_code=status.HTTP_503_SERVICE_UNAVAILABLE)


@app.get("/health/liveness")
def liveness() -> Response:
    """Liveness check endpoint, returns a JSON response with status 'ok'."""
    return JSONResponse({"status": "ok"}, status_code=status.HTTP_200_OK)

Logs:

First health check:

2024-12-23T11:31:13.528988544Z INFO:     Started server process [1]
2024-12-23T11:31:13.529025895Z INFO:     Waiting for application startup.
2024-12-23T11:31:13.529142835Z INFO:     Application startup complete.
2024-12-23T11:31:13.529409517Z INFO:     Uvicorn running on http://0.0.0.0:6502 (Press CTRL+C to quit)
2024-12-23T11:31:38.700639717Z INFO:     10.40.16.115:49578 - "GET /health/liveness HTTP/1.1" 200 OK
2024-12-23T11:31:38.743960452Z INFO:     10.40.16.115:49580 - "GET /health/readiness HTTP/1.1" 200 OK

Failing health check 61 minutes later:

2024-12-23T12:32:38.699077059Z INFO:     10.40.16.115:47082 - "GET /health/liveness HTTP/1.1" 200 OK
2024-12-23T12:32:38.794716051Z INFO:     10.40.16.115:47084 - "GET /health/readiness HTTP/1.1" 503 Service Unavailable

Exception when calling an endpoint that forces a configmap.refresh():

ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/app/lib/python3.13/site-packages/kr8s/_api.py", line 168, in call_api
    response.raise_for_status()
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/app/lib/python3.13/site-packages/httpx/_models.py", line 829, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '401 Unauthorized' for url 'https://172.16.0.1/api/v1/namespaces/sealed-secrets-tools/configmaps/cluster-certificates'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/401

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/lib/python3.13/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        self.scope, self.receive, self.send
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/app/lib/python3.13/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/lib/python3.13/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/app/lib/python3.13/site-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/app/lib/python3.13/site-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/app/lib/python3.13/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/app/lib/python3.13/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/app/lib/python3.13/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/app/lib/python3.13/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/app/lib/python3.13/site-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/app/lib/python3.13/site-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/app/lib/python3.13/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/app/lib/python3.13/site-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/app/lib/python3.13/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/app/lib/python3.13/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/app/lib/python3.13/site-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/app/lib/python3.13/site-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<3 lines>...
    )
    ^
  File "/app/lib/python3.13/site-packages/fastapi/routing.py", line 214, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/lib/python3.13/site-packages/starlette/concurrency.py", line 39, in run_in_threadpool
    return await anyio.to_thread.run_sync(func, *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/lib/python3.13/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        func, args, abandon_on_cancel=abandon_on_cancel, limiter=limiter
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/app/lib/python3.13/site-packages/anyio/_backends/_asyncio.py", line 2505, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/app/lib/python3.13/site-packages/anyio/_backends/_asyncio.py", line 1005, in run
    result = context.run(func, *args)
  File "/app/lib/python3.13/site-packages/sealed_secrets_tools/server/__init__.py", line 119, in get_certificates
    return JSONResponse({"clusters": list(get_configmap().keys())}, status_code=status.HTTP_200_OK)
                                          ~~~~~~~~~~~~~^^
  File "/app/lib/python3.13/site-packages/sealed_secrets_tools/server/__init__.py", line 33, in get_configmap
    configmap.refresh()
    ~~~~~~~~~~~~~~~~~^^
  File "/app/lib/python3.13/site-packages/kr8s/_async_utils.py", line 120, in run_sync_inner
    return portal.call(wrapped)
           ~~~~~~~~~~~^^^^^^^^^
  File "/app/lib/python3.13/site-packages/kr8s/_async_utils.py", line 92, in call
    return self._portal.call(func, *args, **kwargs)
           ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/lib/python3.13/site-packages/anyio/from_thread.py", line 290, in call
    return cast(T_Retval, self.start_task_soon(func, *args).result())
                          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/root/.local/share/uv/python/cpython-3.13.1-linux-x86_64-gnu/lib/python3.13/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ~~~~~~~~~~~~~~~~~^^
  File "/root/.local/share/uv/python/cpython-3.13.1-linux-x86_64-gnu/lib/python3.13/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/app/lib/python3.13/site-packages/anyio/from_thread.py", line 221, in _call_func
    retval = await retval_or_awaitable
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/lib/python3.13/site-packages/kr8s/_objects.py", line 357, in refresh
    await self.async_refresh()
  File "/app/lib/python3.13/site-packages/kr8s/_objects.py", line 375, in async_refresh
    raise e
  File "/app/lib/python3.13/site-packages/kr8s/_objects.py", line 363, in async_refresh
    async with self.api.call_api(
               ~~~~~~~~~~~~~~~~~^
        "GET",
        ^^^^^^
    ...<2 lines>...
        namespace=self.namespace,
        ^^^^^^^^^^^^^^^^^^^^^^^^^
    ) as resp:
    ^
  File "/root/.local/share/uv/python/cpython-3.13.1-linux-x86_64-gnu/lib/python3.13/contextlib.py", line 214, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/app/lib/python3.13/site-packages/kr8s/_api.py", line 186, in call_api
    raise ServerError(
        error_message, status=error, response=e.response
    ) from e
kr8s._exceptions.ServerError: Unauthorized

Anything else?

No response

[jacobtomlinson]

How is your application authenticating with Kubernetes? Is it using a service account, or something else like OIDC? This could be related to #125

[cpressland]

Thanks @jacobtomlinson - authentication is entirely driven by Service Accounts, with the app only getting ConfigMaps and Secrets from the same namespace.