New behavior in kopf 1.37.2 causes kopf to crash with "Ran out of valid credentials"

[jvandemark]

Long story short

Regarding the new behavior of kopf/_cogs/clients/auth.py as it relates to kopf/_cogs/structs/credentials.py expire: the service account token expiration causes kopf to crash. Our application runs with 1.37.1, but crashes after an hour when running 1.37.2.

Existing observation/queueing/watching/api objects stack after an hour with "Ran out of valid credentials" see logs below.

I assume this is related to the code changes in auth.py which is explicitly closing all responses (aiohttp.ClientResponse) prior to re-athentication occurring, or being called. This is really the only code change between 1.37.1 and 1.37.2

Kopf version

1.37.2

Kubernetes version

1.30 AKS

Python version

3.12.8

Code

Logs

[2025-01-29 12:03:23,751] kopf.activities.star [INFO    ] Activity 'startup_tasks' succeeded.
[2025-01-29 12:03:23,751] kopf._core.engines.a [INFO    ] Initial authentication has been initiated.
[2025-01-29 12:03:23,752] kopf.activities.auth [INFO    ] Activity 'login_via_client' succeeded.
[2025-01-29 12:03:23,752] kopf._core.engines.a [INFO    ] Initial authentication has finished.

...

[2025-01-29 13:06:34,041] kopf._core.engines.a [INFO    ] Re-authentication has been initiated.
[2025-01-29 13:06:34,043] kopf.activities.auth [INFO    ] Activity 'login_via_client' succeeded.
[2025-01-29 13:06:34,043] kopf._core.engines.a [INFO    ] Re-authentication has finished.
[2025-01-29 13:06:34,044] kopf._core.reactor.r [ERROR   ] Namespace observer has failed: Ran out of valid credentials. Consider installing an API client library or adding a login handler. See more: https://kopf.readthedocs.io/en/stable/authentication/Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/aiokits/aiotasks.py", line 96, in guard
    await coro
  File "/usr/local/lib/python3.12/site-packages/kopf/_core/reactor/observation.py", line 77, in namespace_observer
    await queueing.watcher(
  File "/usr/local/lib/python3.12/site-packages/kopf/_core/reactor/queueing.py", line 175, in watcher
    async for raw_event in stream:
  File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/clients/watching.py", line 86, in infinite_watch
    async for raw_event in stream:
  File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/clients/watching.py", line 201, in continuous_watch
    async for raw_input in stream:
  File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/clients/watching.py", line 266, in watch_objs
    async for raw_input in api.stream(
  File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/clients/api.py", line 201, in stream
    response = await request(
               ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/clients/auth.py", line 48, in wrapper
    async for key, info, context in vault.extended(APIContext, 'contexts'):
  File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/structs/credentials.py", line 158, in extended
    async for key, item in self._items():
  File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/structs/credentials.py", line 195, in _items
    yielded_key, yielded_item = self.select()
                                ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/structs/credentials.py", line 214, in select
    raise LoginError("Ran out of valid credentials. Consider installing "
kopf._cogs.structs.credentials.LoginError: Ran out of valid credentials. Consider installing an API client library or adding a login handler. See more: https://kopf.readthedocs.io/en/stable/authentication/
[2025-01-29 13:06:34,045] kopf._core.reactor.r [ERROR   ] Resource observer has failed: Ran out of valid credentials. Consider installing an API client library or adding a login handler. See more: https://kopf.readthedocs.io/en/stable/authentication/
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/aiokits/aiotasks.py", line 96, in guard
    await coro
  File "/usr/local/lib/python3.12/site-packages/kopf/_core/reactor/observation.py", line 127, in resource_observer
    await queueing.watcher(
  File "/usr/local/lib/python3.12/site-packages/kopf/_core/reactor/queueing.py", line 175, in watcher
    async for raw_event in stream:
  File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/clients/watching.py", line 86, in infinite_watch
    async for raw_event in stream:
  File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/clients/watching.py", line 201, in continuous_watch
    async for raw_input in stream:
  File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/clients/watching.py", line 266, in watch_objs
    async for raw_input in api.stream(
  File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/clients/api.py", line 201, in stream
    response = await request(
               ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/clients/auth.py", line 48, in wrapper
    async for key, info, context in vault.extended(APIContext, 'contexts'):
  File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/structs/credentials.py", line 158, in extended
    async for key, item in self._items():
  File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/structs/credentials.py", line 195, in _items
    yielded_key, yielded_item = self.select()
                                ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/structs/credentials.py", line 214, in select                                                                                                                                                        raise LoginError("Ran out of valid credentials. Consider installing "                                                                                                                                                                                  kopf._cogs.structs.credentials.LoginError: Ran out of valid credentials. Consider installing an API client library or adding a login handler. See more: https://kopf.readthedocs.io/en/stable/authentication/                                              [2025-01-29 13:06:34,046] kopf._core.reactor.o [ERROR   ] Watcher for <class>@<namespace> has failed: Ran out of valid credentials. Consider installing an API client library or adding a login handler. See more: https://kopf.readthedocs.io/en/stable/authentication/
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/aiokits/aiotasks.py", line 96, in guard
    await coro
  File "/usr/local/lib/python3.12/site-packages/kopf/_core/reactor/queueing.py", line 175, in watcher
    async for raw_event in stream:
  File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/clients/watching.py", line 86, in infinite_watch
    async for raw_event in stream:
  File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/clients/watching.py", line 201, in continuous_watch
    async for raw_input in stream:
  File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/clients/watching.py", line 266, in watch_objs
    async for raw_input in api.stream(
  File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/clients/api.py", line 201, in stream
    response = await request(
               ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/clients/auth.py", line 48, in wrapper
    async for key, info, context in vault.extended(APIContext, 'contexts'):
  File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/structs/credentials.py", line 158, in extended
    async for key, item in self._items():
  File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/structs/credentials.py", line 195, in _items
    yielded_key, yielded_item = self.select()
                                ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/structs/credentials.py", line 214, in select
    raise LoginError("Ran out of valid credentials. Consider installing "
kopf._cogs.structs.credentials.LoginError: Ran out of valid credentials. Consider installing an API client library or adding a login handler. See more: https://kopf.readthedocs.io/en/stable/authentication/
Traceback (most recent call last):
  File "/usr/local/bin/kopf", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.12/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/kopf/cli.py", line 60, in wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/click/decorators.py", line 92, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)                                                                                                                                                                                                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/kopf/cli.py", line 109, in run
    return running.run(
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/kopf/_core/reactor/running.py", line 81, in run
    asyncio.run(coro)
  File "/usr/local/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)                                                                                                                                                                                                                                           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 686, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/kopf/_core/reactor/running.py", line 138, in operator
    await run_tasks(operator_tasks, ignored=existing_tasks)
  File "/usr/local/lib/python3.12/site-packages/kopf/_core/reactor/running.py", line 419, in run_tasks
    await aiotasks.reraise(root_done | root_cancelled | hung_done | hung_cancelled)
  File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/aiokits/aiotasks.py", line 226, in reraise
    task.result()  # can raise the regular (non-cancellation) exceptions.
    ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/aiokits/aiotasks.py", line 96, in guard
    await coro
  File "/usr/local/lib/python3.12/site-packages/kopf/_core/reactor/observation.py", line 77, in namespace_observer
    await queueing.watcher(
  File "/usr/local/lib/python3.12/site-packages/kopf/_core/reactor/queueing.py", line 175, in watcher
    async for raw_event in stream:
  File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/clients/watching.py", line 86, in infinite_watch
    async for raw_event in stream:
  File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/clients/watching.py", line 201, in continuous_watch
    async for raw_input in stream:
  File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/clients/watching.py", line 266, in watch_objs
    async for raw_input in api.stream(
  File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/clients/api.py", line 201, in stream
    response = await request(
               ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/clients/auth.py", line 48, in wrapper
    async for key, info, context in vault.extended(APIContext, 'contexts'):
  File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/structs/credentials.py", line 158, in extended
    async for key, item in self._items():
  File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/structs/credentials.py", line 195, in _items
    yielded_key, yielded_item = self.select()
                                ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/structs/credentials.py", line 214, in select
    raise LoginError("Ran out of valid credentials. Consider installing "
kopf._cogs.structs.credentials.LoginError: Ran out of valid credentials. Consider installing an API client library or adding a login handler. See more: https://kopf.readthedocs.io/en/stable/authentication/

Additional information

No response