0.8.0 version regression: "aiohttp-client-cache" prevents Pytest session from finishing (tests pass, but session never exits)

[simon-liebehenschel]

0.7.3 version works fine, but when I install 0.8.0, my Pytest tests results in:

======================================================== test session starts =========================================================
platform linux -- Python 3.10.9, pytest-7.2.0, pluggy-1.0.0

[A LIST OF MY TESTS HERE, NOT IMPORTANT]

================================================= 1 passed, 208 deselected in 4.16s ==================================================

In other words, after all tests the session hangs forever.

Were there any forever running tasks introduces in the 0.8.0 version? I even do not understand how to debug this.

[simon-liebehenschel]

I believe a deadlock may be introduced somewhere in the last commits, e.g. 931f521

I will try to make a reproducible small code sample...

[simon-liebehenschel]

Reproducible code sample:

from aiohttp_client_cache import CachedSession, SQLiteBackend
import pytest

@pytest.mark.asyncio()
async def test_client_cache():
    async with CachedSession(cache=SQLiteBackend('demo_cache')) as session:
        await session.get('http://httpbin.org/delay/1')

Dependencies (pip freeze output):

aiohttp==3.8.3
aiohttp-client-cache==0.8.0
aiosignal==1.3.1
aiosqlite==0.18.0
async-timeout==4.0.2
attrs==22.2.0
charset-normalizer==2.1.1
exceptiongroup==1.1.0
frozenlist==1.3.3
idna==3.4
iniconfig==1.1.1
itsdangerous==2.1.2
multidict==6.0.4
packaging==22.0
pluggy==1.0.0
pytest==7.2.0
pytest-asyncio==0.20.3
python-forge==18.6.0
six==1.16.0
tomli==2.0.1
url-normalize==1.4.3
yarl==1.8.

pytest tests/ result:

A test pass, but session hangs forever.

[JWCook]

I'm surprised the CI tests didn't catch that, but you're right, it must be a deadlock. My best guess is that it's stuck on closing the database connection when the CachedSession contextmanager exits.

I am a little busy this week, but I will try to get this fixed soon.

[JWCook]

I think it was actually the opposite problem: the database connection wasn't getting closed, and leaving that connection open seems to cause problems when the test session ends, presumably when it's deleting resources from memory. I made some changes in #148 that I believe will fix this. Changes are in main. Give that a try and let me know if that works for you.

I still don't 100% understand the problem, though, because it doesn't seem to occur within the destructor of any of the classes within aiohttp-client-cache. Also, it appears to be a threading lock that's stuck, not an async lock. It's in threading._shutdown(), but I couldn't get any error context beyond that. In aiohttp-client-cache 0.8, thread locks are were no longer needed and were swapped out for async locks.

That leaves aiosqlite, which manages its own thread that it uses to run blocking SQLite operations. There's no traceback coming from aiosqlite, though, and I can't consistently reproduce the problem. In GitHub Actions, for example, the tests work fine.

Very weird! I am probably missing something about how aiosqlite.Connection works, but it's not obvious to me right now.

[JWCook]

Fixed in 0.8.1.

[simon-liebehenschel]

@JWCook The 0.8.1 version used to work great. Today I updated to 0.8.2 and now I have this problem again. Could you please take a look?

[JWCook]

Yeah, looks like a regression bug, sorry about that. I'll get that fixed.

[JWCook]

This is fixed with #162, and I'll include that change in a patch release within a week or so.

Since automatically closing connections might be desirable in some cases and not others, I added an autoclose parameter to turn this behavior on or off.