Random PostgreSQL connection errors

[amureki]

Python Version

3.12

Django Version

4.2.9

Package Version

6.6.0

Description

Greetings dear fellows!

We are experiencing a lot of “InterfaceError: connection already closed” and “TransactionManagementError”. They are not necessarily accompany each other, but both are consistently happening.
PG logs are not showing anything unexpected or suspicious, but Django closing the connections.

Affected endpoints are different, and I see no common behaviour in them. For example, it sometimes fails even in the basic non-customized admin interface during GET requests.
It does not happen locally.

We did post more details in the following thread:
https://forum.djangoproject.com/t/interfaceerror-and-transactionmanagementerror-postgresql/16924

Later I noticed other people were sharing the similar issues:

• https://forum.djangoproject.com/t/seemingly-random-postgresql-connection-drops-while-handling-django-requests/19026
• https://forum.djangoproject.com/t/django-db-issue-interface-error-connection-closed/21535

And one recent comment was about suspecting a WhiteNoise upgrade.

We did downgrade WhiteNoise version from 6.6.0 to 5.3.0 (one major version) and it indeed resolved all mentioned issues on our instances.

I am struggling to pin the issue down to the change, but happy to assist and provide any other details.

Best,
Rust

[adamchainz]

I don’t think I can really do anything without more details or a reproducing sample project. I have used the latest whitenoise versions with PostgreSQL just fine.

If you could try git bisect, or at least manual bisecting beteeen the two versions you found, that would help narrow down the probable cause. But again without a reproduction it’s hard to write a fix.

[adamchainz]

If you want to hire me as a consultant I could work directly within your project to tackle the issue. I’m not entirely conviver it’s due to whitenoise, as this package does not touch the database at all.

[amureki]

I don’t think I can really do anything without more details or a reproducing sample project. I have used the latest whitenoise versions with PostgreSQL just fine.

I agree, it is quite a vague issue that I submitted, but I am struggling to reproduce it to be able to pin it down to certain functionality or commit and share an example project.
As soon as I'll have something, I will surely post it.

Thank you for the response and help proposal! I'll check with my company if we can organize something. ✨

Cheers,
Rust

[lapinvert]

I had same issue for months, it was driving me crazy. Luckily I found the djangoproject thread where someone pointed out whitenoise.

Downgrading to 5.3.0 solved the issue for me as well. Thank you @amureki .

It is concerning tho because many ppl may be actually impacted. The only reason I was able to notice and I guess other ppl in the threads as well, is because I use sentry. If you don't and I suspect most people don't, you just end up with weird things happening in your database, states that were not supposed to happen, without knowing what's going on.

If most users of 6.6.0 have no issues, it maybe a combination of things that cause the issue, maybe it would be nice @amureki if we got to chat a bit and see what our apps had in common besides whitenoise.

[kgrinberg]

I don't want to overpromise, but I do expect that in the coming weeks we'll be able to carve out some time here to bisect and hopefully diagnose the problem (which we did find was solved in the immediate case by downgrading from 6.6.0 to 5.3.0). Gun to my head I'd say it's something in the middleware (possibly in combination with others etc) but I'll report what we find.

[dustinblanchard]

I'm having the same problem. I'm on Python 3.12 and Django 4.2.8 and soon to upgrade to 5, @amureki @kgrinberg @lapinvert did y'all have any compatibility issues with 5.3.0 and current python/django versions?

[kgrinberg]

I've only tested with LTS Django (so 3.2.x and 4.2.x) so can't comment on Django 5 and this issue. To reiterate, I do suspect it's something more subtle/complicated than "these things just don't work together", but hope to have something more concrete to share when we've had some time here to delve in, which likely won't be for a couple weeks.

[breadmaking]

Also had the same issue recently with a cookiecutter project - there was no pattern to where it would occur. All across my app in production. Identical symptoms to @amureki

Running on:
django ==4.2.9
whitenoise==6.6.0
Downgrading to 5.3.0 solved the issue for me as well.

That said I have another cookiecutter project running in production with no issues at all:
django==4.0.10
whitenoise==6.3.0

[brianjp93]

Downgrading to 5.3.0 seems to have fixed the errors for me as well. Was on django==4.2.9 and whitenoise==6.6.0 like the others in this thread. Python 3.11.7

[Block99block99]

Same here with
django==4.2.11
whitenoise==6.6.0

[ahivert]

Same here with:

Django==5.0.6
whitenoise==6.6.0

Fixed with downgrade of whitenoise

[fbarl]

Same here, after a few weeks of trying everything (including testing on different versions of Python / Django), downgrading whitenoise to v5.3 finally fixed it, no idea why 🙄

[longhotsummer]

We have seen the same problem, downgrading to whitenoise 5.3.0 fixes it. I tested whitenoise 6.0.0 and it still causes the issue, so my guess is it's some change between 5.3.0 and 6.0.0

• Python 3.8 and 3.10
• Django 4.2.14

[pernofence]

We are having the same problem. It seems to be related to http range requests.

Our particular issue happens with some mp3 files, but using a range request on any file seems to trigger the issue.

I can reliably reproduce the issue by repeatedly running a request like this while loading some other pages in the django admin.

curl -v -H "Range: bytes=0-" http://localhost:8000/static/admin/css/base.css -o /dev/null

When the range request for the static file hits at the correct time, completely unrelated requests will fail with django.db.utils.InterfaceError: cursor already closed

I can reproduce this with both gunicorn and django's runserver.

I suspect it has something to do with this not being closed properly:

whitenoise/src/whitenoise/responders.py

Line 43 in 946a95e

class SlicedFile(BufferedIOBase):