RFC: backporting support for new Python versions to "old" pyo3 branches?

[decathorpe]

Context:

In Fedora Linux, we are getting more and more Python projects that are built on top of Rust + PyO3. While this is great to see, it is also starting to cause us a fair amount of pain (and major headaches for the Python team) every time the system-wide Python interpreter is updated (CPython).

Projects that only rely on the stable "abi3" are easier to handle - at least they don't (usually) break due to CPython changes. But many Python projects that we package and support are still (or rather, again) stuck on old versions of pyo3:

• v0.19: cryptography, libcst, pendulum, watchfiles
• v0.20: fapolicy-analyzer, bcrypt, cramjam, nh3, rpds-py, setuptools-rust, tiktoken
• v0.21: matrix-synapse, pydantic-core

At least some of these are projects that we need to support (especially the widely-used "cryptography" package), but that's increasingly hard to do if they're always stuck on old versions of pyo3, which might or might not be broken (or even supported) by new Python versions.

However, bumping the pyo3 dependency to add support for new Python versions is basically never trivial, because PyO3 releases that add support for new Python versions also include breaking API changes on the PyO3 side. So I started looking into backporting basic support for Python 3.13 for pyo3 v0.21, v0.20, and v0.19, and got far enough that the tests all pass - looks like there were not that many breaking changes in CPython 3.13.

---

This brings me to my question:

Would it be worthwhile to do this "semi-officially", and add basic support for new Python versions in older release branches of pyo3 (if possible), to publish new versions on those branches for projects that can't move to the latest breaking release yet, but which need support for new Python versions?

That "semi-official" support could even stay hidden behind the UNSAFE_PYO3_SKIP_VERSION_CHECK environment variable - but this would still provide an easier "escape hatch" than having to port to new PyO3 versions immediately.

I can open PRs with my backport-python3.13-support patches against older release branches - I already did the work for the v0.21, v0.20, and v0.19 branches ... or at least enough to get all the tests to pass on all architectures that are supported by Fedora Linux (this includes little-endian x86_64, i686, aarch64, powerpc64le, and big-endian s390x), and on all currently supported versions of "system" Python - 3.13b2 (Rawhide), 3.12.3 (Fedora 40 and 39), and 3.9 (RHEL 9).

[davidhewitt]

Sorry to have missed this discussion until just now.

I am empathetic to the problem of outdated PyO3 versions impeding support for latest CPython releases. It would be good to come up with a policy of how to handle these in a way which doesn't block the ecosystem but doesn't also build an increasing set of PyO3 versions which need to be supported. How far back would you be asking for backports to be delivered?

On a particular note for current versions I'm not hugely keen on encouraging projects using PyO3 0.20 or older to feel like they can be "stable" because they have a known interaction with gevent which is likely enough to cause crashes that I think it's better for everyone if those projects put effort into updating.

---

I think the most practical option, at least to the first order, is to try to build offending projects with the PYO3_USE_ABI3_FORWARD_COMPATIBILITY environment variable, which we added in PyO3 0.20.3. This'll force use of abi3 on Python 3.13 (for 0.20 and 0.21), which should hopefully be suitable for most projects even if they didn't opt for abi3 as standard.

The remaining projects which are on 0.19 should really be updating. I think of that list I believe cryptography and watchfiles should be updated by now at least.

I keep feeling like there should be enough community interest that we should be able to muster resources to distribute the problem amongst multiple interested volunteers, but I haven't found a good way to collect / manage a list of packages which need PyO3 updates and would accept contributions.

[decathorpe]

Thank you for the detailed response!

Sorry to have missed this discussion until just now.

No problem.

I am empathetic to the problem of outdated PyO3 versions impeding support for latest CPython releases. It would be good to come up with a policy of how to handle these in a way which doesn't block the ecosystem but doesn't also build an increasing set of PyO3 versions which need to be supported. How far back would you be asking for backports to be delivered?

I don't have a good answer yet. Maybe a policy like "the current and previous release branch of pyo3" (unless there are $REASONS, see below) would work?

On a particular note for current versions I'm not hugely keen on encouraging projects using PyO3 0.20 or older to feel like they can be "stable" because they have a known interaction with gevent which is likely enough to cause crashes that I think it's better for everyone if those projects put effort into updating.

Sure. If there are known problems that can only be fixed by porting something to the next major version for $REASONS, then I agree that backporting broken "support" to old pyo3 versions is not a good idea.

I think the most practical option, at least to the first order, is to try to build offending projects with the PYO3_USE_ABI3_FORWARD_COMPATIBILITY environment variable, which we added in PyO3 0.20.3. This'll force use of abi3 on Python 3.13 (for 0.20 and 0.21), which should hopefully be suitable for most projects even if they didn't opt for abi3 as standard.

In Fedora we already patch pyo3-ffi to do something like UNSAFE_PYO3_SKIP_VERSION_CHECK unconditionally. Otherwise landing beta pre-releases of CPython would break everything. This worked fine for most things, except for one (I think pydantic-core), which uses APIs that are not covered by abi3 compatibility.

The remaining projects which are on 0.19 should really be updating. I think of that list I believe cryptography and watchfiles should be updated by now at least.

Yeah, looks like there's at least some movement happening here. I can try to push things further for the Python modules we have in Fedora. For the remaining ones that depend on pyo3 v0.19 and v0.20, I can take a look if moving them to at least v0.20 or v0.21. As far as I can tell, API changes between v0.19 and v0.20 / v0.21 were not as big as those between v0.21 and v0.22 (especially the "bound" stuff).

I keep feeling like there should be enough community interest that we should be able to muster resources to distribute the problem amongst multiple interested volunteers, but I haven't found a good way to collect / manage a list of packages which need PyO3 updates and would accept contributions.

I will probably need to keep doing at least some amount of backporting to older PyO3 branches for the foreseeable future. So you could count me in :)