Python for Pants upgrade and syntax plan

[cburroughs]

Thanks to all of the scie-pants and release machinery work (huzzah!) we should at least be clear to upgrade the version of Python used by Pants itself. That's currently 3.9, how do we get to 3.12?

Slow and Steady

Upgrade release by release, leaving lots of time for things for things to shake out. Easy-ish to revert any single step. Spreads benefits and pain out over time. Smaller(?) steps for plugin authors to jump across.

• 2.20: enable pyupgrade backend for pants itself and configure for 3.9 (does not change code); upgrade pants to use python 3.10
• 2.21: configure pyupgrade for 3.10
• 2.23: upgrade pants to use Python 3.11
• 2.24: configure pyupgrade for 3.11
• 2.25: upgrade pants to use Python 3.12
• 2.26: configure pyupgade or 3.12

Just Do It

• 2.20: upgrade pants to use Python 3.12; set pyupgrade to 3.12; Done.

Gets the project there sooner but if issues are not sussed out right away the revert might be a mess? Possibly too much drift for cherry-picking across branches. More risk but done sooner.

My personal inclination leans towards "Slow and Steady", eventually upgrading Python should be pretty boring and we can make it boring by practicing a bunch of boring steps.

[thejcannon]

So FWIW each step in the chain would involve a change to scie-pants to map Pants version -> Python version.

So my preference would be to jump forward to the latest version our tools (namely PyO3) supports.

[thejcannon]

I forgot I did that. Man that's strike two for me on "Josh forgets he did something smart"

[cburroughs]

The work here is (among other things) future proofing this line?

python = "cpython38" if version < Version("2.5") else "cpython39"
https://github.com/pantsbuild/scie-pants/blob/76911604586dab64cc226c5891504be607b506da/tools/src/scie_pants/configure_pants.py#L120

[cburroughs]

One other uncertainty here I think is how to encode the version of Python used when running pants from source as currently I think it's covered by the same if statement above.

• Continue to encode it with if statements in scie-
• Introduce something in pants.toml
• (hand-waving) Something like PANTS_PYTHON_VERSION_OVERRIDE?

Leaning towards the latest one, but contingent on it actually being feasible.

[cburroughs]

Doh. The python version is in the filename!

So... the problem here is that unless I'm totally crazy, GitHub doesn't provide an easy "curl-able" way to find out the assets in a release. You can use the api which would be a rate limit non-starter, or heuristically parse the html which would be obviously brittle.

I'm leaning at this point back towards a oneshot flag day to 3.11.

[kaos]

I'll look at adding support for this in scie-pants.

pantsbuild/scie-pants#351

You can use the api which would be a rate limit non-starter.

Running in CI, you should be able to provide some GH API token. Locally the result is cached, so shouldn't be that bad I think.

[thejcannon]

Also, here was a branch with a working Pants using 3.11: https://github.com/pantsbuild/pants/tree/jcannon/py311

[benjyw]

I think we can jump straight to 3.11, which has been out for a while, and then wait one more release for 3.12?

[cburroughs]

Some earlier related discussion at pantsbuild/scie-pants#154, pantsbuild/scie-pants#94, and #18145.

[cburroughs]

After some noodling, a revised minimal proposal:

• Update scie-pants to say: "Pants >= 2.21 uses CPython 3.11"
• Wait till 2.20 branch is cut
• Switch Pants main to use 3.11 build 3.11 pexes, etc

Longer term I think something along the lines of #18145 would allow this to be less of a production. It would be nice if Pants could provide Python point releases and PBS upgrades, which happen more frequently than CPython major releases.

[sureshjoshi]

I think one missing item of discussion - what is the impact on users with custom plugins?

I'm so hazy on these conversations from like... a year ago, but I thought that custom plugins need to run on the same Python interpreter that Pants is being run by.

Segmenting out (let's say 2.21 is the cutover) - does this sound right?

• For users without custom plugins, they shouldn't notice a difference or even that anything happened after each of their team members upgrade to the "latest" scie-pants (using Pants pre or post 2.21)
• For users with custom plugins .... ? They have to ensure the plugins all work on the same interpreter version (3.11 or 3.12 or whatever) before they update to the latest pants 2.21
• For Pants developers ... After updating sci-pants, there is no noticeable difference?

cc @benjyw @kaos

[kaos]

Sounds about right to me.