Upgrade Python version used in WMCore

[amaltaro]

Impact of the new feature
Everything in WMCore

Is your feature request related to a problem? Please describe.
Our central services are based on Python 3.8, which was EOL in Oct/2024.
WMAgents are using 3.9, and so is our docker Jenkins setup. Python 3.9 is also scheduled to EOL in 2025, see:
https://devguide.python.org/versions/

This python upgrade need to be executed in a conservative manner, because we have the WMRuntime package executed in the grid (including other WMCore python packages), which could be using a variety of python versions, AFAICT 3.6, 3.8 and 3.9.

Describe the solution you'd like
The idea is to upgrade the WMCore python version meeting all the following criteria:

• upgrade to the latest stable version
• adopt a safe version for grid runtime (so something that has low probability of breaking between different python3.x versions)
• least effort upgrade (i.e., a good cost-benefit upgrade).

Based on that, my opinion is that python 3.9 is the safest upgrade. On the other hand, 3.10 potentially give us less maintenance for the coming couple of years. However, are there major/breaking differences (that might affect us) between py3.10 vs py3.11? How about py3.11 vs py3.12?

Note that updates need to happen at least:

1. WMCore codebase - adapting any code that needs changes between these different versions (should we consider compatibility among different py3 versions?)
2. WMCore requirements.txt file
3. WM central services dockerfiles
4. WMAgent dockerfiles
5. WMAgent virtual environment setup
6. WMCore-dev dockerfiles
7. Anything else in Jenkins?

Describe alternatives you've considered
None

Additional context
Some issues that can be directly related to this:

• Upgrade pycurl and pyOpenSSL in our python3 cmsdist stack Upgrade pycurl and pyOpenSSL in our python3 cmsdist stack #11207
• In python 3.9 the sys.setcheckinterval is removed In python 3.9 the sys.setcheckinterval is removed #10925
• Update HTCondor python bindings to release cycle 23.x Update HTCondor python bindings to release cycle 23.x #12031

[mapellidario]

can we decouple the python version that we use in our docker containers from the one that we use in wmruntime on grid nodes?

[amaltaro]

can we decouple the python version that we use in our docker containers from the one that we use in wmruntime on grid nodes?

We rely on CVMFS to load the python environment. Given that we have only one ScramArch for the COMP package deployed in CVMFS - long ago - I think all the other ScramArchs are actually symlinks created to specific CMSSW versions, such that we can load the closest python version (provided by Shahzad).

So I am afraid the answer is no. Unless we put a major effort on this, which I believe not to be the right time now.

[mapellidario]

this is exactly my point. since changing the python version in our docker images is "easy" and chaning the python version distributed via the comp environment on cvmfs is "harder", can we have those two versions diverge? is it there any reason those need to be the same?

or better, why dont we start by upgrading python in our containers, then we focus on wmruntime and how to make it simple enough that it can happily run on whatever python runtime we will ever ship via cvmfs?

[amaltaro]

The problem lies here (from the initial description):

1. WMCore codebase - adapting any code that needs changes between these different versions (should we consider compatibility among different py3 versions?)

As we are probably aware of, python3.x has some "interesting" breaking changes between minor releases (3.6, 3.9, 3.x). That means, if we want to upgrade our docker images to a newer python, the point 1 above is one of the first things that we need to consider.

Based on that, unless we can guarantee that our codebase is compatible with all these python3 versions, there is always a risk that we are using something new/deprecated in our codebase that will fail in a given CVMFS/CMSSW environment.

In other words, we are not planning to install anything in CVMFS. We would keep the same runtime setup as we have right now. Could we deploy the latest WMAgent (or required WMRuntime + other packages) in CVMFS with the latest Python that we decide to adopt? I think so. Would it resolve all the potential runtime issues? Probably not, as we would have to support different architectures as well, and I do not know how we could accomplish that without Shahzad's deep knowledge.