Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Python 3.12 [ci skip] #46205

Closed
wants to merge 1 commit into from
Closed

Conversation

ahesford
Copy link
Member

@ahesford ahesford commented Sep 23, 2023

As summer (in the northern hemisphere) turns to autumn, and the leaves begin to turn, so too does the Python version begin to turn... this time to 3.12.

[ci skip]

@ahesford ahesford marked this pull request as draft September 23, 2023 19:01
@ahesford ahesford force-pushed the py312 branch 2 times, most recently from 6300546 to 7320569 Compare September 24, 2023 00:57
@ahesford
Copy link
Member Author

ahesford commented Sep 24, 2023

The list of x86_64, x86_64-musl, aarch64 and i686 failures is now 2:

  • python3-cypari2
  • python3-fpylll

@ahesford ahesford changed the title [WIP] Python 3.12 [WIP] Python 3.12 [ci skip] Sep 24, 2023
@ahesford ahesford force-pushed the py312 branch 13 times, most recently from 36bfe48 to 81f3e70 Compare September 27, 2023 16:16
@ahesford
Copy link
Member Author

ahesford commented Sep 27, 2023

I believe it's time to start retiring python3-pyside2 and shiboken2. They were never updated to suport Python 3.11 and now do not function with 3.12; shiboken2 segfaults on many of its tests and also when used during the build of python3-pyside2. Dependants of these packages are:

  • freecad
  • pyside2-tools (required by freecad only)
  • syncplay
  • cutter (latest version seems to support Qt 6)
  • hydrus (I have a suspicion the dependencies are done wrong; seems to support PyQt{5,6} and PySide{5,6})
  • python3-QtPy (checkdepends only; should be able to use PyQt6 and PyQt5 exclusively)

Dependants of python3-QtPy (these should be checked with PyQt6 or PyQt5):

  • git-cola
  • python3-jupyter_qtconsole

Related: Upstream explicitly disclaims support for Python 3.12 in PySide 2

EDIT: With an update of shiboken2 and python3-pyside2 to 5.15.10, backporting of some pending Python 3.12 fixes from PySide6, and an ugly CMake hack to allow binding generation for QtQuick widgets, python3-pyside2 builds. This may allow us to continue with freecad and syncplay as is for a bit longer.

@ahesford
Copy link
Member Author

ahesford commented Sep 27, 2023

I think there is a PyQt5 segfault problem. See, e.g., xbps-src check python3-QtPy.

EDIT: resolved by pulling python3-sip-PyQt5 patches from Fedora.

@ahesford ahesford force-pushed the py312 branch 5 times, most recently from 6e6dd4b to 36da526 Compare September 28, 2023 18:31
@ahesford
Copy link
Member Author

@leahneukirchen To get Vim to compile, I added --with-python3-stable-abi=3.12. It seems the Python.h header introduces some inline asserts that require linking with the Python library differently than how Vim handles it unless the "limited API" is used. Related issues:

  1. if_python3.c: Fix building dynamic Python3 interpreter with Python 3.12 vim/vim#12660
  2. Cannot include Python.h in 3.12 without linking against the CPython library python/cpython#108502

I don't know if there are any adverse effects here, or whether an earlier version should be specified via the argument.

@ahesford ahesford force-pushed the py312 branch 2 times, most recently from 29cc8d9 to 85a0e74 Compare September 28, 2023 19:01
@ahesford
Copy link
Member Author

ahesford commented Oct 1, 2023

Most packages aren't run through checks, although I do check some. We don't have the resources to verify that all Python packages pass checks when doing these massive updates.

The new gmpy2 and mpmath pass checks, but I didn't go further than that.

@tornaria
Copy link
Contributor

tornaria commented Oct 2, 2023

Most packages aren't run through checks, although I do check some. We don't have the resources to verify that all Python packages pass checks when doing these massive updates.

The new gmpy2 and mpmath pass checks, but I didn't go further than that.

Ok, I'll investigate, but in principle I would assume sympy is broken with 3.12. I'd rather void stays in the non-bleeding edge...

I'm working on cypari2 now. Note that fpylll and sagemath should suffer from the same issue: cpython doesn't want us to look inside longint, but they don't want to offer a C alternative (they want us to use PyObject_CallMethod to call some python functions: https://github.com/python/cpython/pull/101270/files).

There seems to be a "cheating" way which is to peek inside the structs, i.e. change x->ob_digit for x->long_value.ob_digit (really, why are they messing up with us?). But they changed this in 3.11 and they changed it again in 3.12 and since they are having fun I'd expect they will change it again in 3.13 just because they want to keep us busy. It seems a better (long term) way would be to use _PyLong_FromByteArray and _PyLong_AsByteArray...

@ahesford
Copy link
Member Author

ahesford commented Oct 2, 2023

I pushed a couple of upstream changes to support Python 3.12 in sympy, and the only test that still reports a failure is a deprecation warning. (I believe many of the failing tests fixed by one of the patches do nothing more substantial than silence unexpected deprecation warnings, which are meaningless failures anyway.)

I don't have a strong opinion about how to resolve the long representation problem in fpylll, cypari2 or sagemath. Ideally upstream decides on a path forward and we backport it. Going through the Python interpreter seems like a suboptimal approach, but is consistent with CPython's intention to conceal implementation details and allow future optimizations. If our package upstreams don't have an accepted patch, the "cheat" (going through _PyLongValue) seems the least obtrusive, because it most closely preserves the spirit of the original implementation.

@tornaria
Copy link
Contributor

tornaria commented Oct 2, 2023

I pushed a couple of upstream changes to support Python 3.12 in sympy, and the only test that still reports a failure is a deprecation warning. (I believe many of the failing tests fixed by one of the patches do nothing more substantial than silence unexpected deprecation warnings, which are meaningless failures anyway.)

I don't have a strong opinion about how to resolve the long representation problem in fpylll, cypari2 or sagemath. Ideally upstream decides on a path forward and we backport it. Going through the Python interpreter seems like a suboptimal approach, but is consistent with CPython's intention to conceal implementation details and allow future optimizations. If our package upstreams don't have an accepted patch, the "cheat" (going through _PyLongValue) seems the least obtrusive, because it most closely preserves the spirit of the original implementation.

See: https://github.com/tornaria/void-packages/tree/py312

I'll do fpylll and sagemath tomorrow, and do PRs to upstream that fix. Hopefully I can use the same .h file for all projects, and the api and the 3.12 implementation is taken verbatim from python so it should be easier to move forward in case python 3.13 changes again.

@tornaria
Copy link
Contributor

tornaria commented Oct 2, 2023

Better link: ahesford/void-packages@py312...tornaria:void-packages:py312

So far cypari and fpylll are working (tested x86_64 and i686). Patches taken from submitted upstream PRs (sagemath/cypari2#138 and fplll/fpylll#254).

@ahesford: if you squash, you may want to rename the commit to "patch and rebuild".

I'm now building sagemath deps. I expect a similar fix will suffice.

@ahesford ahesford force-pushed the py312 branch 10 times, most recently from df0ff03 to 73fd7c1 Compare October 4, 2023 01:39
@tornaria
Copy link
Contributor

tornaria commented Oct 4, 2023

f2py in numpy is broken. Steps to reproduce:

With python 3.11, it works:

$ cat test.f
      subroutine foo
          end
$ python3 -m numpy.f2py -c test.f 
/usr/lib/python3.11/site-packages/numpy/f2py/f2py2e.py:686: VisibleDeprecationWarning: distutils has been deprecated since NumPy 1.26.Use the Meson backend instead, or generate wrapperswithout -c and use a custom build script
  builder = build_backend(
running build
running config_cc
INFO: unifing config_cc, config, build_clib, build_ext, build commands --compiler options
running config_fc
INFO: unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
running build_src
INFO: build_src
INFO: building extension "untitled" sources
INFO: f2py options: []
INFO: f2py:> /tmp/tmpw74b41uy/src.linux-x86_64-3.11/untitledmodule.c
creating /tmp/tmpw74b41uy/src.linux-x86_64-3.11
Reading fortran codes...
        Reading file 'test.f' (format:fix,strict)
Post-processing...
        Block: untitled
                        Block: foo
Applying post-processing hooks...
  character_backward_compatibility_hook
Post-processing (stage 2)...
Building modules...
    Building module "untitled"...
    Generating possibly empty wrappers"
    Maybe empty "untitled-f2pywrappers.f"
        Constructing wrapper function "foo"...
          foo()
    Wrote C/API module "untitled" to file "/tmp/tmpw74b41uy/src.linux-x86_64-3.11/untitledmodule.c"
INFO:   adding '/tmp/tmpw74b41uy/src.linux-x86_64-3.11/fortranobject.c' to sources.
INFO:   adding '/tmp/tmpw74b41uy/src.linux-x86_64-3.11' to include_dirs.
copying /usr/lib/python3.11/site-packages/numpy/f2py/src/fortranobject.c -> /tmp/tmpw74b41uy/src.linux-x86_64-3.11
copying /usr/lib/python3.11/site-packages/numpy/f2py/src/fortranobject.h -> /tmp/tmpw74b41uy/src.linux-x86_64-3.11
INFO:   adding '/tmp/tmpw74b41uy/src.linux-x86_64-3.11/untitled-f2pywrappers.f' to sources.
INFO: build_src: building npy-pkg config files
/usr/lib/python3.11/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
!!

        ********************************************************************************
        Please avoid running ``setup.py`` directly.
        Instead, use pypa/build, pypa/installer or other
        standards-based tools.

        See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
        ********************************************************************************

!!
  self.initialize_options()
running build_ext
INFO: customize UnixCCompiler
INFO: customize UnixCCompiler using build_ext
INFO: get_default_fcompiler: matching types: '['arm', 'gnu95', 'intel', 'lahey', 'pg', 'nv', 'absoft', 'nag', 'vast', 'compaq', 'intele', 'intelem', 'gnu', 'g95', 'pathf95', 'nagfor', 'fujitsu']'
INFO: customize ArmFlangCompiler
WARN: Could not locate executable armflang
INFO: customize Gnu95FCompiler
INFO: Found executable /usr/bin/gfortran
INFO: customize Gnu95FCompiler
INFO: customize Gnu95FCompiler using build_ext
INFO: building 'untitled' extension
INFO: compiling C sources
INFO: C compiler: x86_64-unknown-linux-gnu-gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fstack-clash-protection -D_FORTIFY_SOURCE=2 -mtune=generic -O2 -pipe -g -fstack-clash-protection -D_FORTIFY_SOURCE=2 -mtune=generic -O2 -pipe -g -fPIC

creating /tmp/tmpw74b41uy/tmp
creating /tmp/tmpw74b41uy/tmp/tmpw74b41uy
creating /tmp/tmpw74b41uy/tmp/tmpw74b41uy/src.linux-x86_64-3.11
INFO: compile options: '-DNPY_DISABLE_OPTIMIZATION=1 -I/tmp/tmpw74b41uy/src.linux-x86_64-3.11 -I/usr/lib/python3.11/site-packages/numpy/core/include -I/usr/include/python3.11 -c'
INFO: x86_64-unknown-linux-gnu-gcc: /tmp/tmpw74b41uy/src.linux-x86_64-3.11/untitledmodule.c
INFO: x86_64-unknown-linux-gnu-gcc: /tmp/tmpw74b41uy/src.linux-x86_64-3.11/fortranobject.c
INFO: compiling Fortran sources
INFO: Fortran f77 compiler: /usr/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -fPIC -O3 -funroll-loops
Fortran f90 compiler: /usr/bin/gfortran -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loops
Fortran fix compiler: /usr/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loops
INFO: compile options: '-I/tmp/tmpw74b41uy/src.linux-x86_64-3.11 -I/usr/lib/python3.11/site-packages/numpy/core/include -I/usr/include/python3.11 -c'
INFO: gfortran:f77: test.f
INFO: gfortran:f77: /tmp/tmpw74b41uy/src.linux-x86_64-3.11/untitled-f2pywrappers.f
f951: Warning: Nonconforming tab character in column 1 of line 2 [-Wtabs]
INFO: /usr/bin/gfortran -Wall -g -Wall -g -shared /tmp/tmpw74b41uy/tmp/tmpw74b41uy/src.linux-x86_64-3.11/untitledmodule.o /tmp/tmpw74b41uy/tmp/tmpw74b41uy/src.linux-x86_64-3.11/fortranobject.o /tmp/tmpw74b41uy/test.o /tmp/tmpw74b41uy/tmp/tmpw74b41uy/src.linux-x86_64-3.11/untitled-f2pywrappers.o -L/usr/lib64/gcc/x86_64-unknown-linux-gnu/12.2.0/../../../../lib64 -L/usr/lib64/gcc/x86_64-unknown-linux-gnu/12.2.0/../../../../lib64 -L/usr/lib64 -lgfortran -o ./untitled.cpython-311-x86_64-linux-gnu.so
Removing build directory /tmp/tmpw74b41uy
$ ls untitled.*.so
untitled.cpython-311-x86_64-linux-gnu.so

With python 3.12, it doesn't work:

$ python -m numpy.f2py -c test.f 
Cannot use distutils backend with Python 3.12, using meson backend instead.Using meson backend
Will pass --lower to f2py
See https://numpy.org/doc/stable/f2py/buildtools/meson.htmlReading fortran codes...
        Reading file 'test.f' (format:fix,strict)
Post-processing...
        Block: untitled
                        Block: foo
Applying post-processing hooks...
  character_backward_compatibility_hook
Post-processing (stage 2)...
Building modules...
    Building module "untitled"...
    Generating possibly empty wrappers"
    Maybe empty "untitled-f2pywrappers.f"
        Constructing wrapper function "foo"...
          foo()
    Wrote C/API module "untitled" to file "./untitledmodule.c"
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/lib/python3.12/site-packages/numpy/f2py/__main__.py", line 5, in <module>
    main()
  File "/usr/lib/python3.12/site-packages/numpy/f2py/f2py2e.py", line 732, in main
    run_compile()
  File "/usr/lib/python3.12/site-packages/numpy/f2py/f2py2e.py", line 705, in run_compile
    builder.compile()
  File "/usr/lib/python3.12/site-packages/numpy/f2py/_backends/_meson.py", line 130, in compile
    self.write_meson_build(self.build_dir)
  File "/usr/lib/python3.12/site-packages/numpy/f2py/_backends/_meson.py", line 108, in write_meson_build
    src = meson_template.generate_meson_build()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/numpy/f2py/_backends/_meson.py", line 71, in generate_meson_build
    template = Template(self.meson_build_template())
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/numpy/f2py/_backends/_meson.py", line 44, in meson_build_template
    raise FileNotFoundError(
FileNotFoundError: [Errno 2] Meson build template /usr/lib/python3.12/site-packages/numpy/f2py/_backends/meson.build.template does not exist.

@tornaria
Copy link
Contributor

tornaria commented Oct 4, 2023

BTW, it's also the case that /usr/bin/f2py is missing, although that happens also with python 3.11. In fact 1.25.2_1 has the binaries but 1.26.0_1 does not.

For sagemath, we only call f2py as a module so 1.26 with python 3.11 is ok.

@ahesford ahesford force-pushed the py312 branch 2 times, most recently from dfa70f7 to b08ed85 Compare October 4, 2023 17:34
@tornaria
Copy link
Contributor

tornaria commented Oct 5, 2023

I have a candidate for sagemath here: ahesford/void-packages@py312...tornaria:void-packages:py312.

I split the patches in two:

@ahesford
Copy link
Member Author

ahesford commented Oct 6, 2023

Merged at 2628218. Thanks @tornaria for your diligence in getting sagemath ready.

@ahesford ahesford closed this Oct 6, 2023
@ychin
Copy link

ychin commented Oct 11, 2023

Just to follow up on my earlier comment, you should be able to build Vim with Python 3.12 without need to use tricks like turning on stable ABI now (fixed in vim/vim#13290). The flag to use stable ABI is fine to use btw, it's just unnecessary since you aren't using dynamic library loading anyway.

@ahesford
Copy link
Member Author

Thanks for the heads-up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants