Unvendor libmpdec sources

[zware]

To facilitate cleaner updates of the externally-maintained libmpdec required by the _decimal module, we should migrate away from the bundled copy in Modules/_decimal/libmpdec and towards an "external" in cpython-source-deps for Windows and --with-system-libmpdec by default elsewhere.

My tentative plan is as follows:

• In 3.13:
  • Add mpdecimal-2.5.1 and mpdecimal-4.0.0 to cpython-source-deps:
    • Sort list of downloads, and add link to mpdecimal sources cpython-source-deps#32
    • https://github.com/python/cpython-source-deps/tree/mpdecimal
  • Switch Windows build to mpdecimal-2.5.1 external (gh-115119: Switch Windows build to mpdecimal external #115182)
  • gh-115119: Detect _decimal dependencies using pkg-config #115406
  • Update deps and build instructions wrt. libmpdecimal devguide#1313
  • gh-115119: Update macOS installer to build with libmpdec 4.0.0 #118382
  • gh-115119: Update Windows installer to mpdecimal 4.0.0 #118506
  • Update macOS build to mpdecimal-4.0.0
  • gh-115119: Default to --with-system-libmpdec=yes #118539
  • gh-115119: Bump CI to use Ubuntu 22.04 #118631
  • Post #118539: amend Ubuntu CI to use system installed mpdecimal
  • Update the docs; see Unvendor libmpdec sources #115119 (comment)
  • gh-115119: Fall back to bundled libmpdec if system libmpdec is not found #119196
• In 3.14:
  • Remove the implicit fallback to the bundled copy
  • Keep support for explicit --without-system-libmpdec (with warning)
• In 3.15:
  • Remove --with[out]-system-libmpdec configure options and Modules/_decimal/libmpdec/

Linked PRs

• gh-115119: Switch Windows build to mpdecimal external #115182
• gh-115119: Detect _decimal dependencies using pkg-config #115406
• gh-115119: Update macOS installer to build with libmpdec 4.0.0 #118382
• gh-115119: Update Windows installer to mpdecimal 4.0.0 #118506
• gh-115119: Default to --with-system-libmpdec=yes #118539
• gh-115119: Bump CI to use Ubuntu 22.04 #118631
• [3.12] gh-115119: Bump CI to use Ubuntu 22.04 (#118631) #118634
• gh-115119: Fall back to bundled libmpdec if system libmpdec is not found #119196
• [3.13] gh-115119: Fall back to bundled libmpdec if system libmpdec is not found (GH-119196) #119217
• gh-115119: Removed bundled copy of the libmpdec #133964
• gh-115119: defer removal of bundled libmpdec to 3.16 #133997

[erlend-aasland]

[...] with a fallback to the bundled copy (maybe with a warning?)

+1 for an autoconf warning.

[ned-deily]

Adding an external libmpdec to the macOS installer build shouldn't be a big issue. Assign it to me if you proceed with this.

[erlend-aasland]

FTR: the source deps repo is now updated.

[erlend-aasland]

@ned-deily: for build-installer.py, is there more to do than adding a configuration for mpdecimal? I've used the following configure options: ['--disable-cxx', 'MACHINE=universal']

[ned-deily]

Likely yes, I'll take care of it. Thanks.

[ned-deily]

I took a quick look at this and be aware that this change proposal will likely impact everyone building Python 3.13 on macOS. macOS does not ship a copy of libmpdec so, if the bundled version is removed, anyone building Python on macOS will now need to provide a local copy, either building from source or using a version from one of the widely-used third-party package managers, like Homebrew or MacPorts. Both currently have mpdecimal packages at the 4.0.0 level. (Recall that, unlike Windows builds, we do not currently provide non-vendored third-party library binaries for macOS users, other than those built for and linked into the Pythons provided by python.org installers for macOS.) In a quick test, it appeared that with ./configure --with-system-libmpdec today, pkg-config info for libmpdec was not being used. If so, that should be corrected. And, in any case, the devguide instructions for installing dependencies will need to be updated to cover libmpdec. We should add all this and get this working before moving on to changing the installer.

[erlend-aasland]

In a quick test, it appeared that with ./configure --with-system-libmpdec today, pkg-config info for libmpdec was not being used. If so, that should be corrected.

Last time I checked, libmpdec did not provide pkgconfig files:

$ ls $(brew --prefix libmpdec)/lib/pkgconfig
ls: /opt/homebrew/opt/mpdecimal/lib/pkgconfig: No such file or directory
$ cd mpdecimal-2.5.1; find . | grep pkg
$

[ned-deily]

With the current 4.0.0 version, it appears both Homebrew and MacPorts ship them:

$ pkg-config libmpdec --cflags
-I/opt/homebrew/Cellar/mpdecimal/4.0.0/include
$ pkg-config libmpdec --libs
-L/opt/homebrew/Cellar/mpdecimal/4.0.0/lib -lmpdec -lm

...

$ pkg-config libmpdec --libs
-L/opt/macports/lib -lmpdec -lm

[erlend-aasland]

Yes, the 4.0.0 version does indeed provide pkg-config configuration files. I'll work on an Autoconf patch.

[erlend-aasland]

@zware are we still aiming for 3.13 for the first batch of items?

[erlend-aasland]

In a quick test, it appeared that with ./configure --with-system-libmpdec today, pkg-config info for libmpdec was not being used. If so, that should be corrected.

Resolved by:

• gh-115119: Detect _decimal dependencies using pkg-config #115406

[vfazio]

Is the plan to still unvendor this if Debian doesn't reinstate the package?

[skirpichev]

Why Debian should be a special snowflake for CPython? Referenced bug thread ignores any feedback, including comments from CPython core dev. I think it's a Debian problem, not ours.

[vfazio]

Well, Debian is a major distro and the root of other distros like Ubuntu. But i guess they'll figure it out once Python fails to build

[skirpichev]

Well, Debian is a major distro and the root of other distros like Ubuntu.

Say also thanks to people, who aren't part of the Debian maintainers now.

But i guess they'll figure it out once Python fails to build

It will not fail. IIUC, with --with-system-libmpdec=no & without bundled sources it will use the pure-Python decimal module, see above: #115119 (comment).

Apparently they don't notice this.

[vfazio]

I missed the decimal -> _decimal or _pydecimal so thanks for pointing that out. I'll have to update my build scripts so i don't suffer the performance penalty

[skirpichev]

I don't think you can do here something. Once the CPython will not bundle libmpdec - we will have no choice but fallback to the _pydecimal if --with-system-libmpdec=no.

Someone must do real job and package back the libmpdec for Debian.

[vfazio]

Hmm..

I'll have to look into it more. I'll was expecting to have to --with-system-libmpdec=yes and specify LIBMPDEC_CFLAGS and LIBMPDEC_LIBS to work around the calls to pkg-config. Worst case I compile and deploy the files to a custom directory and set PKG_CONFIG_LIBDIR and let the build system figure it out without the overrides.

Obviously this is not ideal, but until us Debian users get some traction on that issue, that's probably the best I can hope for.

When I get some spare time I'll test out a build with the vendored code nuked and see if the overrides work like I expect.

[skirpichev]

I'll was expecting to have to --with-system-libmpdec=yes and specify LIBMPDEC_CFLAGS and LIBMPDEC_LIBS to work around the calls to pkg-config.

Ah, yes - that will work.

[tanzim]

What would be the guidance for folks building embedded distributions this case? Use relevant CFLAGS to relatively link to a copy of libmpdec shipped with the embedded distribution?

[erlend-aasland]

Is the plan to still unvendor this if Debian doesn't reinstate the package?

Ondřej Surý has stepped up to reinstate and maintain mpdecimal in Debian:

https://salsa.debian.org/debian/mpdecimal

[vfazio]

Is the plan to still unvendor this if Debian doesn't reinstate the package?

Ondřej Surý has stepped up to reinstate and maintain mpdecimal in Debian:

https://salsa.debian.org/debian/mpdecimal

That's great! Thanks for the update.

[skirpichev]

@zware, it seems we are out of scheduled process in the 3.14 - as fallback to the bundled copy is still here. Shall we remove that in beta? Or we can skip this and just remove the mpdecimal copy in 3.15, together with the configure.ac fallback?

BTW, it seems the mpdecimal is in the Debian again. Probably there are no good reasons to not follow to schedule in 3.15. CC @erlend-aasland

[erlend-aasland]

@tanzim:

What would be the guidance for folks building embedded distributions this case? Use relevant CFLAGS to relatively link to a copy of libmpdec shipped with the embedded distribution?

Yes, as you would do with any of the other third party dependencies.

@skirpichev:

My gut reaction would be to simply postpone everything with one release. That is, do in 3.15 what was planned for 3.14.

[skirpichev]

CC @hugovk, is this too late for 3.14 to remove fallback to bundled copy? This wasn't announced in 3.13, but removal of bundled copy in 3.15 - was. Sorry for asking, just to be sure.

Will you also suggest to shift planned stuff by one release?

[hugovk]

I've not read through the ~90 comments here, but seems this is not a bugfix and there's no pressing need for this in 3.14, so I suggest we do as @erlend-aasland said:

My gut reaction would be to simply postpone everything with one release. That is, do in 3.15 what was planned for 3.14.

Devguide reminder:

After a first beta release is published, no new features are accepted. Only bug fixes and improvements to documentation and tests can now be committed. This is when core developers should concentrate on the task of fixing regressions and other new issues filed by users who have downloaded the alpha and beta releases.

[skirpichev]

I've not read through the ~90 comments here

The evil plan is right in the description.

seems this is not a bugfix

I'm in doubts, but rather "yes".

and there's no pressing need for this in 3.14, so I suggest we do as @erlend-aasland said

Ok, then I'll prepare a documentation patch for 3.13/14 and remove fallback in 3.15.

[AA-Turner]

@skirpichev I'm a little confused, we shouldn't change anything in 3.14 at this point, but the 3.15 development cycle is open. Do you intend to change anything in Python 3.15 relating to the vendored libmpdec?

I've no objections to delaying the removal of the vendored sources to 3.16, I'd just like to understand the rationale.

A

[skirpichev]

we shouldn't change anything in 3.14 at this point, but the 3.15 development cycle is open.

The puzzle is the plan (see description) to:

1. Remove the implicit fallback to the bundled copy (at 3.14), then
2. remove bundled copy (next release, i.e. 3.15).

Only 2) was announced and doing 1) seems now impossible for the 3.14. So, we have options:

• a) just drop 1) and remove bundled copy in the 3.15, as planned, in one shot.
• b) shift schedule by one release (i.e. do 1) in 3.15 and 2) in 3.16).

Our bundled copy is 2.5.1 vs 4.0.1 latest. But I don't see big changes. So, probably removing is not urgent and we should prefer b), as suggested above. I prefer a), for the record.