PEP 755: Status quo of the CPython 3.12 build system

peps/pep-0755.rst

@@ -0,0 +1,181 @@
+PEP: 755
+Title: Status quo of the CPython 3.12 build system
+Author: Erlend Egeberg Aasland <[email protected]>,
+        Zachary Ware <[email protected]>
+Status: Draft
+Type: Informational
+Content-Type: text/x-rst
+Created: 04-Oct-2023
+Python-Version: 3.12
+
+
+Abstract
+========
+
+This PEP describes the pains core developers have with the CPython build system,
+and their desired goals and wishes for its future.
+
+
+
+Motivation
+==========
+
+The current build system has been criticized by contributors and core
+developers, and the idea of replacing it surfaces every now and then.
+If we are to make large scale changes to the current build system,
+we need to be sure we know what we're getting into.
+
+This PEP intends to document the current build system;
+how is it perceived by contributors and core developers,
+and what kind of build system they envision in the future.
+
+This PEP can be used as input to decide whether to improve the existing build
+system, rewrite the existing build system, or implement a completely new build
+system.
+
+
+Poll Results
+============
+
+`June 20, 2023 <https://discuss.python.org/t/28197>`__, we asked:
+
+   Hypothetically, if we had to recreate the build process from just the
+   available code today, which qualities and features would we want to see in
+   tomorrow’s new build system?
+
+`August 17, 2023 <https://discuss.python.org/t/31815>`__, we asked:
+
+   [...] what do you consider pain points of todays build system?
+
+The desire for a unified build system was mentioned several times.
+This need was accompanied by the pain point that we currently have _two_ build
+systems:
+
+1. A Windows build system
+2. A GNU Autoconf/Make build system
+
+It is hard to keep the Windows and \*nix build systems in sync.
+Windows developers often find the GNU build system incomprehensible;
+\*nix developers find the Windows build system incomprehensible.
+Many developers simply avoid the build system completely.
+
+Also mentioned multiple times was the desire for correct incremental builds.
+This need was accompanied by the observation that today's build system
+does _not_ guarantee this.
+Related was the desire for fast incremental builds and deterministic builds.
+The latter is not guaranteed by the current build system.
+Deterministic builds imply the ability to easily constrain dependencies,
+a need that was also mentioned.
+
+Several people also mentioned the desire to support specific use cases,
+such as cross-compilation support,
+being easily reusable and extendable by 3rd parties who build from source,
+and being able to produce an installed-shape layout.
+
+Ease of maintenance and supporting all PEP-11 platforms was also mentioned
+as end goals.
+
+Some particular pain points was mentioned:
+
+* the configure step is slow and full of serialized checks
+* configure is re-run seemingly at random
+* generated files must be updated manually
+* some code generation scripts in ``Tools/`` duplicate data defined in the
+  standard library
+
+The current build system contains decades of cruft that may or may not still
+be relevant to anyone.
+
+Last, we want a build system where developers and contributors are comfortable
+with making and reviewing changes to it, in years to come.
+This implies some needs:
+
+* good quality documentation
+* support, for example in the form of an active community
+* a build system that is being actively developed
+
+
+The Current Build System
+========================
+
+GNU Autoconf and Make
+---------------------
+
+:file:`Makefile.pre.in` and :file:`configure.ac` make up most of
+the GNU Autoconf/Make build system.
+GNU Autoconf is based on the M4 macro language;
+it takes care of the configure step,
+which includes detecting 3rd party dependencies,
+compiler and linker traits, platform specifics, etc.
+It also provides a command-line interface so the user can
+enable and disable various features.
+
+At the end of the configure step, the :file:`pyconfig.h` header file
+and the Makefile are generated.
+The Makefile file contains a set of hand-written targets and build dependencies,
+and takes care of the build and test use cases.
+Keeping the Makefile compatible with both BSD and GNU Make is a pain point.
+
+Few developers, core devs included, have deep M4 and GNU Autoconf knowledge,
+and it is hard to find contributors and new core developers with these skills.
+Also, GNU Autoconf has no regular releases,
+and it has few active core developers.
+
+
+The Windows Build System
+------------------------
+
+The Windows build consists of a collection of :file:`.vcxproj` and
+:file:`.proj` XML files used by the MSBuild tool to
+generate binaries and perform some other file generation tasks, and a
+series of :file:`.bat` files to prepare the environment, drive MSBuild,
+and perform other tasks such as running tests.  The XML files were
+originally generated by Visual Studio, but have been extensively modified
+or rewritten by hand, and are now maintained manually.
+
+
+Use Cases
+=========
+
+The most common use case is the one performed by the CI:
+
+1. configure
+2. build
+3. test
+
+We want the CI as fast as possible, so it is imperative,
+as mentioned in the poll responses, that the build system is fast.
+Currently, the CI is implemented using GitHub Actions,
+so for convenience, the build system should be supported by all GitHub runners.
+
+For core developers, a common use case is _reconfigure_ and _rebuild_:
+this was reflected in the poll responses
+desiring fast and correct incremental builds.
+
+
+Build Issues in the Bug Tracker
+===============================
+
+As of 2023-10-04, there are 266 open and 3566 closed issues with
+the ``build`` label on our bug tracker.
+Of these, 69 open and 324 closed issues are marked as feature requests,
+and 76 open and 467 closed issues are marked as bugs.
+Grouping by platform, 31 open and 368 closed are Windows specific issues;
+19 open and 204 closed are macOS specific issues.
+
+
+Copyright
+=========
+
+This document is placed in the public domain or under the
+CC0-1.0-Universal license, whichever is more permissive.
+
+
+..
+    Local Variables:
+    mode: indented-text
+    indent-tabs-mode: nil
+    sentence-end-double-space: t
+    fill-column: 70
+    coding: utf-8
+    End: