From b9878e6f3c6b61cb9278d44d9dcbfb742561037d Mon Sep 17 00:00:00 2001 From: Jean Abou Samra Date: Sat, 30 Dec 2023 14:38:36 +0100 Subject: [PATCH 01/18] Update tool recommendations Fixes #1468 --- source/guides/tool-recommendations.rst | 183 +++++++++++++---------- source/guides/writing-pyproject-toml.rst | 1 + 2 files changed, 107 insertions(+), 77 deletions(-) diff --git a/source/guides/tool-recommendations.rst b/source/guides/tool-recommendations.rst index 1a59201fa..2e7c64173 100644 --- a/source/guides/tool-recommendations.rst +++ b/source/guides/tool-recommendations.rst @@ -4,111 +4,140 @@ Tool recommendations ==================== -If you're familiar with Python packaging and installation, and just want to know -what tools are currently recommended, then here it is. +The Python packaging landscape consists of many different tools. For many tasks, +the Python Packaging Authority (PyPA, the umbrella organization which +encompasses many packaging tools and maintains this guide) purposefully does not +make a blanket recommendation; for example, the reason there exist many build +backends is that the landscape was opened in order to enable the development of +new backends serving certain users' needs better than the previously unique +backend, setuptools. This guide does point to some tools that are widely +recognized, and also makes some recommendations of tools that you should *not* +use because they are deprecated or insecure. + + +Virtual environments +==================== + +The standard tools to create and use virtual environments manually are +:ref:`virtualenv` (PyPA project) and :doc:`venv ` (part of +the Python standard library, though missing some features of virtualenv). + + +Installing packages +=================== + +:ref:`Pip` is a standard tool to install packages from :term:`PyPI `. It can install into the global environment or into +virtual environments. You may want to read pip's recommendations for +:doc:`secure installs `. Pip is available by default +in most Python installations. + +Alternatively, for installing Python command line applications specifically, +consider :ref:`pipx`, which is a wrapper around pip that installs each +application into a dedicated virtual environment in order to avoid conflicts +with other applications and, on Linux, conflicts with the system. + +For scientific software specifically, consider :ref:`Conda` or :ref:`Spack`. + +.. todo:: Write a "pip vs. Conda" comparison, here or in a new discussion. + +Do **not** use ``easy_install`` (part of :ref:`setuptools`), which is deprecated +in favor of pip (see :ref:`pip vs easy_install` for details). Likewise, do +**not** use ``python setup.py install`` or ``python setup.py develop``, which +are also deprecated (see :ref:`setup-py-deprecated` for background and +:ref:`modernize-setup-py-project` for migration advice). + + +Lock files +========== +:ref:`pip-tools` and :ref:`Pipenv` are two recognized tools to create lock +files, which contain the exact versions of all packages installed into an +environment, for reproducibility purposes. -Application dependency management -================================= -* Use :ref:`pip` in a `secure manner`_ to install a Python application and its - dependencies during deployment. +Build backends +============== -* Use :ref:`virtualenv` or :doc:`venv ` to isolate - application-specific dependencies from a shared Python installation. [4]_ +Popular :term:`build backends ` for pure-Python packages include: -* Use `pip-tools`_, :ref:`pipenv`, or `poetry`_ to generate the fully-specified - application-specific dependencies, when developing Python applications. +- Hatchling, which is part of :ref:`Hatch` (but can be used without + Hatch as well). Hatchling is extensible through a plugin system. -.. _secure manner: https://pip.pypa.io/en/latest/topics/secure-installs/ -.. _pip-tools: https://github.com/jazzband/pip-tools -.. _Poetry: https://python-poetry.org/ +- :ref:`setuptools`, the historical build backend. It can be configured + programmatically through the :file:`setup.py` file. -Installation tool recommendations -================================= + If you use setuptools, please be aware that it contains many deprecated + features which are currently kept for compatibility, but should not be used. + For example, do not use ``python setup.py`` invocations + (cf. :ref:`setup-py-deprecated`), the ``python_requires`` argument to + ``setup()`` (use the :ref:`[build-system] table + ` of :file:`pyproject.toml` instead), or + the ``easy_install`` command (cf. :ref:`pip vs easy_install`). -* Use :ref:`pip` to install Python :term:`packages ` from - :term:`PyPI `. [1]_ [2]_ Depending on how :ref:`pip` - is installed, you may need to also install :ref:`wheel` to get the benefit - of wheel caching. [3]_ +- Flit-core, part of :ref:`Flit` (but usable standalone). It is meant to be a + minimal and opinionated build backend. It is not extensible. -* Use :ref:`virtualenv` or :doc:`venv ` to isolate - project-specific dependencies from a shared Python installation. [4]_ +- PDM-backend_, part of :ref:`PDM` (but usable standalone). It provides build + hooks for extensibility. -* If you're looking for management of fully integrated cross-platform software - stacks, consider: +- Poetry-core, part of :ref:`Poetry` (but usable standalone). It is extensible + through plugins. - * :ref:`buildout`: primarily focused on the web development community +Do **not** use distutils, which is deprecated, and has been removed from the +standard library in Python 3.12, although it still remains available from +setuptools. - * :ref:`spack`, :ref:`hashdist`, or :ref:`conda`: primarily focused - on the scientific community. +For packages with :term:`extension modules `, you may use +setuptools, but consider using a build system dedicated to the language the +extension is written in, such as Meson or CMake for C/C++, or Cargo for Rust, +and bridging this build system to Python using a dedicated build backend: +- :ref:`meson-python` for Meson, -Packaging tool recommendations -============================== +- :ref:`scikit-build-core` for CMake, -* Use :ref:`setuptools` to define projects. [5]_ [6]_ +- :ref:`maturin` for Cargo. -* Use :ref:`build` to create :term:`Source Distributions - ` and :term:`wheels `. -If you have binary extensions and want to distribute wheels for multiple -platforms, use :ref:`cibuildwheel` as part of your CI setup to build -distributable wheels. +Building distributions +====================== -* Use `twine `_ for uploading distributions - to :term:`PyPI `. +The standard tool to build :term:`source distributions ` and :term:`wheels ` for uploading to PyPI is :ref:`build`. It +will invoke whichever build backend you :ref:`declared +` in :file:`pyproject.toml`. +Do **not** use ``python setup.py sdist`` and ``python setup.py bdist_wheel`` for +this task. All direct invocations of :file:`setup.py` are :ref:`deprecated +`. -Publishing platform migration -============================= +If you have :term:`extension modules ` and want to distribute +wheels for multiple platforms, use :ref:`cibuildwheel` as part of your CI setup +to build distributable wheels. -The original Python Package Index implementation (previously hosted at -`pypi.python.org `_) has been phased out in favour -of an updated implementation hosted at `pypi.org `_. -See :ref:`Migrating to PyPI.org` for more information on the status of the -migration, and what settings to change in your clients. +Uploading to PyPI +================= ----- +The standard tool for this task is :ref:`twine`. -.. [1] There are some cases where you might choose to use ``easy_install`` (from - :ref:`setuptools`), e.g. if you need to install from :term:`Eggs ` - (which pip doesn't support). For a detailed breakdown, see :ref:`pip vs - easy_install`. +**Never** use ``python setup.py upload`` for this task. In addition to being +:ref:`deprecated `, it is insecure. -.. [2] The acceptance of :pep:`453` means that :ref:`pip` - will be available by default in most installations of Python 3.4 or - later. See the :pep:`rationale section <453#rationale>` from :pep:`453` - as for why pip was chosen. -.. [3] `get-pip.py `_ and - :ref:`virtualenv` install - :ref:`wheel`, whereas :ref:`ensurepip` and :ref:`venv ` do not - currently. Also, the common "python-pip" package that's found in various - linux distros, does not depend on "python-wheel" currently. +Integrated workflow tools +========================= -.. [4] Beginning with Python 3.4, ``venv`` will create virtualenv environments - with ``pip`` installed, thereby making it an equal alternative to - :ref:`virtualenv`. However, using :ref:`virtualenv` will still be - recommended for users that need cross-version consistency. +These are tools that combine many features in one command line application, such +as automatically managing virtual environments for a project, building +distributions, uploading to PyPI, or creating and using lock files. -.. [5] Although you can use pure :ref:`distutils` for many projects, it does not - support defining dependencies on other projects and is missing several - convenience utilities for automatically populating distribution metadata - correctly that are provided by ``setuptools``. Being outside the - standard library, ``setuptools`` also offers a more consistent feature - set across different versions of Python, and (unlike ``distutils``), - recent versions of ``setuptools`` support all of the modern metadata - fields described in :ref:`core-metadata`. +- :ref:`Hatch`, +- :ref:`Flit`, +- :ref:`PDM`, +- :ref:`Poetry`. - Even for projects that do choose to use ``distutils``, when :ref:`pip` - installs such projects directly from source (rather than installing - from a prebuilt :term:`wheel ` file), it will actually build - your project using :ref:`setuptools` instead. -.. [6] `distribute`_ (a fork of setuptools) was merged back into - :ref:`setuptools` in June 2013, thereby making setuptools the default - choice for packaging. -.. _distribute: https://pypi.org/project/distribute +.. _pdm-backend: https://backend.pdm-project.org diff --git a/source/guides/writing-pyproject-toml.rst b/source/guides/writing-pyproject-toml.rst index 187a44d50..938ebbb93 100644 --- a/source/guides/writing-pyproject-toml.rst +++ b/source/guides/writing-pyproject-toml.rst @@ -40,6 +40,7 @@ three possible TOML tables in this file. :ref:`setup-py-deprecated`. +.. _pyproject-guide-build-system-table: Declaring the build backend =========================== From dc4ff6692806bb4aa35b00eeb630b2fa6bd142ee Mon Sep 17 00:00:00 2001 From: Jean Abou Samra Date: Sat, 30 Dec 2023 16:20:14 +0100 Subject: [PATCH 02/18] Address @chrysle's review --- source/guides/tool-recommendations.rst | 34 ++++++++++++++------------ 1 file changed, 19 insertions(+), 15 deletions(-) diff --git a/source/guides/tool-recommendations.rst b/source/guides/tool-recommendations.rst index 2e7c64173..a34df5d65 100644 --- a/source/guides/tool-recommendations.rst +++ b/source/guides/tool-recommendations.rst @@ -5,14 +5,15 @@ Tool recommendations ==================== The Python packaging landscape consists of many different tools. For many tasks, -the Python Packaging Authority (PyPA, the umbrella organization which -encompasses many packaging tools and maintains this guide) purposefully does not -make a blanket recommendation; for example, the reason there exist many build -backends is that the landscape was opened in order to enable the development of -new backends serving certain users' needs better than the previously unique -backend, setuptools. This guide does point to some tools that are widely -recognized, and also makes some recommendations of tools that you should *not* -use because they are deprecated or insecure. +the :term:`Python Packaging Authority ` +(PyPA, the umbrella organization which encompasses many packaging tools and +maintains this guide) purposefully does not make a blanket recommendation; for +example, the reason there exist many build backends is that the landscape was +opened in order to enable the development of new backends serving certain users' +needs better than the previously unique backend, setuptools. This guide does +point to some tools that are widely recognized, and also makes some +recommendations of tools that you should *not* use because they are deprecated +or insecure. Virtual environments @@ -26,7 +27,7 @@ the Python standard library, though missing some features of virtualenv). Installing packages =================== -:ref:`Pip` is a standard tool to install packages from :term:`PyPI `. It can install into the global environment or into virtual environments. You may want to read pip's recommendations for :doc:`secure installs `. Pip is available by default @@ -34,8 +35,10 @@ in most Python installations. Alternatively, for installing Python command line applications specifically, consider :ref:`pipx`, which is a wrapper around pip that installs each -application into a dedicated virtual environment in order to avoid conflicts -with other applications and, on Linux, conflicts with the system. +application into a dedicated virtual environment. This avoids conflicts between +the dependencies of different applications. On Linux, pipx is especially +important because it also avoids conflicts with the system (which are the reason +for :ref:`externally managed environments `). For scientific software specifically, consider :ref:`Conda` or :ref:`Spack`. @@ -65,11 +68,12 @@ Popular :term:`build backends ` for pure-Python packages include: Hatch as well). Hatchling is extensible through a plugin system. - :ref:`setuptools`, the historical build backend. It can be configured - programmatically through the :file:`setup.py` file. + programmatically through the :file:`setup.py` file (but for basic metadata, + :file:`pyproject.toml` is preferred). If you use setuptools, please be aware that it contains many deprecated features which are currently kept for compatibility, but should not be used. - For example, do not use ``python setup.py`` invocations + For example, do **not** use ``python setup.py`` invocations (cf. :ref:`setup-py-deprecated`), the ``python_requires`` argument to ``setup()`` (use the :ref:`[build-system] table ` of :file:`pyproject.toml` instead), or @@ -84,8 +88,8 @@ Popular :term:`build backends ` for pure-Python packages include: - Poetry-core, part of :ref:`Poetry` (but usable standalone). It is extensible through plugins. -Do **not** use distutils, which is deprecated, and has been removed from the -standard library in Python 3.12, although it still remains available from +Do **not** use :ref:`distutils`, which is deprecated, and has been removed from +the standard library in Python 3.12, although it still remains available from setuptools. For packages with :term:`extension modules `, you may use From f5df94159862516cad4d4d0adf379a644fa8ed61 Mon Sep 17 00:00:00 2001 From: Jean Abou-Samra Date: Sun, 31 Dec 2023 14:12:27 +0100 Subject: [PATCH 03/18] Fix brain fart --- source/guides/tool-recommendations.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/guides/tool-recommendations.rst b/source/guides/tool-recommendations.rst index a34df5d65..11c57aa7d 100644 --- a/source/guides/tool-recommendations.rst +++ b/source/guides/tool-recommendations.rst @@ -74,7 +74,7 @@ Popular :term:`build backends ` for pure-Python packages include: If you use setuptools, please be aware that it contains many deprecated features which are currently kept for compatibility, but should not be used. For example, do **not** use ``python setup.py`` invocations - (cf. :ref:`setup-py-deprecated`), the ``python_requires`` argument to + (cf. :ref:`setup-py-deprecated`), the ``setup_requires`` argument to ``setup()`` (use the :ref:`[build-system] table ` of :file:`pyproject.toml` instead), or the ``easy_install`` command (cf. :ref:`pip vs easy_install`). From 620222cb291413358d6860f33f76210e74fe3456 Mon Sep 17 00:00:00 2001 From: Jean Abou Samra Date: Wed, 3 Jan 2024 12:38:37 +0100 Subject: [PATCH 04/18] Address review comments from @webknjaz and @eli-schwartz --- source/guides/tool-recommendations.rst | 47 +++++++++++++++----------- 1 file changed, 27 insertions(+), 20 deletions(-) diff --git a/source/guides/tool-recommendations.rst b/source/guides/tool-recommendations.rst index 11c57aa7d..1778851d5 100644 --- a/source/guides/tool-recommendations.rst +++ b/source/guides/tool-recommendations.rst @@ -6,10 +6,10 @@ Tool recommendations The Python packaging landscape consists of many different tools. For many tasks, the :term:`Python Packaging Authority ` -(PyPA, the umbrella organization which encompasses many packaging tools and +(PyPA, the working group which encompasses many packaging tools and maintains this guide) purposefully does not make a blanket recommendation; for -example, the reason there exist many build backends is that the landscape was -opened in order to enable the development of new backends serving certain users' +example, the reason there are many build backends is that the landscape was +opened up in order to enable the development of new backends serving certain users' needs better than the previously unique backend, setuptools. This guide does point to some tools that are widely recognized, and also makes some recommendations of tools that you should *not* use because they are deprecated @@ -31,14 +31,15 @@ Installing packages Package Index (PyPI)>`. It can install into the global environment or into virtual environments. You may want to read pip's recommendations for :doc:`secure installs `. Pip is available by default -in most Python installations. +in most Python installations through the standard library package +:doc:`ensurepip `. -Alternatively, for installing Python command line applications specifically, -consider :ref:`pipx`, which is a wrapper around pip that installs each +Alternatively, consider :ref:`pipx` for the specific use case of installing Python +command line applications that are distributed through PyPI. +Pipx is a wrapper around pip and venv that installs each application into a dedicated virtual environment. This avoids conflicts between -the dependencies of different applications. On Linux, pipx is especially -important because it also avoids conflicts with the system (which are the reason -for :ref:`externally managed environments `). +the dependencies of different applications, and also with the system +(especially on Linux). For scientific software specifically, consider :ref:`Conda` or :ref:`Spack`. @@ -64,10 +65,10 @@ Build backends Popular :term:`build backends ` for pure-Python packages include: -- Hatchling, which is part of :ref:`Hatch` (but can be used without - Hatch as well). Hatchling is extensible through a plugin system. +- Hatchling_, which is developed along with :ref:`Hatch`, but is separate and can + be used without Hatch. Hatchling is extensible through a plugin system. -- :ref:`setuptools`, the historical build backend. It can be configured +- :ref:`setuptools` (which used to be the only build backend). It can be configured programmatically through the :file:`setup.py` file (but for basic metadata, :file:`pyproject.toml` is preferred). @@ -79,13 +80,13 @@ Popular :term:`build backends ` for pure-Python packages include: ` of :file:`pyproject.toml` instead), or the ``easy_install`` command (cf. :ref:`pip vs easy_install`). -- Flit-core, part of :ref:`Flit` (but usable standalone). It is meant to be a +- Flit-core_ (developed with but separate from :ref:`Flit`). It is meant to be a minimal and opinionated build backend. It is not extensible. -- PDM-backend_, part of :ref:`PDM` (but usable standalone). It provides build +- PDM-backend_ (developed with but separate from :ref:`PDM`). It provides build hooks for extensibility. -- Poetry-core, part of :ref:`Poetry` (but usable standalone). It is extensible +- Poetry-core_ (developed with but separate from :ref:`Poetry`). It is extensible through plugins. Do **not** use :ref:`distutils`, which is deprecated, and has been removed from @@ -93,9 +94,11 @@ the standard library in Python 3.12, although it still remains available from setuptools. For packages with :term:`extension modules `, you may use -setuptools, but consider using a build system dedicated to the language the -extension is written in, such as Meson or CMake for C/C++, or Cargo for Rust, -and bridging this build system to Python using a dedicated build backend: +setuptools. However, you can also use a build system with dedicated support for +the language the extension is written in. For example, you could choose Meson or +CMake for C, C++, Fortran and many other compiled languages; or Cargo for Rust +specifically. You can then bridge this build system to Python using a dedicated +build backend: - :ref:`meson-python` for Meson, @@ -135,13 +138,17 @@ Integrated workflow tools These are tools that combine many features in one command line application, such as automatically managing virtual environments for a project, building -distributions, uploading to PyPI, or creating and using lock files. +distributions, uploading to PyPI, or creating and using (tool-specific) lock +files. They often call the tools mentioned above under the hood. - :ref:`Hatch`, - :ref:`Flit`, - :ref:`PDM`, - :ref:`Poetry`. +- :ref:`Pipenv` - +.. _flit-core: https://pypi.org/project/flit-core/ +.. _hatchling: https://pypi.org/project/hatchling/ .. _pdm-backend: https://backend.pdm-project.org +.. _poetry-core: https://pypi.org/project/poetry-core/ From fc6f4207a65a0d4bb7db5ae72f54f2c2bb1ce32b Mon Sep 17 00:00:00 2001 From: Jean Abou Samra Date: Wed, 3 Jan 2024 12:40:15 +0100 Subject: [PATCH 05/18] Sort tools alphabetically --- source/guides/tool-recommendations.rst | 28 ++++++++++++++------------ 1 file changed, 15 insertions(+), 13 deletions(-) diff --git a/source/guides/tool-recommendations.rst b/source/guides/tool-recommendations.rst index 1778851d5..b883b60cc 100644 --- a/source/guides/tool-recommendations.rst +++ b/source/guides/tool-recommendations.rst @@ -63,11 +63,21 @@ environment, for reproducibility purposes. Build backends ============== -Popular :term:`build backends ` for pure-Python packages include: +Popular :term:`build backends ` for pure-Python packages include, +in alphabetical order: + +- Flit-core_ (developed with but separate from :ref:`Flit`). It is meant to be a + minimal and opinionated build backend. It is not extensible. - Hatchling_, which is developed along with :ref:`Hatch`, but is separate and can be used without Hatch. Hatchling is extensible through a plugin system. +- PDM-backend_ (developed with but separate from :ref:`PDM`). It provides build + hooks for extensibility. + +- Poetry-core_ (developed with but separate from :ref:`Poetry`). It is extensible + through plugins. + - :ref:`setuptools` (which used to be the only build backend). It can be configured programmatically through the :file:`setup.py` file (but for basic metadata, :file:`pyproject.toml` is preferred). @@ -80,15 +90,6 @@ Popular :term:`build backends ` for pure-Python packages include: ` of :file:`pyproject.toml` instead), or the ``easy_install`` command (cf. :ref:`pip vs easy_install`). -- Flit-core_ (developed with but separate from :ref:`Flit`). It is meant to be a - minimal and opinionated build backend. It is not extensible. - -- PDM-backend_ (developed with but separate from :ref:`PDM`). It provides build - hooks for extensibility. - -- Poetry-core_ (developed with but separate from :ref:`Poetry`). It is extensible - through plugins. - Do **not** use :ref:`distutils`, which is deprecated, and has been removed from the standard library in Python 3.12, although it still remains available from setuptools. @@ -139,13 +140,14 @@ Integrated workflow tools These are tools that combine many features in one command line application, such as automatically managing virtual environments for a project, building distributions, uploading to PyPI, or creating and using (tool-specific) lock -files. They often call the tools mentioned above under the hood. +files. They often call the tools mentioned above under the hood. In alphabetical +order: -- :ref:`Hatch`, - :ref:`Flit`, +- :ref:`Hatch`, - :ref:`PDM`, +- :ref:`Pipenv`, - :ref:`Poetry`. -- :ref:`Pipenv` .. _flit-core: https://pypi.org/project/flit-core/ From 52e7eac7b7bba631a505f05b7d924a0d53622d6b Mon Sep 17 00:00:00 2001 From: Jean Abou-Samra Date: Wed, 3 Jan 2024 17:40:00 +0100 Subject: [PATCH 06/18] Suggestions from @abravalheri Co-authored-by: Anderson Bravalheri --- source/guides/tool-recommendations.rst | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/source/guides/tool-recommendations.rst b/source/guides/tool-recommendations.rst index b883b60cc..3a6ec813c 100644 --- a/source/guides/tool-recommendations.rst +++ b/source/guides/tool-recommendations.rst @@ -79,6 +79,8 @@ in alphabetical order: through plugins. - :ref:`setuptools` (which used to be the only build backend). It can be configured + using modern standards like `pyproject.toml`, but can also be extended + and supports customisation via `setup.py`. programmatically through the :file:`setup.py` file (but for basic metadata, :file:`pyproject.toml` is preferred). @@ -95,11 +97,13 @@ the standard library in Python 3.12, although it still remains available from setuptools. For packages with :term:`extension modules `, you may use -setuptools. However, you can also use a build system with dedicated support for -the language the extension is written in. For example, you could choose Meson or -CMake for C, C++, Fortran and many other compiled languages; or Cargo for Rust -specifically. You can then bridge this build system to Python using a dedicated -build backend: +a build system with dedicated support for the language the extension is written in, +for example: + +- :ref:`setuptools` - natively supports C/C++ (with 3rd-party plugins for Go and Rust); +- :ref:`meson-python` - C, C++, Fortran, and Rust and other languages supported by Meson; +- :ref:`scikit-build-core` - C, C++, Fortran and other languages supported by CMake; +- :ref:`maturin` - Rust, via Cargo. - :ref:`meson-python` for Meson, From 874ebbc35ed3fac5c9669c379387364c59927acc Mon Sep 17 00:00:00 2001 From: Jean Abou Samra Date: Wed, 3 Jan 2024 17:44:02 +0100 Subject: [PATCH 07/18] Minor fixes/improvements --- source/guides/tool-recommendations.rst | 23 ++++++++--------------- 1 file changed, 8 insertions(+), 15 deletions(-) diff --git a/source/guides/tool-recommendations.rst b/source/guides/tool-recommendations.rst index 3a6ec813c..de7cd9e70 100644 --- a/source/guides/tool-recommendations.rst +++ b/source/guides/tool-recommendations.rst @@ -78,11 +78,10 @@ in alphabetical order: - Poetry-core_ (developed with but separate from :ref:`Poetry`). It is extensible through plugins. -- :ref:`setuptools` (which used to be the only build backend). It can be configured - using modern standards like `pyproject.toml`, but can also be extended - and supports customisation via `setup.py`. - programmatically through the :file:`setup.py` file (but for basic metadata, - :file:`pyproject.toml` is preferred). +- :ref:`setuptools` (which used to be the only build backend). It can be + configured using modern standards like the :ref:`[project] table in + pyproject.toml `, but can also be extended and + customized via :file:`setup.py`. If you use setuptools, please be aware that it contains many deprecated features which are currently kept for compatibility, but should not be used. @@ -100,16 +99,10 @@ For packages with :term:`extension modules `, you may use a build system with dedicated support for the language the extension is written in, for example: -- :ref:`setuptools` - natively supports C/C++ (with 3rd-party plugins for Go and Rust); -- :ref:`meson-python` - C, C++, Fortran, and Rust and other languages supported by Meson; -- :ref:`scikit-build-core` - C, C++, Fortran and other languages supported by CMake; -- :ref:`maturin` - Rust, via Cargo. - -- :ref:`meson-python` for Meson, - -- :ref:`scikit-build-core` for CMake, - -- :ref:`maturin` for Cargo. +- :ref:`setuptools` -- natively supports C and C++ (with third-party plugins for Go and Rust), +- :ref:`meson-python` -- C, C++, Fortran, Rust, and other languages supported by Meson, +- :ref:`scikit-build-core` -- C, C++, Fortran, and other languages supported by CMake, +- :ref:`maturin` -- Rust, via Cargo. Building distributions From 6d285fe3ea64f6cda14879e5b5e2d7e950adae6b Mon Sep 17 00:00:00 2001 From: Jean Abou Samra Date: Wed, 3 Jan 2024 17:58:31 +0100 Subject: [PATCH 08/18] Mention tox and nox --- source/guides/tool-recommendations.rst | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/source/guides/tool-recommendations.rst b/source/guides/tool-recommendations.rst index de7cd9e70..0c7be1f54 100644 --- a/source/guides/tool-recommendations.rst +++ b/source/guides/tool-recommendations.rst @@ -131,6 +131,15 @@ The standard tool for this task is :ref:`twine`. :ref:`deprecated `, it is insecure. +Task runners +============ + +These tools allow you to define and invoke "tasks", such as running tests, +compiling documentation, regenerating some files, etc. Each task can be executed +in a dedicated virtual environment with the dependencies it requires. The most +common tools for this are :ref:`tox` and :ref:`nox`. + + Integrated workflow tools ========================= From 8c34a17ed313c188d1ef3dee81c9962edb7270dd Mon Sep 17 00:00:00 2001 From: Jean Abou Samra Date: Wed, 3 Jan 2024 18:03:07 +0100 Subject: [PATCH 09/18] Fix tox/nox links --- source/guides/tool-recommendations.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/source/guides/tool-recommendations.rst b/source/guides/tool-recommendations.rst index 0c7be1f54..80769f331 100644 --- a/source/guides/tool-recommendations.rst +++ b/source/guides/tool-recommendations.rst @@ -137,7 +137,7 @@ Task runners These tools allow you to define and invoke "tasks", such as running tests, compiling documentation, regenerating some files, etc. Each task can be executed in a dedicated virtual environment with the dependencies it requires. The most -common tools for this are :ref:`tox` and :ref:`nox`. +common tools for this are tox_ and nox_. Integrated workflow tools @@ -158,5 +158,7 @@ order: .. _flit-core: https://pypi.org/project/flit-core/ .. _hatchling: https://pypi.org/project/hatchling/ +.. _nox: https://nox.thea.codes .. _pdm-backend: https://backend.pdm-project.org .. _poetry-core: https://pypi.org/project/poetry-core/ +.. _tox: https://tox.wiki From dc02b6d8a361809bb8ab96920f7876d678b6b4e8 Mon Sep 17 00:00:00 2001 From: Jean Abou-Samra Date: Thu, 4 Jan 2024 00:01:00 +0100 Subject: [PATCH 10/18] Rewording from @abravalheri Co-authored-by: Anderson Bravalheri --- source/guides/tool-recommendations.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/source/guides/tool-recommendations.rst b/source/guides/tool-recommendations.rst index 80769f331..c9b15d164 100644 --- a/source/guides/tool-recommendations.rst +++ b/source/guides/tool-recommendations.rst @@ -83,8 +83,8 @@ in alphabetical order: pyproject.toml `, but can also be extended and customized via :file:`setup.py`. - If you use setuptools, please be aware that it contains many deprecated - features which are currently kept for compatibility, but should not be used. + If you use setuptools, please be aware that some features that predate + standardisation efforts are now deprecated and only *temporarily kept* for compatibility. For example, do **not** use ``python setup.py`` invocations (cf. :ref:`setup-py-deprecated`), the ``setup_requires`` argument to ``setup()`` (use the :ref:`[build-system] table From 9671ddbd04576dd068d332780d516b41bfd6da3d Mon Sep 17 00:00:00 2001 From: Jean Abou Samra Date: Tue, 16 Jan 2024 19:09:39 +0100 Subject: [PATCH 11/18] Various minor rewordings --- source/guides/tool-recommendations.rst | 43 +++++++++++++++----------- 1 file changed, 25 insertions(+), 18 deletions(-) diff --git a/source/guides/tool-recommendations.rst b/source/guides/tool-recommendations.rst index c9b15d164..cb5bb549a 100644 --- a/source/guides/tool-recommendations.rst +++ b/source/guides/tool-recommendations.rst @@ -66,36 +66,43 @@ Build backends Popular :term:`build backends ` for pure-Python packages include, in alphabetical order: -- Flit-core_ (developed with but separate from :ref:`Flit`). It is meant to be a - minimal and opinionated build backend. It is not extensible. +- Flit-core_ -- developed with but separate from :ref:`Flit`. A minimal and + opinionated build backend. It does not support plugins. -- Hatchling_, which is developed along with :ref:`Hatch`, but is separate and can - be used without Hatch. Hatchling is extensible through a plugin system. +- Hatchling_ -- developed with but separate from :ref:`Hatch`. Supports plugins. -- PDM-backend_ (developed with but separate from :ref:`PDM`). It provides build - hooks for extensibility. +- PDM-backend_ -- developed with but separate from :ref:`PDM`. Supports plugins. -- Poetry-core_ (developed with but separate from :ref:`Poetry`). It is extensible - through plugins. +- Poetry-core_ -- developed with but separate from :ref:`Poetry`. Supports + plugins. -- :ref:`setuptools` (which used to be the only build backend). It can be - configured using modern standards like the :ref:`[project] table in - pyproject.toml `, but can also be extended and - customized via :file:`setup.py`. + Unlike other backends on this list, Poetry-core does not support the standard + :ref:`[project] table ` (it uses a different format, + in the ``[tool.poetry]`` table). + +- :ref:`setuptools`, which used to be the only build backend. Supports plugins. If you use setuptools, please be aware that some features that predate - standardisation efforts are now deprecated and only *temporarily kept* for compatibility. - For example, do **not** use ``python setup.py`` invocations - (cf. :ref:`setup-py-deprecated`), the ``setup_requires`` argument to - ``setup()`` (use the :ref:`[build-system] table - ` of :file:`pyproject.toml` instead), or + standardisation efforts are now deprecated and only *temporarily kept* + for compatibility. + + In particular, do **not** use direct ``python setup.py`` invocations. On the + other hand, configuring setuptools with a :file:`setup.py` file is still fully + supported, although it is recommended to use the modern :ref:`[project] table + in pyproject.toml ` whenever possible and keep + :file:`setup.py` only if programmatic configuration is needed. See + :ref:`setup-py-deprecated`. + + Other examples of deprecated features you should **not** use include the + ``setup_requires`` argument to ``setup()`` (use the :ref:`[build-system] table + ` in :file:`pyproject.toml` instead), and the ``easy_install`` command (cf. :ref:`pip vs easy_install`). Do **not** use :ref:`distutils`, which is deprecated, and has been removed from the standard library in Python 3.12, although it still remains available from setuptools. -For packages with :term:`extension modules `, you may use +For packages with :term:`extension modules `, it is best to use a build system with dedicated support for the language the extension is written in, for example: From 80a18861ec58883f54d77263727d058202ae7e78 Mon Sep 17 00:00:00 2001 From: Jean Abou Samra Date: Thu, 25 Jan 2024 10:42:44 +0100 Subject: [PATCH 12/18] Address review comments --- source/guides/tool-recommendations.rst | 43 +++++++++++++------------- 1 file changed, 21 insertions(+), 22 deletions(-) diff --git a/source/guides/tool-recommendations.rst b/source/guides/tool-recommendations.rst index cb5bb549a..788e51b09 100644 --- a/source/guides/tool-recommendations.rst +++ b/source/guides/tool-recommendations.rst @@ -35,11 +35,11 @@ in most Python installations through the standard library package :doc:`ensurepip `. Alternatively, consider :ref:`pipx` for the specific use case of installing Python -command line applications that are distributed through PyPI. +applications that are distributed through PyPI and run from the command line. Pipx is a wrapper around pip and venv that installs each application into a dedicated virtual environment. This avoids conflicts between -the dependencies of different applications, and also with the system -(especially on Linux). +the dependencies of different applications, and also with system-wide applications +making use of the same Python interpreter (especially on Linux). For scientific software specifically, consider :ref:`Conda` or :ref:`Spack`. @@ -66,8 +66,8 @@ Build backends Popular :term:`build backends ` for pure-Python packages include, in alphabetical order: -- Flit-core_ -- developed with but separate from :ref:`Flit`. A minimal and - opinionated build backend. It does not support plugins. +- :doc:`Flit-core ` -- developed with but separate from :ref:`Flit`. + A minimal and opinionated build backend. It does not support plugins. - Hatchling_ -- developed with but separate from :ref:`Hatch`. Supports plugins. @@ -82,21 +82,23 @@ in alphabetical order: - :ref:`setuptools`, which used to be the only build backend. Supports plugins. - If you use setuptools, please be aware that some features that predate - standardisation efforts are now deprecated and only *temporarily kept* - for compatibility. + .. caution:: - In particular, do **not** use direct ``python setup.py`` invocations. On the - other hand, configuring setuptools with a :file:`setup.py` file is still fully - supported, although it is recommended to use the modern :ref:`[project] table - in pyproject.toml ` whenever possible and keep - :file:`setup.py` only if programmatic configuration is needed. See - :ref:`setup-py-deprecated`. + If you use setuptools, please be aware that some features that predate + standardisation efforts are now deprecated and only *temporarily kept* + for compatibility. - Other examples of deprecated features you should **not** use include the - ``setup_requires`` argument to ``setup()`` (use the :ref:`[build-system] table - ` in :file:`pyproject.toml` instead), and - the ``easy_install`` command (cf. :ref:`pip vs easy_install`). + In particular, do **not** use direct ``python setup.py`` invocations. On the + other hand, configuring setuptools with a :file:`setup.py` file is still fully + supported, although it is recommended to use the modern :ref:`[project] table + in pyproject.toml ` whenever possible and keep + :file:`setup.py` only if programmatic configuration is needed. See + :ref:`setup-py-deprecated`. + + Other examples of deprecated features you should **not** use include the + ``setup_requires`` argument to ``setup()`` (use the :ref:`[build-system] table + ` in :file:`pyproject.toml` instead), and + the ``easy_install`` command (cf. :ref:`pip vs easy_install`). Do **not** use :ref:`distutils`, which is deprecated, and has been removed from the standard library in Python 3.12, although it still remains available from @@ -144,7 +146,7 @@ Task runners These tools allow you to define and invoke "tasks", such as running tests, compiling documentation, regenerating some files, etc. Each task can be executed in a dedicated virtual environment with the dependencies it requires. The most -common tools for this are tox_ and nox_. +common tools for this are :doc:`tox ` and :doc:`nox `. Integrated workflow tools @@ -163,9 +165,6 @@ order: - :ref:`Poetry`. -.. _flit-core: https://pypi.org/project/flit-core/ .. _hatchling: https://pypi.org/project/hatchling/ -.. _nox: https://nox.thea.codes .. _pdm-backend: https://backend.pdm-project.org .. _poetry-core: https://pypi.org/project/poetry-core/ -.. _tox: https://tox.wiki From b5cdeb9228b3b7b18e88f06b3834fe006531ac28 Mon Sep 17 00:00:00 2001 From: Jean Abou Samra Date: Thu, 25 Jan 2024 20:46:31 +0100 Subject: [PATCH 13/18] Mention PyPI trusted publishing --- ...ution-releases-using-github-actions-ci-cd-workflows.rst | 2 ++ source/guides/tool-recommendations.rst | 7 ++++++- 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/source/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows.rst b/source/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows.rst index ca75d159b..99ac3a9e2 100644 --- a/source/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows.rst +++ b/source/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows.rst @@ -1,3 +1,5 @@ +.. _trusted-publishing: + ============================================================================= Publishing package distribution releases using GitHub Actions CI/CD workflows ============================================================================= diff --git a/source/guides/tool-recommendations.rst b/source/guides/tool-recommendations.rst index 788e51b09..594191182 100644 --- a/source/guides/tool-recommendations.rst +++ b/source/guides/tool-recommendations.rst @@ -134,7 +134,12 @@ to build distributable wheels. Uploading to PyPI ================= -The standard tool for this task is :ref:`twine`. +For projects hosted on GitHub, PyPI recommends using :ref:`trusted publishing +`, which allows the package to be securely uploaded to PyPI +from a GitHub Actions job. (This is not yet supported on software forges other +than GitHub.) + +The other available method is to upload the package manually using :ref:`twine`. **Never** use ``python setup.py upload`` for this task. In addition to being :ref:`deprecated `, it is insecure. From eab0fd88be7c7a1c2e04601e81f95a4b5f629ef3 Mon Sep 17 00:00:00 2001 From: Jean Abou-Samra Date: Thu, 25 Jan 2024 20:48:05 +0100 Subject: [PATCH 14/18] Mention setup.cfg MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Sviatoslav Sydorenko (Святослав Сидоренко) --- source/guides/tool-recommendations.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/guides/tool-recommendations.rst b/source/guides/tool-recommendations.rst index 594191182..cfbd7d59f 100644 --- a/source/guides/tool-recommendations.rst +++ b/source/guides/tool-recommendations.rst @@ -91,7 +91,7 @@ in alphabetical order: In particular, do **not** use direct ``python setup.py`` invocations. On the other hand, configuring setuptools with a :file:`setup.py` file is still fully supported, although it is recommended to use the modern :ref:`[project] table - in pyproject.toml ` whenever possible and keep + in pyproject.toml ` (or :file:`setup.cfg`) whenever possible and keep :file:`setup.py` only if programmatic configuration is needed. See :ref:`setup-py-deprecated`. From c08cee83b22c4b7aea99a5b3ba62a5c63ae628dc Mon Sep 17 00:00:00 2001 From: Jean Abou-Samra Date: Thu, 25 Jan 2024 20:56:24 +0100 Subject: [PATCH 15/18] =?UTF-8?q?PyPI=20recommends=20=E2=86=92=20it=20is?= =?UTF-8?q?=20recommended?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Sviatoslav Sydorenko (Святослав Сидоренко) --- source/guides/tool-recommendations.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/guides/tool-recommendations.rst b/source/guides/tool-recommendations.rst index cfbd7d59f..88ff6884c 100644 --- a/source/guides/tool-recommendations.rst +++ b/source/guides/tool-recommendations.rst @@ -134,7 +134,7 @@ to build distributable wheels. Uploading to PyPI ================= -For projects hosted on GitHub, PyPI recommends using :ref:`trusted publishing +For projects hosted on GitHub, it is recommended to use the :ref:`trusted publishing `, which allows the package to be securely uploaded to PyPI from a GitHub Actions job. (This is not yet supported on software forges other than GitHub.) From 44f086f8b82694f22a7bd8f9d9b0568da92b529a Mon Sep 17 00:00:00 2001 From: Jean Abou Samra Date: Thu, 25 Jan 2024 22:01:24 +0100 Subject: [PATCH 16/18] Remove sentence about global/virtual environments --- source/guides/tool-recommendations.rst | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/source/guides/tool-recommendations.rst b/source/guides/tool-recommendations.rst index 88ff6884c..9ba538b9d 100644 --- a/source/guides/tool-recommendations.rst +++ b/source/guides/tool-recommendations.rst @@ -28,8 +28,7 @@ Installing packages =================== :ref:`Pip` is the standard tool to install packages from :term:`PyPI `. It can install into the global environment or into -virtual environments. You may want to read pip's recommendations for +Package Index (PyPI)>`. You may want to read pip's recommendations for :doc:`secure installs `. Pip is available by default in most Python installations through the standard library package :doc:`ensurepip `. From 330d33ccfa7277900da6f76f1ba16d78b8196d99 Mon Sep 17 00:00:00 2001 From: Jean Abou Samra Date: Tue, 30 Jan 2024 11:43:51 +0100 Subject: [PATCH 17/18] Present tox/nox along workflow tools --- source/guides/tool-recommendations.rst | 29 +++++++++++--------------- 1 file changed, 12 insertions(+), 17 deletions(-) diff --git a/source/guides/tool-recommendations.rst b/source/guides/tool-recommendations.rst index 9ba538b9d..fb394deca 100644 --- a/source/guides/tool-recommendations.rst +++ b/source/guides/tool-recommendations.rst @@ -144,29 +144,24 @@ The other available method is to upload the package manually using :ref:`twine`. :ref:`deprecated `, it is insecure. -Task runners -============ - -These tools allow you to define and invoke "tasks", such as running tests, -compiling documentation, regenerating some files, etc. Each task can be executed -in a dedicated virtual environment with the dependencies it requires. The most -common tools for this are :doc:`tox ` and :doc:`nox `. - - -Integrated workflow tools -========================= +Workflow tools +============== -These are tools that combine many features in one command line application, such -as automatically managing virtual environments for a project, building -distributions, uploading to PyPI, or creating and using (tool-specific) lock -files. They often call the tools mentioned above under the hood. In alphabetical -order: +These tools are environment managers that automatically manage virtual +environments for a project. They also act as "task runners", allowing you to +define and invoke tasks such as running tests, compiling documentation, +regenerating some files, etc. Some of them provide shortcuts for building +distributions and uploading to PyPI, and some support lock files for +applications. They often call the tools mentioned above under the hood. In +alphabetical order: - :ref:`Flit`, - :ref:`Hatch`, +- :doc:`nox `, - :ref:`PDM`, - :ref:`Pipenv`, -- :ref:`Poetry`. +- :ref:`Poetry`, +- :doc:`tox `. .. _hatchling: https://pypi.org/project/hatchling/ From 8bd667683c54f24ac99a161db3e206ed960dfc4c Mon Sep 17 00:00:00 2001 From: Jean Abou Samra Date: Thu, 1 Feb 2024 12:11:33 +0100 Subject: [PATCH 18/18] Add reminder about not steering to a particular tool --- source/guides/tool-recommendations.rst | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/source/guides/tool-recommendations.rst b/source/guides/tool-recommendations.rst index fb394deca..3903232b9 100644 --- a/source/guides/tool-recommendations.rst +++ b/source/guides/tool-recommendations.rst @@ -62,6 +62,12 @@ environment, for reproducibility purposes. Build backends ============== +.. important:: + + Please, remember: this document does not seek to steer the reader towards + a particular tool, only to enumerate common tools. Different use cases often + need specialized workflows. + Popular :term:`build backends ` for pure-Python packages include, in alphabetical order: