diff --git a/source/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows.rst b/source/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows.rst index ca75d159b..99ac3a9e2 100644 --- a/source/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows.rst +++ b/source/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows.rst @@ -1,3 +1,5 @@ +.. _trusted-publishing: + ============================================================================= Publishing package distribution releases using GitHub Actions CI/CD workflows ============================================================================= diff --git a/source/guides/tool-recommendations.rst b/source/guides/tool-recommendations.rst index 1a59201fa..3903232b9 100644 --- a/source/guides/tool-recommendations.rst +++ b/source/guides/tool-recommendations.rst @@ -4,111 +4,172 @@ Tool recommendations ==================== -If you're familiar with Python packaging and installation, and just want to know -what tools are currently recommended, then here it is. +The Python packaging landscape consists of many different tools. For many tasks, +the :term:`Python Packaging Authority ` +(PyPA, the working group which encompasses many packaging tools and +maintains this guide) purposefully does not make a blanket recommendation; for +example, the reason there are many build backends is that the landscape was +opened up in order to enable the development of new backends serving certain users' +needs better than the previously unique backend, setuptools. This guide does +point to some tools that are widely recognized, and also makes some +recommendations of tools that you should *not* use because they are deprecated +or insecure. + + +Virtual environments +==================== + +The standard tools to create and use virtual environments manually are +:ref:`virtualenv` (PyPA project) and :doc:`venv ` (part of +the Python standard library, though missing some features of virtualenv). + + +Installing packages +=================== + +:ref:`Pip` is the standard tool to install packages from :term:`PyPI `. You may want to read pip's recommendations for +:doc:`secure installs `. Pip is available by default +in most Python installations through the standard library package +:doc:`ensurepip `. + +Alternatively, consider :ref:`pipx` for the specific use case of installing Python +applications that are distributed through PyPI and run from the command line. +Pipx is a wrapper around pip and venv that installs each +application into a dedicated virtual environment. This avoids conflicts between +the dependencies of different applications, and also with system-wide applications +making use of the same Python interpreter (especially on Linux). + +For scientific software specifically, consider :ref:`Conda` or :ref:`Spack`. + +.. todo:: Write a "pip vs. Conda" comparison, here or in a new discussion. + +Do **not** use ``easy_install`` (part of :ref:`setuptools`), which is deprecated +in favor of pip (see :ref:`pip vs easy_install` for details). Likewise, do +**not** use ``python setup.py install`` or ``python setup.py develop``, which +are also deprecated (see :ref:`setup-py-deprecated` for background and +:ref:`modernize-setup-py-project` for migration advice). + + +Lock files +========== + +:ref:`pip-tools` and :ref:`Pipenv` are two recognized tools to create lock +files, which contain the exact versions of all packages installed into an +environment, for reproducibility purposes. + + +Build backends +============== + +.. important:: + Please, remember: this document does not seek to steer the reader towards + a particular tool, only to enumerate common tools. Different use cases often + need specialized workflows. -Application dependency management -================================= +Popular :term:`build backends ` for pure-Python packages include, +in alphabetical order: -* Use :ref:`pip` in a `secure manner`_ to install a Python application and its - dependencies during deployment. +- :doc:`Flit-core ` -- developed with but separate from :ref:`Flit`. + A minimal and opinionated build backend. It does not support plugins. -* Use :ref:`virtualenv` or :doc:`venv ` to isolate - application-specific dependencies from a shared Python installation. [4]_ +- Hatchling_ -- developed with but separate from :ref:`Hatch`. Supports plugins. -* Use `pip-tools`_, :ref:`pipenv`, or `poetry`_ to generate the fully-specified - application-specific dependencies, when developing Python applications. +- PDM-backend_ -- developed with but separate from :ref:`PDM`. Supports plugins. -.. _secure manner: https://pip.pypa.io/en/latest/topics/secure-installs/ -.. _pip-tools: https://github.com/jazzband/pip-tools -.. _Poetry: https://python-poetry.org/ +- Poetry-core_ -- developed with but separate from :ref:`Poetry`. Supports + plugins. -Installation tool recommendations -================================= + Unlike other backends on this list, Poetry-core does not support the standard + :ref:`[project] table ` (it uses a different format, + in the ``[tool.poetry]`` table). -* Use :ref:`pip` to install Python :term:`packages ` from - :term:`PyPI `. [1]_ [2]_ Depending on how :ref:`pip` - is installed, you may need to also install :ref:`wheel` to get the benefit - of wheel caching. [3]_ +- :ref:`setuptools`, which used to be the only build backend. Supports plugins. -* Use :ref:`virtualenv` or :doc:`venv ` to isolate - project-specific dependencies from a shared Python installation. [4]_ + .. caution:: -* If you're looking for management of fully integrated cross-platform software - stacks, consider: + If you use setuptools, please be aware that some features that predate + standardisation efforts are now deprecated and only *temporarily kept* + for compatibility. - * :ref:`buildout`: primarily focused on the web development community + In particular, do **not** use direct ``python setup.py`` invocations. On the + other hand, configuring setuptools with a :file:`setup.py` file is still fully + supported, although it is recommended to use the modern :ref:`[project] table + in pyproject.toml ` (or :file:`setup.cfg`) whenever possible and keep + :file:`setup.py` only if programmatic configuration is needed. See + :ref:`setup-py-deprecated`. - * :ref:`spack`, :ref:`hashdist`, or :ref:`conda`: primarily focused - on the scientific community. + Other examples of deprecated features you should **not** use include the + ``setup_requires`` argument to ``setup()`` (use the :ref:`[build-system] table + ` in :file:`pyproject.toml` instead), and + the ``easy_install`` command (cf. :ref:`pip vs easy_install`). +Do **not** use :ref:`distutils`, which is deprecated, and has been removed from +the standard library in Python 3.12, although it still remains available from +setuptools. -Packaging tool recommendations -============================== +For packages with :term:`extension modules `, it is best to use +a build system with dedicated support for the language the extension is written in, +for example: -* Use :ref:`setuptools` to define projects. [5]_ [6]_ +- :ref:`setuptools` -- natively supports C and C++ (with third-party plugins for Go and Rust), +- :ref:`meson-python` -- C, C++, Fortran, Rust, and other languages supported by Meson, +- :ref:`scikit-build-core` -- C, C++, Fortran, and other languages supported by CMake, +- :ref:`maturin` -- Rust, via Cargo. -* Use :ref:`build` to create :term:`Source Distributions - ` and :term:`wheels `. -If you have binary extensions and want to distribute wheels for multiple -platforms, use :ref:`cibuildwheel` as part of your CI setup to build -distributable wheels. +Building distributions +====================== -* Use `twine `_ for uploading distributions - to :term:`PyPI `. +The standard tool to build :term:`source distributions ` and :term:`wheels ` for uploading to PyPI is :ref:`build`. It +will invoke whichever build backend you :ref:`declared +` in :file:`pyproject.toml`. +Do **not** use ``python setup.py sdist`` and ``python setup.py bdist_wheel`` for +this task. All direct invocations of :file:`setup.py` are :ref:`deprecated +`. -Publishing platform migration -============================= +If you have :term:`extension modules ` and want to distribute +wheels for multiple platforms, use :ref:`cibuildwheel` as part of your CI setup +to build distributable wheels. -The original Python Package Index implementation (previously hosted at -`pypi.python.org `_) has been phased out in favour -of an updated implementation hosted at `pypi.org `_. -See :ref:`Migrating to PyPI.org` for more information on the status of the -migration, and what settings to change in your clients. +Uploading to PyPI +================= ----- +For projects hosted on GitHub, it is recommended to use the :ref:`trusted publishing +`, which allows the package to be securely uploaded to PyPI +from a GitHub Actions job. (This is not yet supported on software forges other +than GitHub.) -.. [1] There are some cases where you might choose to use ``easy_install`` (from - :ref:`setuptools`), e.g. if you need to install from :term:`Eggs ` - (which pip doesn't support). For a detailed breakdown, see :ref:`pip vs - easy_install`. +The other available method is to upload the package manually using :ref:`twine`. -.. [2] The acceptance of :pep:`453` means that :ref:`pip` - will be available by default in most installations of Python 3.4 or - later. See the :pep:`rationale section <453#rationale>` from :pep:`453` - as for why pip was chosen. +**Never** use ``python setup.py upload`` for this task. In addition to being +:ref:`deprecated `, it is insecure. -.. [3] `get-pip.py `_ and - :ref:`virtualenv` install - :ref:`wheel`, whereas :ref:`ensurepip` and :ref:`venv ` do not - currently. Also, the common "python-pip" package that's found in various - linux distros, does not depend on "python-wheel" currently. -.. [4] Beginning with Python 3.4, ``venv`` will create virtualenv environments - with ``pip`` installed, thereby making it an equal alternative to - :ref:`virtualenv`. However, using :ref:`virtualenv` will still be - recommended for users that need cross-version consistency. +Workflow tools +============== -.. [5] Although you can use pure :ref:`distutils` for many projects, it does not - support defining dependencies on other projects and is missing several - convenience utilities for automatically populating distribution metadata - correctly that are provided by ``setuptools``. Being outside the - standard library, ``setuptools`` also offers a more consistent feature - set across different versions of Python, and (unlike ``distutils``), - recent versions of ``setuptools`` support all of the modern metadata - fields described in :ref:`core-metadata`. +These tools are environment managers that automatically manage virtual +environments for a project. They also act as "task runners", allowing you to +define and invoke tasks such as running tests, compiling documentation, +regenerating some files, etc. Some of them provide shortcuts for building +distributions and uploading to PyPI, and some support lock files for +applications. They often call the tools mentioned above under the hood. In +alphabetical order: - Even for projects that do choose to use ``distutils``, when :ref:`pip` - installs such projects directly from source (rather than installing - from a prebuilt :term:`wheel ` file), it will actually build - your project using :ref:`setuptools` instead. +- :ref:`Flit`, +- :ref:`Hatch`, +- :doc:`nox `, +- :ref:`PDM`, +- :ref:`Pipenv`, +- :ref:`Poetry`, +- :doc:`tox `. -.. [6] `distribute`_ (a fork of setuptools) was merged back into - :ref:`setuptools` in June 2013, thereby making setuptools the default - choice for packaging. -.. _distribute: https://pypi.org/project/distribute +.. _hatchling: https://pypi.org/project/hatchling/ +.. _pdm-backend: https://backend.pdm-project.org +.. _poetry-core: https://pypi.org/project/poetry-core/ diff --git a/source/guides/writing-pyproject-toml.rst b/source/guides/writing-pyproject-toml.rst index 187a44d50..938ebbb93 100644 --- a/source/guides/writing-pyproject-toml.rst +++ b/source/guides/writing-pyproject-toml.rst @@ -40,6 +40,7 @@ three possible TOML tables in this file. :ref:`setup-py-deprecated`. +.. _pyproject-guide-build-system-table: Declaring the build backend ===========================