Skip to content

Commit

Permalink
PEP 735: apply community feedback, including reversion of [project] t…
Browse files Browse the repository at this point in the history
…able changes (#3990)
  • Loading branch information
sirosen authored Oct 1, 2024
1 parent 23530a8 commit 79a41d1
Showing 1 changed file with 97 additions and 135 deletions.
232 changes: 97 additions & 135 deletions peps/pep-0735.rst
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,8 @@ which this PEP seeks to support:

* non-package projects, such as data science projects

Several motivating use cases are defined in detail in the :ref:`Use Cases Appendix <use_cases>`.

Limitations of ``requirements.txt`` files
-----------------------------------------

Expand Down Expand Up @@ -95,8 +97,11 @@ package specifiers which are published as part of a package's metadata, and
which a user can request under that name, as in ``pip install 'foo[bar]'`` to
install ``foo`` with the ``bar`` extra.

Because ``extras`` are package metadata, they are not usable when a project
does not build a distribution (i.e., is not a package).
Because ``extras`` are package metadata, they are not guaranteed to be
statically defined and may require a build system to resolve.
Furthermore, definition of a ``[project.optional-dependencies]`` indicates to
many tools that a project is a package, and may drive tool behaviors such as
validation of the ``[project]`` table.

For projects which are packages, ``extras`` are a common solution for defining
development dependencies, but even under these circumstances they have
Expand Down Expand Up @@ -163,24 +168,6 @@ defined in greater detail in the :ref:`Use Cases Appendix <use_cases>`.
be used as a location for locked dependency data)
* Input data to an environment manager, such as tox, Nox, or Hatch
* Configurable IDE discovery of test and linter requirements
* Exposure of package dependencies for install, without the package itself

Support for Inclusion from ``project.dependencies``
---------------------------------------------------

The ``project.dependencies`` and ``project.optional-dependencies`` tables are
allowed to include Dependency Groups, requiring an update to the specification
of these tables.

The drivers for such changes are that some usages are well solved by the
addition of such support, and that failing to include support in the initial
Dependency Group PEP, but adding such support later in a subsequent PEP,
would make the support landscape significantly more difficult for tool
maintainers.

Inclusion of a Dependency Group in ``project.dependencies`` or
``project.optional-dependencies`` takes the form of a Dependency Group Include,
defined in the specification section below.

Regarding Poetry and PDM Dependency Groups
------------------------------------------
Expand Down Expand Up @@ -255,12 +242,7 @@ Strings in requirement lists must be valid
`Dependency Specifiers <https://packaging.python.org/en/latest/specifications/dependency-specifiers/>`__,
as defined in :pep:`508`.

Tables in requirement lists must be valid Dependency Object Specifiers,
defined below.

The ``project`` table in ``pyproject.toml`` is modified such that
``project.dependencies`` and the values of ``project.optional-dependencies``
may contain Dependency Object Specifiers.
Tables in requirement lists must be valid Dependency Object Specifiers.

Dependency Object Specifiers
----------------------------
Expand Down Expand Up @@ -310,21 +292,6 @@ Includes, in which case those includes should be expanded as well. Dependency
Group Includes MUST NOT include cycles, and tools SHOULD report an error if
they detect a cycle.

``project`` Table Changes
-------------------------

The ``[project]`` table, originally defined in :pep:`621` is extended in two ways.

``dependencies``

In addition to :pep:`508` strings, the array may contain Dependency Object
Specifiers.

``optional-dependencies``

In addition to :pep:`508` strings, the array values in this table may contain
Dependency Object Specifiers.

Example Dependency Groups Table
-------------------------------

Expand Down Expand Up @@ -404,11 +371,10 @@ Tools MAY choose to provide the same interfaces for installing Dependency
Groups as they do for installing extras.

Note that this specification does not forbid having an extra whose name matches
a Dependency Group. In such cases, tools must define their own semantics for
precedence order or disambiguation.
a Dependency Group.

Users are advised to avoid creating Dependency Groups whose names match extras.
Tools SHOULD NOT treat such matching as an error.
Tools MAY treat such matching as an error.

Validation and Compatibility
----------------------------
Expand Down Expand Up @@ -482,7 +448,7 @@ The output is therefore valid ``requirements.txt`` data.
def _resolve_dependency_group(
dependency_groups: dict, group: str, past_groups: tuple[str] = ()
dependency_groups: dict, group: str, past_groups: tuple[str, ...] = ()
) -> list[str]:
if group in past_groups:
raise ValueError(f"Cyclic dependency group include: {group} -> {past_groups}")
Expand Down Expand Up @@ -542,16 +508,15 @@ At time of writing, the ``dependency-groups`` namespace within a
reserved for use only by standards specified at packaging.python.org,
there are no direct backwards compatibility concerns.

However, the introduction of the feature as a potential component of
``project`` data has implications for a number of ecosystem tools.
However, the introduction of the feature has implications for a
number of ecosystem tools, especially those which attempt to support
examination of data in ``setup.py`` and ``requirements.txt``.

Audit and Update Tools
----------------------

A wide range of tools understand Python dependency data as expressed in
``project.dependencies`` and ``project.optional-dependencies`` and may
additionally support ``setup.cfg``, ``requirements.txt``, and even
``setup.py``. (e.g., Dependabot, Tidelift, etc)
``requirements.txt`` files. (e.g., Dependabot, Tidelift, etc)

Such tools inspect dependency data and, in some cases, offer tool-assisted or
fully automated updates.
Expand All @@ -563,31 +528,6 @@ As a result, users of Dependency Groups would experience a degradation in their
workflows and tool support at the time that they start using Dependency Groups.
This is true of any new standard for where and how dependency data are encoded.

Repackaging
-----------

Repackaging, and in particular tool-assisted repackaging such as Grayskull or
PyInstaller, will need to contend with the change to package metadata
definitions as well.

Because repackagers for alternate ecosystems such as conda and linux distros
are often distinct persons vs the package publishers, this compatibility
concern is more difficult to address than cases in which the package
maintainers experience the impact. Package maintainers may be unaware of the
impact of beginning to use Dependency Groups, and may unknowingly make changes
which harm downstream repackaging workflows.

There are two primary ways in which this issue can be addressed:

* via education and "How to Teach This" -- users whose packages are repackaged
should be made aware that using new standards may cause issues for downstream
package consumers

* by driving more behavior through build-backends -- so long as dependency
metadata are gathered via :pep:`517` interfaces, downstream repackagers can
remain ignorant of which build system is being used and whether or not it
supports Dependency Groups

Security Implications
=====================

Expand Down Expand Up @@ -632,22 +572,6 @@ defined dynamically. Requirements loaded from ``requirements.txt`` files and
definitions of static lists prior to ``setup()`` invocation readily analogize
with Dependency Groups.

Notes for Packages which are Repackaged
---------------------------------------

A special note should be given to package maintainers whose packages are
repackaged by linux distros, homebrew, conda, etc.

Use of Dependency Group Includes in core project metadata,
``project.dependencies`` and ``project.optional-dependencies``, may break these
consumers' uses of your package. Because they may be consuming and directly
interact with the source for your repository, their toolchains may not support
Dependency Groups at the same time that the package maintainers' tools are
updated.

Ensure that repackaging consumers can contact you if there is an issue, and
make sure to note transitions to use Dependency Groups in your changelogs.

Interfaces for Use of Dependency Groups
---------------------------------------

Expand Down Expand Up @@ -779,18 +703,92 @@ has impacts on high-level tools like IDEs and Dependabot, which cannot support
deep integration with these Dependency Groups. (For example, at time of writing
Dependabot will not flag dependencies which are pinned in ``tox.ini`` files.)

Deferred Ideas
==============

Why not support Dependency Group Includes in ``[project.dependencies]`` or ``[project.optional-dependencies]``?
---------------------------------------------------------------------------------------------------------------

Earlier drafts of this specification allowed Dependency Group Includes to be
used in the ``[project]`` table.
However, there were several issues raised during community feedback which led
to its removal.

Only a small number of additional use cases would be addressed by the inclusion
of Dependency Groups, and it increased the scope of the specification
significantly. In particular, this inclusion would increase the number of parties
impacted by the addition. Many readers of the ``[project]`` table, including build
backends, SBOM generators, and dependency analyzers are implicated by a change to
``[project]`` but may continue to operate as-is in the presence of a new (but
unconnected) ``[dependency-groups]`` table.

Separately from the above concern, allowing inclusion of dependency groups from the
``[project]`` table encourages package maintainers to move dependency metadata out
of the current standard location.
This complicates static ``pyproject.toml`` metadata and conflicts with the goal of
:pep:`621` to store dependency metadata in a single location.

Finally, exclusion of ``[project]`` support from this PEP is not final. The
use of includes from that table, or an inclusion syntax from
``[dependency-groups]`` into ``[project]``, could be introduced by a future
PEP and considered on its own merits.

Use Cases for Dependency Group Includes From ``[project]``
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

Although deferred in this PEP, allowing includes from the ``[project]``
table would address several use cases.

In particular, there are cases in which package developers would like to
install only the dependencies of a package, without the package itself.

For example:

* Specify different environment variables or options when building dependencies
vs when building the package itself

* Creating layered container images in which the dependencies are isolated from
the package being installed

* Providing the dependencies to analysis environments (e.g., type checking)
without having to build and install the package itself

For an example of the last case, consider the following sample
``pyproject.toml``:

.. code-block:: toml
[project]
dependencies = [{include = "runtime"}]
[optional-dependencies]
foo = [{include = "foo"}]
[dependency-groups]
runtime = ["a", "b"]
foo = ["c", "d"]
typing = ["mypy", {include = "runtime"}, {include = "foo"}]
In this case, a ``typing`` group can be defined, with all of the package's
runtime dependencies, but without the package itself. This allows uses of the
``typing`` Dependency Group to skip installation of the package -- not only is
this more efficient, but it may reduce the requirements for testing systems.

Why not support Dependency Group Includes in ``[build-system.requires]``?
-------------------------------------------------------------------------

Although it may be interesting to allow this in the future, incorporating the
proposal into the build-system table reduces the ability of users to rely on
this to bootstrap support. :pep:`517` frontends would be required to support
Dependency Groups in order to achieve this, and the versions of these frontends
are not easily controlled by packages.
Given that we will not allow for ``[project]`` usage of Dependency Groups,
``[build-system.requires]`` can be considered in comparison with
``[project.dependencies]``.

There are fewer theoretical usages for build requirements specified in a group
than package requirements. Additionally, the impact of such a change implicates
:pep:`517` frontend, which would need to support Dependency Groups in order to
prepare a build environment.

By only defining build backend support, it is possible for packages to start
leveraging the new syntax and capabilities without being concerned about
controlling the environment in which the package is built and installed.
Compared with changes to ``[project.dependencies]`` and
``[project.optional-dependencies]``, changing the behaviors of
``[build-system.requires]`` is higher impact and has fewer potential uses.
Therefore, given that this PEP declines to make changes to the ``[project]``
table, changing ``[build-system]`` is also deferred.

.. _prior_art:

Expand Down Expand Up @@ -1411,42 +1409,6 @@ for various purposes.
This declaration allows the project author's knowledge of the appropriate tools
for the project to be shared with all editors of that project.

Exposure of package dependencies without the package itself
-----------------------------------------------------------

There are a variety of use-cases in which package developers would like to
install only the dependencies of a package, without the package itself.

For example:

* Specify different environment variables or options when building dependencies
vs when building the package itself

* Creating layered container images in which the dependencies are isolated from
the package being installed

* Providing the dependencies to analysis environments (e.g., type checking)
without having to build and install the package itself

For an example of the last case, consider the following sample
``pyproject.toml``:

.. code-block:: toml
[project]
dependencies = [{include = "runtime"}]
[optional-dependencies]
foo = [{include = "foo"}]
[dependency-groups]
runtime = ["a", "b"]
foo = ["c", "d"]
typing = ["mypy", {include = "runtime"}, {include = "foo"}]
In this case, a ``typing`` group can be defined, with all of the package's
runtime dependencies, but without the package itself. This allows uses of the
``typing`` Dependency Group to skip installation of the package -- not only is
this more efficient, but it may reduce the requirements for testing systems.

Copyright
=========

Expand Down

0 comments on commit 79a41d1

Please sign in to comment.