Support specifying project.optional-dependencies extras on the python_requirements target.

[AlirezaRoshanzamir]

Problem

In Pants 2.16, there is a requirement to specify a lock file for each tool. Currently, this specification is not mandatory, but the requirements field for each tool does not support additional dependencies, such as flake8-annotations for flake8 or pyenchant for pylint. This limitation exists because of Pants' internal lock files. Additionally, Pants enforces the use of Pants-generated lock files for the tools. While Poetry and Pipenv support grouping requirements, they strangely do not support generating multiple lock files for each group. As a result, when working with our project's repository, we end up with multiple requirements files and corresponding lock files, totaling around 20 files. We decided to address this by placing all the requirements in the pyproject.toml file using the PEP 621 format and organizing each tool's requirements in a separate group inside the project.optional-dependencies section. However, since there are different lock files for each group, we cannot specify this in the python_requirements target.

Proposed solution

To address this issue, we propose enhancing the python_requirements target to allow the selection of a subset of the pyproject.toml file. Currently, Pants understands optional dependencies and generates a tag for each one. Our suggestion is to add the capability to specify a specific part of the pyproject.toml file in the python_requirements target:

python_requirements(
    name="lint",
    source="pyproject.toml",
    extra="lint",
    resolve="lint",
)

However, we acknowledge that this idea conflicts with the installation of requirements using Pip and pyproject.toml. According to PEP 621 and similar to the extra_requires concept in the setuptools literature, these optional dependencies should be installed in addition to the main requirements, rather than being separate entities. Perhaps we should consider avoiding the use of the extra keyword and instead find a more generic approach to solve this issue:

python_requirements(
    name="lint",
    source="pyproject.toml",
    variable="[project.optional-dependencies].lint",
    resolve="lint",
)

Alternative approaches attempted

We have explored alternative lock file generators such as Poetry and Pipenv. While both tools support grouping requirements, they do not support the generation of multiple lock files. Furthermore, Poetry enforces that all requirement groups should be compatible with each other, even if they are optional. These limitations have been discussed in the following GitHub issues: link 1, link 2, and link 3. In addition to the aforementioned problems, Pants mandates the use of Pants-generated lock files for the tools, further complicating the situation.

Despite our attempts, using a single requirements.txt and requirements.lock file led to conflicts with the importlib-metadata library when incorporating new versions of flake8 and sphinx.

Finally, I would greatly appreciate any alternative solutions that can help avoid the proliferation of multiple requirements files and lock files.