Entry [tool.setuptools.packages.find] in pyproject.toml is ignored

[mscheltienne]

As described here: pypa/build#481 I'm trying to build a package without the test files using setuptools and build. My pyproject.toml starts with:

[build-system]
requires = ['setuptools >= 61.0.0']
build-backend = 'setuptools.build_meta'

Followed by the tool configuration for setuptools:

[tool.setuptools]
include-package-data = true

[tool.setuptools.packages.find]
include = ['template*']
exclude = ['template*tests']

Yet, the tests modules are still present in the created wheel. I thought maybe this was due to include-package-data = true, as mentioned here #3260 but removing it does not solve the problem and the build still includes the tests modules. What am I doing wrong?

Also on a sidenote, does the setuptools >= 61.0.0' req. make sense for a project using pyproject.toml?

[abravalheri]

Hi @mscheltienne thank you very much for reporting this.
There is already an issue that covers this problem in #3260.

Unfortunately, although in theory I know how to solve this problem, I cannot implement a solution straight away since it requires changing behaviour that other people might currently rely on. So we need to go through a deprecation cycle first (#3308).

You mentioned that you tried removing include-package-data = true and it did not work. Have you tried setting include-package-data = false? In pyproject.toml the default value of include-package-data is true...

---

Also on a sidenote, does the setuptools >= 61.0.0' req. make sense for a project using pyproject.toml?

In theory, a project builder like build or an installer like pipwill always download the latest version of the dependencies listed in [build-system] requires when no version is specified.

There might be some edge cases though:

• In the future, build (or other project builder/installer) might decide to start caching build dependencies and perform an optimisation that will accept any version of setuptools that is already in cache.
  • Currently, I don't know about any project builder/installer that does that.
• Someone might decide to build/install your project using python -m build --no-isolation, pip install --no-build-isolation, pip wheel --no-build-isolation
  • In this scenario the builder/installer will attempt to reuse the build dependencies that are already installed in your system or virtual environment...
  • As part of this process they might decide to run a check to see if the correct versions are installed and raise an error if they don't
  • In the case the tool perform the check (I honestly don't know if the existing tools perform this check...), adding a version range will make debugging easier when the build fails because the available setuptools is too old.
• Someone might attempt to run a pip install ... with multiple sdists at once or your project has a build dependency that is only available as an sdist.
  • In this scenario, I don't know how pip (or another installer) will try to build the individual wheels for each package, but there is some chance pip will use a single virtual environment when building all the build dependencies in a single command.

So in the end of the day is a matter of risk assessment... If you believe that the chance of users running in one edge case is low enough, you might simply skip the version constraint.

[abravalheri]

I think this is one of the corner cases that would require MANIFEST.in to differentiate the files between the sdist and the wheel.

I tried the following changes on top of the master branch:

• MANIFEST.in

  - include pycrostates/datasets/lemon/data/PREPROCESSED_LEMON_registry.txt
  + graft pycrostates/**/tests

• pyproject.toml

  + [tool.setuptools.package-data]
  + "*" = ["*.txt"]

This seems to deliver the expected result (i.e. tests present in the sdist but not in the wheel)
It is not necessarily a bug but rather a limitation that would be good if we could solve...

Unfortunately improving this limitation is not something we cannot do immediately, because we need to think about a clean/clear interface. There some suggestions to simply integrate the MANIFEST.in file into pyproject.toml, but I am not a huge fan of that idea. The MANIFEST.in API is difficult to use, difficult to explain and not very obvious...¹. I hope we can find a better way...

---

On a side note, if you don't mind having your CI-related files distributed together in your sdist you might want to give setuptools-scm a try, and ditch the MANIFEST.in file completely. (That is what I do in my projects).

Footnotes

1. Also in terms of bandwidth my current priorities are PEP 660, then PEP 566, PEP 643, etc... ↩

[mscheltienne]

Wait, now I'm getting a bit lost. I am trying to get a build where the tests modules are absent and where the registry.txt file is included. This is not specific to this project, and I often have a couple of 'package-data' files inside the package that I want to distribute. Until now, I would put them in the MANIFEST.in.

The wheel does not contain the tests files and does contain the registry.txt file.

This is what I'm trying to obtain. But I don't understand why the .tar.gz is different from the wheel when the MANIFEST.in is empty and when the configuration in the pyproject.toml is:

[tool.setuptools]
include-package-data = false

[tool.setuptools.packages.find]
include = ['pycrostates*']
exclude = ['pycrostates*tests']

[tool.setuptools.package-data]
"pycrostates.datasets.lemon.data" = ["*.txt"]

With this setup, for some reason that I don't understand, the .tar.gz also includes all the tests modules; except one, the one from the pycrostates.datasets module; while I would expect it to be similar to the .whl, thus not including any tests module.

---

I am not trying to get different .tar.gz and .whl, so the MANIFEST.in with a graft is not useful here. But that is a good example of where a MANIFEST.in is required!

[abravalheri]

Without a MANIFEST.in file both sdist and wheel should have the same "package files" (i.e. the files inside the package directory).

Have you tried running rm -rf dist build *.egg-info to clean the cache?

If I try the following:

rm -rf /tmp/pycrostates
git clone https://github.com/vferat/pycrostates.git /tmp/pycrostates
cd /tmp/pycrostates
rm MANIFEST.in
cat <<EOS > patch
--- c/pyproject.toml
+++ i/pyproject.toml
@@ -93,7 +93,10 @@ source = 'https://github.com/vferat/pycrostates'
 tracker = 'https://github.com/vferat/pycrostates/issues'

 [tool.setuptools]
-include-package-data = true
+include-package-data = false
+
+[tool.setuptools.package-data]
+"pycrostates.datasets.lemon.data" = ["*.txt"]

 [tool.setuptools.packages.find]
 include = ['pycrostates*']
EOS
git apply patch
rm patch
virtualenv -p python3.10 .venv
.venv/bin/python -m pip install -U build
.venv/bin/python -m build
tar tf dist/*.tar.gz | grep 'tests\|registry.txt'
unzip -l dist/*.whl | grep 'tests\|registry.txt'

I can see that both distribution archives:

1. Contain PREPROCESSED_LEMON_registry.txt
2. Don't contain any tests folder

However, if I do (inside the same directory):

rm -rf dist build *.egg-info
echo 'graft pycrostates/**/tests' >> MANIFEST.in
.venv/bin/python -m build
tar tf dist/*.tar.gz | grep 'tests\|registry.txt'
unzip -l dist/*.whl | grep 'tests\|registry.txt'

I can see now that:

1. The sdist contains all the tests folders AND PREPROCESSED_LEMON_registry.txt
2. The wheel contains ONLY PREPROCESSED_LEMON_registry.txt (no tests folder)

[abravalheri]

(Some developers like to distribute the tests in the sdist¹, so I assumed that was your original intention, sorry for the misunderstanding)

Footnotes

1. https://discuss.python.org/t/should-sdists-include-docs-and-tests/14578 ↩

[mscheltienne]

You got it right.. I did not remove the *.egg-info.. I did not think it had an impact. Thanks, now the behavior is as expected.

For the înclusion of the tests, and by extension of a snapshot of the repository when the wheel is built, inside the sdist, those are some very good arguments that are provided, although I always used a curl or git clone command for now. I'll think about it and which road I'll prefer to go on, at least now I have the option and knowledge to do so!

A big big thank you! That discussion was super helpful to me!

[stdedos]

Hi @mscheltienne thank you very much for reporting this. There is already an issue that covers this problem in #3260.

Unfortunately, although in theory I know how to solve this problem, I cannot implement a solution straight away since it requires changing behaviour that other people might currently rely on. So we need to go through a deprecation cycle first (#3308).

You mentioned that you tried removing include-package-data = true and it did not work. Have you tried setting include-package-data = false? In pyproject.toml the default value of include-package-data is true...

Hello there!

In my case:

• I have set include-package-data = false on my pyproject.toml, and I still see:

   /tmp/build-env-xv42xuc3/lib/python3.9/site-packages/setuptools/command/build_py.py:202: SetuptoolsDeprecationWarning:     Installing 'gosqp.osqp.tests.basic_qp' as data is deprecated, please list it in `packages`.
       !!
   
   
       ############################
       # Package would be ignored #
       ############################
       Python recognizes 'gosqp.osqp.tests.basic_qp' as an importable package,
       but it is not listed in the `packages` configuration of setuptools.
   
       'gosqp.osqp.tests.basic_qp' has been automatically added to the distribution only
       because it may contain data files, but this behavior is likely to change
       in future versions of setuptools (and therefore is considered deprecated).
   
       Please make sure that 'gosqp.osqp.tests.basic_qp' is included as a package by using
       the `packages` configuration field or the proper discovery methods
       (for example by using `find_namespace_packages(...)`/`find_namespace:`
       instead of `find_packages(...)`/`find:`).
   
       You can read more about "package discovery" and "data files" on setuptools
       documentation page.
   
   
   !!
   
     check.warn(importable)
  

• Inheriting include-package-data = true still does not actually "include package data", and fails to exclude what I've asked to exclude (with the above message)
  (e.g., at least one *.css file is missing, but packaging is my weak point, and my pyproject.toml is on the https://setuptools.pypa.io/en/latest/userguide/package_discovery.html#finding-namespace-packages level, so YMMV)

I also don't have an easy way to check if "all of the necessary files" are included, unless I have good testing that runs in isolation, and then "some kind of integration/e2e tests" (e.g., for the Dash/Flask framework that I am using) - even if I had an src layout (which, sadly, I don't).

---

On another note,

So we need to go through a deprecation cycle first (#3308).

when does that end? Depreciation notice was at v62.3.0, now we have v67.2.0

[abravalheri]

Hi @stdedos, can you provide a minimal reproducer so I can understand what is going on?

when does that end? Depreciation notice was at v62.3.0, now we have v67.2.0

I would say that we should keep the deprecation warning until it completes at least an year.
Regardless, when the moment comes, I am 100% sure that someone will complain that it breaks their build so we will have to further analyse the case (and decide for a reversion)... It is a catch 22, one part (or the other) of the community will invariably be upset...

[stdedos]

when does that end? Depreciation notice was at v62.3.0, now we have v67.2.0

I would say that we should keep the deprecation warning until it completes at least an year. Regardless, when the moment comes, I am 100% sure that someone will complain that it breaks their build so we will have to further analyse the case (and decide for a reversion)... It is a catch 22, one part (or the other) of the community will invariably be upset...

Apologies, I assumed Semantic Versioning for some reason, hence the "it's 5 majors ahead" 😕

Yeah - it sounds like a catch-22. At the same time I like "update at your own pace" python, and I want to cut cords like this one asap 😓