Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Include tests in source distributions #183

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

sbraz
Copy link

@sbraz sbraz commented Aug 21, 2018

Hi,
We at Gentoo rely on PyPI tarballs to build docs and run tests, could you include them in future releases so that we don't have to use GitHub's tarballs?

@tartley
Copy link
Owner

tartley commented Oct 13, 2020

Hey. FYI, yesterday I created a PR to test releases before we push them to PyPI. When that is merged, I'll be more confident about resuming merges and releases. I'll try to look at this PR soon. Thank you for creating it!

@sbraz
Copy link
Author

sbraz commented Oct 13, 2020

Hi @tartley. If your PR works, it means tests are included so my PR would not be needed :)

@tartley
Copy link
Owner

tartley commented Oct 13, 2020

Understood, thanks for the clarification. I'll double check when I'm not meant to be doing my real job... :-)

@tartley
Copy link
Owner

tartley commented Oct 7, 2021

Apologies for being AWOL forever. Interesting to hear that you want this feature. It has been my instinct not to include tests. I shall sleep on this and reconsider. Be aware, there are changes afoot that may affect this, see #293 .

@sbraz
Copy link
Author

sbraz commented Oct 7, 2021

No worries, it's not a massive problem. We (Gentoo) simply use GitHub releases instead of PyPI packages for now. Howevern, it is good practice to include tests in sdists if they aren't too large. I know Debian also relies on this to test packages.

@tartley
Copy link
Owner

tartley commented Oct 7, 2021

Thank you for the response. I confess I do not yet understand why it's good to bundle tests in sdists. If such a test fails for you, but passes for me (the only scenario when it is useful to have bundled them) then that (to my way of thinking) indicates that it's a poor test, i.e. depends on something contextual that isn't captured in the repo, breaking all the invariants I thought we relied upon - eg. to see a failing test start passing by editing something that is captured in the repo.

I guess maybe when the test fails for you, it's a sign that whatever made it fail possibly ought to be something that is captured / controlled by / compensated for by something that ought to be in the repo, but isn't yet. (or it's a bad test)

I'll try to think harder about how I'm wrong about this, see if I can read around it or steelman myself into understanding it. Thanks for raising the issue.

@sbraz
Copy link
Author

sbraz commented Oct 7, 2021

If such a test fails for you, but passes for me (the only scenario when it is useful to have bundled them) then that (to my way of thinking) indicates that it's a poor test, i.e. depends on something contextual that isn't captured in the repo.

There are many reasons for failing tests downstream. Here are a few I can think of:

  1. missing dependencies (maybe not declared where we expected them to be or missed by us)
  2. wrong version for some dependencies (we don't usually pin versions like upstreams do because it makes dependency handling harder for the whole system)
  3. of course actual bugs in packages

The main purpose of tests for us are to make sure that we aren't shipping a broken package for some reason (points 1 and 2).

But of course, we often find actual bugs that you might not have noticed (because you're not running the same Python version, because of different non-Python dependencies, etc.). All this makes the test suites very valuable to us (and indirectly to you because we can help you find bugs 😃).

@tartley
Copy link
Owner

tartley commented Oct 8, 2021

OK then. Thank you for expanding on the idea. I'll continue to educate myself about it and see whether we can incorporate it here.

@tartley tartley added the 0.4.5 Get merged for 0.4.5 release label Oct 8, 2021
@tartley tartley added 0.4.6 Get merged for 0.4.6. release and removed 0.4.5 Get merged for 0.4.5 release labels Jun 14, 2022
@mtelka
Copy link

mtelka commented Nov 14, 2022

+1 for including tests in sdist (including the tox.ini file). OpenIndiana have similar workflow like Gentoo (and others mentioned above) so tests in sdist would make the world better. Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
0.4.6 Get merged for 0.4.6. release requires-thought
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants