Testing python noarch recipe on several python versions

[beenje]

Your question:

CFEP-25 introduced python_min and decided to use that version for testing:

test:
  requires:
    - python {{ python_min }}

The rationale being:

While noarch: python packages may be incompatible with either newer or older Python versions, we judged that testing against the oldest Python version would catch more issues than the other way around.

With conda-build, one version had to be chosen to run the tests.
With the v1 recipe format, python import can be tested with different python versions (python_version accepts a list).

tests:
  - python:
      python_version:
        - ${{ python_min }}.*
        - 3.*

I started to use this pattern when converting my recipes to the new format.
But that raised some questions when I submitted a new recipe to staged-recipes.

I believe that testing against the latest python is important and can catch some issues:

• the upstream package might be incompatible and has put an upper-bound -> this has to be reflected in the feedstock
• a dependency might not be compatible - could be not actively maintained or maybe not updated yet on conda-forge -> if it can't be updated, we might have to patch the repodata

Doing a search on conda-forge, it seems that several people started using this pattern: https://github.com/search?q=org%3Aconda-forge+%22-+%24%7B%7B+python_min+%7D%7D%22&type=code

What is the general opinion about testing an upper bound as well?

1. Testing on an additional python version isn't a good use of CI time?
2. Should this be let to the feedstock maintainer to decide?
3. Should it be encouraged?

In any case, it would be good to update the documentation with the community agreement.

[ocefpaf]

Pinging @beckermr who the authored CFEP-25 and probably remembers more why we choose only py min.

[h-vetinari]

Testing the lower bound is natural because otherwise it's easy to run into cases where a noarch package is only tested with latest python, but actually dropped support for py_min already (or doesn't have all dependencies available anymore, etc.), and we had plenty of that before CFEP-25.

But testing with latest python can be useful as well, to ensure that current python (with various deprecations, removals etc.) still works.

So, I think testing with py_min and unpinned python can make sense in some cases, but testing all python versions in between would be gratuitous for a noarch package.

I don't think testing with the upper bound makes sense for every noarch feedstock (most pure-python packages are so stable that this doesn't come up), but I think it's fair enough for feedstock maintainers to add extra tests where justified.

PS. Generally I wouldn't discourage additional testing, as long as covers a realistic failure mode. The human cost of chasing down failures, debugging etc. (to say nothing about extra infra strain for rebuilds and repopatches) IMO outweighs the few cycles for extra tests.

[bollwyvl]

Yep: I think with the acceleration of dead-battery removal, this is a bit more relevant, and with the ferocious solve/install speed of rattler-build, the maintainer cost is pretty minimal.

I've been using an explicit minor version (e.g. 3.13.*), just so I have some idea what is changing. Especially older packages can make assumptions about things like pip and setuptools (and their deps) being around. When the bit flips and 3.14 is no longer label-gated, 3.* will be a bit exciting, as many upstream authors seem to wait until they get reports from the field about broken things

What I have also found is testing even just a solve/import against a "new, but not too new python" reveals less-maintained dependency feedstocks that have dangling python migrations, which can sometimes lead to good overall outcomes like making more packages noarch: python (especially as special cases roll off with python_min), adding abi3, or upgrading to v1 recipes.

[beckermr]

Testing against the latest and oldest python is fine if folks want. More than that seems like a waste of CI time.

You can use a v1 recipe to make this easy to implement.

[ocefpaf]

We should formalize that in the docs IMO. Something along the lines that for V1 recipes only we should do min and max.