Allow to specify the python, and OS, combinations one wants to test

[gforcada]

tox is known and widely used to simplify the task of running a distribution test suite across different python versions.

With GitHub Actions, one can run those tox tasks in different OSes.

For now, plone/meta has been enforcing to only run on ubuntu-latest with python-3.11 🤷🏾

Being new in tox I'm still not comfortable/capable of pulling this off, I haven't done my research on it 😓 🤓

Thinking outside of Plone core, we would add yet another axis on the matrix: against which Plone version we want the tests to run.

Small steps

A 3-way matrix (OS / python / plone) seems quite complicated, so let's start small, and allow first to specify the python versions one wants to test.

There is a promising plugin for tox that seems to fit the bill: https://pypi.org/project/tox-gh/ with GitLab we should see what we do.

Any takers? 👐🏾 😄

[mauritsvanrees]

I have used a similar tox plugin: https://pypi.org/project/tox-gh-actions/
See usage in the readme here: https://github.com/collective/tox-action

[1letter]

@gforcada @mauritsvanrees

i have a working matrix based on test.yml for projects with mxdev support

https://gist.github.com/1letter/29043a3aaf5249b8fd0aaed3fff31b84

minimal changes in test.yml are needed.

here is the workflow in action: https://github.com/1letter/my.addon or here https://github.com/collective/collective.z3cform.norobots/actions/runs/8993006983 or https://github.com/collective/collective.outputfilters.tinymceaccordion/actions/runs/9007051232

[gforcada]

@1letter nice! 🤩

beware: long rant

I see, though, that the workflow hardcodes the python versions and the plone versions. For that to work for plone core packages (but also generally for add-ons), we should allow that to be configured via .meta.toml/tox.ini, and have the GHA workflow to pick those values somehow.

Though, as a first step, having a hardcoded version would be enough, or actually, we could split the plone/meta/.github/workflows/test.yml to be a per plone version, i.e.

• plone/meta/.github/workflows/test-plone60.yml
• plone/meta/.github/workflows/test-plone61.yml
• plone/meta/.github/workflows/test-plone70.yml

As the supported python versions depend on what Plone upstream supports anyway, so we don't need to define that on a per-package basis.

Though, that would be only for CI, locally one still wants to test against all versions too, or? 🤔 I mean, as soon as we have all versions in CI, testing only one version locally might be enough...

Though (again 😅 ) for debugging purposes, would be nice to have all versions locally as well ✨

Sorry for the rant 🙇🏾

I created a smaller PR that, for now only allows multiple python versions.

[1letter]

@gforcada only for gitlab-ci i use this matrix for tests, all is configured in the .meta.toml:

[gitlab]
jobs = [
  "lint",
  "release-ready",
  "dependencies",
  "circular-dependencies",
  "coverage",
  "plonetests",
]

extra_lines = """
.Test Plone:
  variables:
    PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"
  stage: test
  tags:
    - docker
  cache:
    paths:
      - $PIP_CACHE_DIR
  before_script:
    - pip install --upgrade pip
    - pip install tox
  image: python:$PYTHON_VERSIONS-bookworm
  script:    
    - echo "-c https://dist.plone.org/release/$PLONE_VERSIONS-latest/constraints.txt" > constraints.txt
    - tox -e init
    - tox -e test -- --xml reports

Test Plone 6.0:
  extends: .Test Plone
  parallel:
    matrix:
      - PYTHON_VERSIONS: [ "3.8", "3.9", "3.10", "3.11" ]
        PLONE_VERSIONS: ["6.0"]

Test Plone 6.1:
  extends: .Test Plone
  parallel:
    matrix:
      - PYTHON_VERSIONS: [ "3.10", "3.11", "3.12" ]
        PLONE_VERSIONS: ["6.1"]

"""