Evaluate Python MPI testing strategy

[heplesser]

We currently perform MPI-based tests under Python in two different ways

• running mpirun -np N python -m pytest ... on the pytest/mpi/{2,3,4} directories (direct approach)
• running PyNEST indirectly from pytest as in the tests in sli2py_mpi (indirect approach)

Direct approach

• In some of these tests, ranks test assertions independently of each other, in other tests they use mpi4py to collect and test results.
• The approach is not fully robust, in some cases we have observed that output written by junitxml resulted in corrupt XML files, likely because all ranks write and might interfere
• For some tests (eg test_issue_3108.py), the number of threads used can become large an execution slow. Correct errors in slicing of node collections #3132 provides rough filtering for this, but more fine-grained filtering (e.g. skip tests with more than 6 threads in total) could be useful.

Indirect approach

• Here, the test script first creates a temporary runner file with some restrictions on test content and structure, then executes the test via mpirun -np N python test_... and collects results from files afterwards.
• This supports the logic of SLI-based MPI tests which execute a fixed number of VPs with different Rank-Thread splits and check for consistent results.

We should see if both approaches can be integrated, simplified and made more robust. Code from the following two projects may be useful to consider:

• https://github.com/NOhs/pytest-easyMPI
• https://github.com/minrk/pytest-mpiexec/tree/main

For selective test skipping/marking, material from the following page may be useful

• https://stackoverflow.com/questions/52785961/pytest-select-tests-based-on-mark-parameterize-value

[github-actions]

Issue automatically marked stale!