Speed up unit tests by running ctest in parallel

[chongchonghe]

Description

Speed up CPU unit tests by running ctest on parallel, e.g. ctest -j4.

Related issues

#783

Checklist

Before this pull request can be reviewed, all of these tasks should be completed. Denote completed tasks with an x inside the square brackets [ ] in the Markdown source below:

• I have added a description (see above).
• I have added a link to any related issues see (see above).
• I have read the Contributing Guide.
• I have added tests for any new physics that this PR adds to the code.
• I have tested this PR on my local computer and all tests pass.
• I have manually triggered the GPU tests with the magic comment /azp run.
• I have requested a reviewer for this PR.

[BenWibking]

Did this actually make it faster than before? It looks like it still takes about the same amount of time, based on the timings I remember.

[chongchonghe]

This indeed makes the running process faster for the three checks I changed. There could be other checks that take the majority of the time but aren't optimized.

I would also try to increase the number of parallelization in the build process to as high as possible. Currently most of them are 4.

[BenWibking]

This indeed makes the running process faster for the three checks I changed. There could be other checks that take the majority of the time but aren't optimized.

I would also try to increase the number of parallelization in the build process to as high as possible. Currently most of them are 4.

How much faster does it make it?

Also, I don't think all of the virtual machines have 4 cores available. It might be good to check here: https://docs.github.com/en/actions/using-github-hosted-runners/using-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories and/or empirically test whether 2, 3 or 4 simultaneous tests is fastest.

[BenWibking]

Another thing to test would be whether it's faster to run multiple test in parallel, or to just turn on MPI for the tests, so each test is itself parallelized. You can do that by adding CMAKE_TEST_LAUNCHER to the MPI flags:

cmake .. -DCMAKE_TEST_LAUNCHER=mpirun

[BenWibking]

Although as long as we have documented somewhere how much faster this is than before, we can search for the optimal solution later.

[chongchonghe]

Also, I don't think all of the virtual machines have 4 cores available. It might be good to check here: https://docs.github.com/en/actions/using-github-hosted-runners/using-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories and/or empirically test whether 2, 3 or 4 simultaneous tests is fastest.

So, Windows and Linux has 4 cores and macOS has 3 cores. I've changed the parallelisation of the macOS tests from 2 to 3.

[sonarcloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[chongchonghe]

How much faster does it make it?

I can do a benchmark, but it seems obvious to me that running ctest in parallel is faster than on a single core. I tested it on my own Mac Studio. It scales almost linear with the number of processes up to 8 cores.

Another thing to test would be whether it's faster to run multiple test in parallel, or to just turn on MPI for the tests, so each test is itself parallelized. You can do that by adding CMAKE_TEST_LAUNCHER to the MPI flags:

If you have many (tens of) tests, which is the case here, since all the tests are independent, cmake -j4 should always be faster than MPI due to lower overhead.

[chongchonghe]

For the macOS test with -j3, the speedup is 89.91 sec vs 179.89 sec. For Intel with -j4, the speedup is 194.02 sec vs 407.64 sec. So, both are about 2 times faster.

[BenWibking]

How much faster does it make it?

I can do a benchmark, but it seems obvious to me that running ctest in parallel is faster than on a single core. I tested it on my own Mac Studio. It scales almost linear with the number of processes up to 8 cores.

You have to measure, not conjecture. GitHub actions runs on virtual machines that have multiple VMs running on the same machine. It is not necessarily the case that it will scale in the VM the same way as on your own machine.

Another thing to test would be whether it's faster to run multiple test in parallel, or to just turn on MPI for the tests, so each test is itself parallelized. You can do that by adding CMAKE_TEST_LAUNCHER to the MPI flags:

If you have many (tens of) tests, which is the case here, since all the tests are independent, cmake -j4 should always be faster than MPI due to lower overhead.

Again, you must put on your engineering hat and measure the outcome, rather than conjecture as a theorist. It is not necessarily the case that running independent simulations will always be faster. It is possible that cache contention effects will make it slower.