rft: wrap all tests in @testset

[haakon-e]

Purpose

This pull request wraps the entire test suite in @testset. This ensures that even if one @testset fails, the rest of the testsuite attempts to run, so that you get an overview of all issues with a proposed change from running the tests just once.

• There might be specific cases where this is undesirable. For those cases, we can add the option failfast=true to the @testset (ref) which has the effect:
  • failfast: if true, any test failure or error will cause the testset and any child testsets to return immediately (the default is false).

Additionally:

• all tests now consistently import Test as TT (instead of e.g. using Test)
• any @testset that loops over options, e.g. for FT in (Float64, Float32), does so with a consistent pattern that is supported by Test, e.g.:

TT.@testset "Aerosol Activation Tests ($FT)" for FT in (Float64, Float32)
    test_aerosol_activation(FT)
end

• these kinds of @testset loops is an example of a nice place to add failfast=true... presumably if a test fails on Float64, it does so too on Float32.
• add artifact_calling_tests.jl (was missing)

To compare how a successful test suite would look like, compare this PR's buildkite (link) to e.g. this PR's buildkite (link)
If any tests fail, the Test Summary will expand to show much more detail about the failing test(s). In addition, the stack trace of each failing test is printed, of course:

click to see example of failing test

# [...]

Melting Smoke Test: Test Failed at /Users/haakon/Documents/CliMA/clima-codes/CloudMicrophysics.jl/test/p3_tests.jl:542
  Expression: false

Stacktrace:
 [1] macro expansion
   @ ~/.julia/juliaup/julia-1.11.5+0.aarch64.apple.darwin14/share/julia/stdlib/v1.11/Test/src/Test.jl:679 [inlined]
 [2] macro expansion
   @ ~/Documents/CliMA/clima-codes/CloudMicrophysics.jl/test/p3_tests.jl:542 [inlined]
 [3] macro expansion
   @ ~/.julia/juliaup/julia-1.11.5+0.aarch64.apple.darwin14/share/julia/stdlib/v1.11/Test/src/Test.jl:1704 [inlined]
 [4] test_p3_melting(FT::Type{Float64})
   @ Main ~/Documents/CliMA/clima-codes/CloudMicrophysics.jl/test/p3_tests.jl:514
Melting Smoke Test: Test Failed at /Users/haakon/Documents/CliMA/clima-codes/CloudMicrophysics.jl/test/p3_tests.jl:542
  Expression: false

Stacktrace:
 [1] macro expansion
   @ ~/.julia/juliaup/julia-1.11.5+0.aarch64.apple.darwin14/share/julia/stdlib/v1.11/Test/src/Test.jl:679 [inlined]
 [2] macro expansion
   @ ~/Documents/CliMA/clima-codes/CloudMicrophysics.jl/test/p3_tests.jl:542 [inlined]
 [3] macro expansion
   @ ~/.julia/juliaup/julia-1.11.5+0.aarch64.apple.darwin14/share/julia/stdlib/v1.11/Test/src/Test.jl:1704 [inlined]
 [4] test_p3_melting(FT::Type{Float32})
   @ Main ~/Documents/CliMA/clima-codes/CloudMicrophysics.jl/test/p3_tests.jl:514

# [...]

Test Summary:                                                       | Pass  Fail  Broken  Total     Time
All tests                                                           | 3159     2      28   3189  3m34.6s
  Aerosol Activation Tests (Float64)                                |   44                   44     1.5s
  Aerosol Activation Tests (Float32)                                |   44                   44     1.5s
  Heterogeneous Ice Nucleation Tests (Float64)                      |   37                   37     0.7s
  Heterogeneous Ice Nucleation Tests (Float32)                      |   37                   37     0.8s
  Homogeneous Ice Nucleation Tests (Float64)                        |    4                    4     0.1s
  Homogeneous Ice Nucleation Tests (Float32)                        |    4                    4     0.1s
  Microphysics 0M Tests (Float64)                                   |   16                   16     0.1s
  Microphysics 0M Tests (Float32)                                   |   16                   16     0.1s
  Microphysics 1M Tests (Float64)                                   |   63                   63     1.4s
  Microphysics 1M Tests (Float32)                                   |   63                   63     1.4s
  Microphysics 2M Tests (Float64)                                   |  241                  241     7.7s
  Microphysics 2M Tests (Float32)                                   |  241                  241     8.1s
  Microphysics Non-Equilibrium Tests (Float64)                      |   27                   27     0.8s
  Microphysics Non-Equilibrium Tests (Float32)                      |   27                   27     0.7s
  Cloud Diagnostics Tests (Float64)                                 |   26                   26     0.3s
  Cloud Diagnostics Tests (Float32)                                 |   26                   26     0.3s
  logistic_function unit tests                                      |    8                    8     0.0s
  logistic_function_integral unit tests                             |    8                    8     0.0s
  Common Functions Tests (Float64)                                  |   22                   22     0.5s
  Common Functions Tests (Float32)                                  |   22                   22     0.5s
  Broadcasting over common types tests (Float64)                    |                         0     0.0s
  Broadcasting over common types tests (Float32)                    |                         0     0.0s
  Nucleation Unit Tests                                             |   24                   24     0.0s
  precipitation_susceptibility_SB2006                               |    5                    5     0.0s
  P3 tests (Float64)                                                |  828     1      11    840     4.0s
    P3State Creation and Properties                                 |    9                    9     0.0s
    Thresholds - exact solution                                     |   29             1     30     0.0s
    Thresholds - mass, area, density, aspect ratio                  |   18                   18     0.0s
    Shape parameters - nonlinear solver                             |  296                  296     0.0s
    Shape parameters - nonlinear solver                             |  296                  296     0.0s
    Numerical integrals sanity checks for N, velocity and diameter  |   80                   80     1.3s
    Smoke tests for cloud/rain particle terminal vel from Chen 2022 |   10                   10     0.0s
    Smoke tests for ice particle terminal vel from Chen 2022        |   20                   20     0.0s
    Smoke tests for mixed phase particle terminal velocity          |   10            10     20     0.0s
    Mass and number weighted terminal velocities                    |   24                   24     0.6s
    Mass-weighted mean diameters                                    |    4                    4     0.0s
    Heterogeneous Freezing Smoke Test                               |   24                   24     0.0s
    Melting Smoke Test                                              |    8     1              9     0.2s
  P3 tests (Float32)                                                |  828     1      11    840     3.8s
    P3State Creation and Properties                                 |    9                    9     0.0s
    Thresholds - exact solution                                     |   29             1     30     0.0s
    Thresholds - mass, area, density, aspect ratio                  |   18                   18     0.0s
    Shape parameters - nonlinear solver                             |  296                  296     0.0s
    Shape parameters - nonlinear solver                             |  296                  296     0.0s
    Numerical integrals sanity checks for N, velocity and diameter  |   80                   80     1.4s
    Smoke tests for cloud/rain particle terminal vel from Chen 2022 |   10                   10     0.0s
    Smoke tests for ice particle terminal vel from Chen 2022        |   20                   20     0.0s
    Smoke tests for mixed phase particle terminal velocity          |   10            10     20     0.0s
    Mass and number weighted terminal velocities                    |   24                   24     0.2s
    Mass-weighted mean diameters                                    |    4                    4     0.0s
    Heterogeneous Freezing Smoke Test                               |   24                   24     0.0s
    Melting Smoke Test                                              |    8     1              9     0.2s
  Quality assurance - Aqua                                          |   10                   10     1.1s
  Performance Tests (Float64)                                       |  225                  225    49.8s
  Performance Tests (Float32)                                       |  225                  225    48.5s
  Aerosol activation test: Calibrated ARG                           |    7                    7  1m06.0s
  Ice Nucleation perfect model calibration on ABDINM                |                  2      2     4.2s
  Ice Nucleation perfect model calibration on ABIFM                 |                  2      2     2.9s
  Ice Nucleation perfect model calibration on ABHOM                 |                  2      2     5.0s
  Ventilation Tests (Float64)                                       |    5                    5     0.0s
  Ventilation Tests (Float32)                                       |    5                    5     0.1s
  DistributionTools                                                 |   21                   21     0.0s

To-do

Content

---

• I have read and checked the items on the review checklist.

[haakon-e]

This change is part of the following stack:

• rft: wrap all tests in @testset #595 ◀
  • rft: clean up calibration test #597

_{Change managed by git-spice.}

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 92.75%. Comparing base (3b4d142) to head (aecc284).
Report is 3 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #595   +/-   ##
=======================================
  Coverage   92.75%   92.75%           
=======================================
  Files          47       47           
  Lines        1752     1752           
=======================================
  Hits         1625     1625           
  Misses        127      127           

Components	Coverage Δ
src	94.08% <ø> (ø)
ext	69.79% <ø> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[sajjadazimi]

LGTM, thanks!