Testing

Before making a Pull Request to the ESCMI/CIME repository on GitHub, test your changes.

• Testing changes
  • Testing methods
    • pytest
      • Examples
        • Run all tests
        • Running tests by category
        • Running a specific test file
        • Running a specific test case
    • scripts_regression_tests.py
      • Examples
        • Running all tests
        • Running tests by category
        • Running a specific test file
        • Running a specific test case
• Code Quality
  • Install pre-commit
  • Run pre-commit
  • Installing git hook scripts
• Continuous Intgeration Testing
• Changes to Fortran or to the test scripts in SystemTests

Testing changes

CIME splits it's testing into two categories unit and sys.

The unit category covers both doctests and unit tests. While the sys category covers regression tests. The tests are named accordingly e.g. unit tests are found as CIME/tests/test_unit*.

Testing methods

There are two possible methods to run these tests.

pytest

CIME also supports running tests using pytest. By using pytest coverage reports are automatically generated.

To get started install pytest and pytest-cov.

pip install pytest pytest-cov
# or
# conda install -c conda-forge pytest pytest-cov

Examples

Run all tests

pytest

Running tests by category

pytest CIME/tests/test_unit*
# or
pytest CIME/tests/test_sys*

Running a specific test file

pytest CIME/tests/test_unit_case.py

Running a specific test case

pytest CIME/tests/test_unit_case.py::TestCaseSubmit::test_check_case

scripts_regression_tests.py

The scripts_regression_tests.py script is located under CIME/tests.

You can pass either the module name or the file path of a test.

Examples

Running all tests

python CIME/tests/scripts_regression_tests.py

Running tests by category

python CIME/tests/scripts_regression_tests.py CIME/tests/test_unit*
# or
python CIME/tests/scripts_regression_tests.py CIME/tests/test_sys*

Running a specific test file

python CIME/tests/scripts_regression_tests.py CIME.tests.test_unit_case

Running a specific test case

python CIME/tests/scripts_regression_tests.py CIME.tests.test_unit_case.TestCaseSubmit.test_check_case

Code Quality

To ensure code quality we require all code to be linted by pylint and formatted using black. We run a few other tools to check XML formatting, ending files with newlines and trailing white spaces.

To ensure consistency when running these checks we require the use of pre-commit.

Our GitHub actions will lint and check the format of each PR but will not automatically fix any issues. It's up to the developer to resolve linting and formatting issues. We encourage installing pre-commit's Git hooks that will run these checks before code can be committed.

Install pre-commit

pip install pre_commit
# or
# conda install -c conda-forge pre_commit

Run pre-commit

pre-commit run -a

Installing git hook scripts

If you install these scripts then pre-commit will automatically run on git commit.

pre-commit install

Continuous Intgeration Testing

CIME uses Jenkins to run scripts_regression_tests nightly. Latest results are on http://my.cdash.org/index.php?project=CIME.

Changes to Fortran or to the test scripts in SystemTests

If there are changes to Fortran code in CIME or to the scripts in /utils/python/CIME/SystemTests, you must run the cime_developer tests.

1. Run the X and A system tests with ::

   ./create_test cime_developer

Note: The list of cime_developer tests is defined in cime/scripts/lib/update_acme_tests.py

As noted under "Changes to python", the cime_developer system tests are run automatically if you run scripts_regression_tests.py without the --fast argument - so if you have already run that, you do not have to separately run the above command.

2. If any changes have been made to Fortran code in share/ or driver_cpl/, then run cime unit tests: From the top-level directory in cime, follow the instructions in README.unit_testing (currently only works out-of-the-box on yellowstone).

In addition to CIME standalone testing there is testing associated with each of the CIME models - ACME and CESM. The workflows for each modeling group are a little different and are described below:

• CESM System testing workflow