Testing

Testing

Before making a Pull Request to the ESCMI/CIME github, test your changes.

---

Changes to python

CIME uses doctest and unittest for unit testing, and unittest for regression tests. All of these tests can be run at once by running scripts_regression_tests.py.

Required

Run the regression test suite.

cd to scripts/tests ::

./scripts_regression_tests.py > scripts_regression_tests.out 2>&1

This will run X and A system tests (the same as what you get from running ./create_test cime_developer) in addition to the python-specific tests. If you want to skip the system tests, you can add --fast, but running the system tests is required prior to making a pull request. Capturing the output as shown above allows you to either verify that all tests pass or track down a problem.

The indication that all tests passed is from the last line of the captured output: All pass, removing directory: <scratch>/scripts_regression_test.date_time

If errors are indicated, see the What if something fails? section below.

** On Cheyenne **

1. Set the CIME_MODEL environment variable to cesm
2. Load the python environment:
   • Run either module load python/2.7.16 or module load python/3.6.8
   • Run ncar_pylib
3. Run using qcmd, tests will not run on login nodes

Recommended

./scripts_regression_tests.py B_CheckCode will run pylint on all python files with a few exceptions.

The above command invokes "code_checker" that lives in scripts/Tools. It supports checking the full code base in parallel and checking individual files. It also encapsulated the pylint settings that the team has agreed upon.

For more information on using scripts_regression_tests, see the CIME manual

scripts/Tools/advanced-py-prof Will profile your python code.

Useful

python doctest (https://docs.python.org/2/library/doctest.html) is used in many of the modules and the doctests are run as part of the A_RunUnitTests portion of scripts_regression_tests.py It's also possible to run doctest on an individual file. To do this you would run PYTHONPATH=$CIMEROOT/scripts/lib python -mdoctest testthis.py where testthis.py is the module you are interested in testing.

Continuous Intgeration Testing

CIME uses Jenkins to run scripts_regression_tests nightly. Latest results are on http://my.cdash.org/index.php?project=CIME.

What if something fails?

scripts_regression_tests.py outputs one line for each test class (some classes include more than one test). If the test passes that line will look like:

test_a_createnewcase (__main__.J_TestCreateNewcase) ... ok

if however the test fails that line will look like:

test_build_time_base_flags (__main__.H_TestMakeMacros) Test selection of base flags based on build-time attributes. ... FAIL

After all tests have completed, more detailed output will be printed for failed tests:

FAIL: test_build_time_base_flags_same_parent (__main__.I_TestCMakeMacros)

Test selection of base flags in the same parent element.

----------------------------------------------------------------------

Traceback (most recent call last):

File "./scripts_regression_tests.py", line 1800, in test_build_time_base_flags_same_parent

`tester.assert_variable_equals("FFLAGS", "-O2")`

File "./scripts_regression_tests.py", line 1599, in assert_variable_equals

`self.parent.assertEqual(self.query_var(var_name, env, var), value)`

AssertionError: '-O3' != '-O2'

If you want to rerun just the failed test you can do it by specifying the test name on the command line:

scripts_regression_tests.py H_TestMakeMacros

---

Changes to Fortran or to the test scripts in SystemTests

If there are changes to Fortran code in CIME or to the scripts in /utils/python/CIME/SystemTests, you must run the cime_developer tests.

1. Run the X and A system tests with ::

   ./create_test cime_developer

Note: The list of cime_developer tests is defined in cime/scripts/lib/update_acme_tests.py

As noted under "Changes to python", the cime_developer system tests are run automatically if you run scripts_regression_tests.py without the --fast argument - so if you have already run that, you do not have to separately run the above command.

2. If any changes have been made to Fortran code in share/ or driver_cpl/, then run cime unit tests: From the top-level directory in cime, follow the instructions in README.unit_testing (currently only works out-of-the-box on yellowstone).

In addition to CIME standalone testing there is testing associated with each of the CIME models - ACME and CESM. The workflows for each modeling group are a little different and are described below:

• CESM System testing workflow