Using pullrequest_test.py prior to merging to master

[Benjamin-Maglio]

@rarutter developed a testing framework for equilibrium to compare output variables between two branches (or commits on a single branch) which generates a PDF containing a series of plots which can be evaluated prior to merging, and included with the merge.

The PDF results may or may not demonstrate a change in the results and it is up to the user to interpret whether the result is as expected. In many instances no change in result will be expected, suggesting the structure of the model has been conserved. Yet, with feature development, a change is expected and it will be necessary to document why this change is as expected prior to merging with master.

The test can be initiated by altering dictionaries in /work/scripts/tests/pullrequest_test.py. This will provide a number of "test cases" which will be compared. At least two are needed, but theoretically more can be compared. An example is below.

test_config = {
  "parent_out_dir": "/data/workflows"
}

test_cases = {
  "0": {
    "name": "master",
    "branch_name": "master",
    "SHA": "",
    "input_dir": "/work/demo-data/cru-ts40_ar5_rcp85_ncar-ccsm4_toolik_field_station_10x10/",
    "working_dir": "/data/workflows/master",
    "cmt_num": 5,
    "pr_years": 10,
    "eq_years": 30,
    "sp_years": 0,
    "tr_years": 0,
    "sc_years": 0,
    "line_color": "black",
    "line_width": 4
  },
  "1": {
    "name": "Testing init test framework",
    "branch_name": "init_test_framework",
    "SHA": "",
    "input_dir": "/work/demo-data/cru-ts40_ar5_rcp85_ncar-ccsm4_toolik_field_station_10x10/",
    "working_dir": "/data/workflows/pr_test",
    "cmt_num": 5,
    "pr_years": 10,
    "eq_years": 30,
    "sp_years": 0,
    "tr_years": 0,
    "sc_years": 0,
    "line_color": "blue",
    "line_width": 2
  }
}

Here we are able to define two test cases to be output within parent_out_dir. In this case we want to compare to master so the branch in test case 0 is set to "master". Note "name" is used for book keeping.

It comparing cases with equivalent input forcing and parameterization "input_dir" and "cmt_num" need to be the same. Similarly with the run stage year lengths: "pr_years": 10, "eq_years": 30, "sp_years": 0, "tr_years": 0, "sc_years": 0. At this stage we are using this for equilibrium stage.

Note: you may need to install xarray and pypdf to generate the plots for the PDF.

Once adjusted, you may run the test with the following command python /work/scripts/pullrequest_test.py

As of writing more tests (i.e. transient run stage) are in the process of being developed.