Implement re-organized CCPP physics external (#306)

Originator(s): nusbaume

Summary (include the keyword ['closes', 'fixes', 'resolves'] and issue
number):

This PR bring in a new atmospheric_physics version with a re-organized
directory structure. The way this impacts CAM-SIMA is that now when a
physics suite is listed it will first look in the case's SourceMods
first, `ncar_ccpp/suites` second, and `ncar_ccpp/test/test_suites`
third, after which it will error if it hasn't found the Suite Definition
File (SDF). The CAM-SIMA build system also now looks in
`ncar_ccpp/schemes` for any CCPP physics source code and metadata files.

Fixes #305 

The associated atmospehric_physics PR can be found here:
ESCOMP/atmospheric_physics#126

Describe any changes made to build system:

The CAM-SIMA configuration routines will now look under `suites` or
`test_suites` for SDFs, with the `suites` entries taking precedence
(unless the case has SourceMods, in which those always take precedence).
All of the relevant source code and metadata files for atmospheric
physics must also now be present under the `schemes` directory in the
atmospheric_physics repo.

Describe any changes made to the namelist:  N/A

List any changes to the defaults for the input datasets (e.g. boundary
datasets): N/A

List all files eliminated and why:

Remove unused "test/include" directory:
D       test/include/Makefile
D       test/include/cam_abortutils.F90
D       test/include/cam_logfile.F90
D       test/include/ccpp_kinds.F90
D       test/include/shr_infnan_mod.F90
D       test/include/shr_kind_mod.F90
D       test/include/spmd_utils.F90

List all files added and what they do: N/A

List all existing files that have been modified, and describe the
changes:
(Helpful git command: git diff --name-status
development...<your_branch_name>)

Update atmospheric_physics external:
M       .gitmodules
M       src/physics/ncar_ccpp

Update CCPP SDF, source, and metadata file search locations and
precedence:
M       cime_config/cam_autogen.py

Add FTJ16 compset and cleanup simple physics configuration options:
M       cime_config/config_component.xml

If there are new failures (compare to the existing-test-failures.txt
file),
have them OK'd by the gatekeeper, note them here, and add them to the
file.
If there are baseline differences, include the test and the reason for
the
diff. What is the nature of the change? Roundoff?

derecho/intel/aux_sima:  ALL PASS

derecho/gnu/aux_sima:  ALL PASS

CAM-SIMA date used for the baseline comparison tests if different than
latest:

.gitmodules

@@ -14,7 +14,7 @@
 [submodule "ncar-physics"]
         path          = src/physics/ncar_ccpp
         url           = https://github.com/ESCOMP/atmospheric_physics
-        fxtag         = atmos_phys0_05_000
+        fxtag         = atmos_phys0_05_001
         fxrequired    = AlwaysRequired
         fxDONOTUSEurl = https://github.com/ESCOMP/atmospheric_physics
 [submodule "ccs_config"]

cime_config/cam_autogen.py

@@ -435,22 +435,31 @@ def generate_physics_suites(build_cache, preproc_defs, host_name,
     if not os.path.exists(physics_blddir):
         os.makedirs(physics_blddir)
     # End if
-    # Collect all source directories
-    atm_phys_src_dir = os.path.join(atm_root, "src", "physics", "ncar_ccpp")
-    source_search = [source_mods_dir, atm_phys_src_dir]
-    # Find all metadata files, organize by scheme name
+    # Set top-level CCPP physics directory
+    atm_phys_top_dir = os.path.join(atm_root, "src", "physics", "ncar_ccpp")
+    # Collect all possible Suite Definition File (SDF) locations
+    atm_suites_path      = os.path.join(atm_phys_top_dir, "suites")
+    atm_test_suites_path = os.path.join(atm_phys_top_dir, "test", "test_suites")
+    suite_search         = [source_mods_dir, atm_suites_path, atm_test_suites_path]
+    # Find all scheme metadata files, organized by scheme name
+    atm_schemes_path = os.path.join(atm_phys_top_dir, "schemes")
+    source_search    = [source_mods_dir, atm_schemes_path]
     all_scheme_files = _find_metadata_files(source_search, find_scheme_names)
 
     # Find the SDFs specified for this model build
     sdfs = []
     scheme_files = []
     xml_files = {} # key is scheme, value is xml file path
     for sdf in phys_suites_str.split(';'):
-        sdf_path = _find_file(f"suite_{sdf}.xml", source_search)
+        sdf_path = _find_file(f"suite_{sdf}.xml", suite_search)
         if not sdf_path:
             emsg = f"ERROR: Unable to find SDF for suite '{sdf}'"
             raise CamAutoGenError(emsg)
         # End if
+        if os.path.dirname(os.path.abspath(sdf_path)) == atm_test_suites_path:
+          #Notify user that a test suite is being used
+          _LOGGER.info("Using non-standard test suite: %s", sdf)
+        # End if
         sdfs.append(sdf_path)
         # Given an SDF, find all the schemes it calls
         _, suite = read_xml_file(sdf_path)
@@ -587,7 +596,7 @@ def generate_physics_suites(build_cache, preproc_defs, host_name,
     # there to the bld directory:
     if do_gen_ccpp:
         # Set CCPP physics "utilities" path
-        atm_phys_util_dir = os.path.join(atm_phys_src_dir, "utilities")
+        atm_phys_util_dir = os.path.join(atm_schemes_path, "utilities")
 
         # Check that directory exists
         if not os.path.isdir(atm_phys_util_dir):

cime_config/config_component.xml

@@ -159,12 +159,10 @@
       <value compset="_CAM\d0%WX.*%SDYN">-nlev 145</value> -->
 
       <!-- Simple models -->
-<!--      <value compset="_CAM%HS94" grid="(%ne[0-9])|(%[0-9.]+x)">-analytic_ic</value>
-      <value compset="_CAM%ADIAB">-phys adiabatic</value>
-      <value compset="_CAM%DABIP04">-phys adiabatic</value>
-      <value compset="_CAM%TJ16">-phys tj2016 -analytic_ic</value>
-      <value compset="_CAM%HS94">-phys held_suarez</value>
-      <value compset="_CAM%KESSLER">-phys kessler -chem terminator -analytic_ic</value> -->
+<!--      <value compset="_CAM%ADIAB">-phys adiabatic</value>
+      <value compset="_CAM%DABIP04">-phys adiabatic</value> -->
+      <value compset="_CAM%TJ16">--physics-suites tj2016 --analytic_ic</value>
+<!--      <value compset="_CAM%KESSLER">-phys kessler -chem terminator -analytic_ic</value> -->
       <value compset="_CAM%KESSLER">--physics-suites kessler --analytic_ic</value>
       <value compset="_CAM%HS94">--physics-suites held_suarez_1994 --analytic_ic</value>
       <value compset="_CAM%PHYSTEST">--dyn none --physics-suites adiabatic</value>