From 66a99fd1a2d37118ac22b4e3a431e7c673df05a7 Mon Sep 17 00:00:00 2001 From: Michael Waxmonsky Date: Wed, 24 Jul 2024 13:08:08 -0600 Subject: [PATCH] Updates from latest documentation branch. --- site/404.html | 308 +++- .../ccpp-conversion-guide/index.html | 318 +++- site/conversion/fork-cam-sima-2.png | Bin 0 -> 273513 bytes site/design/cam-build-process/index.html | 320 +++- site/design/cam-run-process/index.html | 389 +++- site/design/ccpp-in-cam-sima/index.html | 1261 +++++++++++++ site/design/constituents/index.html | 312 +++- .../{ => figures}/buildnml_workflow.jpg | Bin site/design/{ => figures}/cam_history_classes | Bin site/design/figures/capgen-design.PNG | Bin 0 -> 122805 bytes .../figures/ccpp-framework-in-cam-sima.PNG | Bin 0 -> 78428 bytes site/design/figures/ccpp-framework.PNG | Bin 0 -> 60012 bytes .../{ => figures}/constituents-classes.PNG | Bin .../{ => figures}/hist_config_classes.PNG | Bin site/design/figures/process-vs-time-split.PNG | Bin 0 -> 78252 bytes site/design/figures/run-sequence-api.png | Bin 0 -> 78035 bytes site/design/figures/run-sequence-dynamics.png | Bin 0 -> 73504 bytes site/design/figures/run-sequence-history.png | Bin 0 -> 75189 bytes site/design/figures/run-sequence-physics.png | Bin 0 -> 78920 bytes site/design/{ => figures}/run-sequence.gif | Bin site/design/history/index.html | 390 +++- site/design/sima-design-goals/index.html | 1067 +++++++++++ .../cam-coding-standards/index.html | 1241 +++++++++++++ site/development/cam-testing/index.html | 1112 ++++++++++++ site/development/debugging/index.html | 1095 +++++++++++ site/development/figures/fork_button.png | Bin 0 -> 12719 bytes site/development/git-basics/index.html | 1600 +++++++++++++++++ site/development/git-faq/index.html | 1038 +++++++++++ site/development/git-fleximod/index.html | 959 ++++++++++ .../tool-recommendations}/index.html | 459 ++++- site/index.html | 402 ++++- site/search/search_index.json | 2 +- site/sitemap.xml.gz | Bin 127 -> 127 bytes site/stylesheets/extra.css | 2 +- site/usage/creating-a-case/index.html | 1150 ++++++++++++ site/usage/history/index.html | 1199 ++++++++++++ 36 files changed, 14356 insertions(+), 268 deletions(-) create mode 100644 site/conversion/fork-cam-sima-2.png create mode 100644 site/design/ccpp-in-cam-sima/index.html rename site/design/{ => figures}/buildnml_workflow.jpg (100%) rename site/design/{ => figures}/cam_history_classes (100%) create mode 100644 site/design/figures/capgen-design.PNG create mode 100644 site/design/figures/ccpp-framework-in-cam-sima.PNG create mode 100644 site/design/figures/ccpp-framework.PNG rename site/design/{ => figures}/constituents-classes.PNG (100%) rename site/design/{ => figures}/hist_config_classes.PNG (100%) create mode 100644 site/design/figures/process-vs-time-split.PNG create mode 100644 site/design/figures/run-sequence-api.png create mode 100644 site/design/figures/run-sequence-dynamics.png create mode 100644 site/design/figures/run-sequence-history.png create mode 100644 site/design/figures/run-sequence-physics.png rename site/design/{ => figures}/run-sequence.gif (100%) create mode 100644 site/design/sima-design-goals/index.html create mode 100644 site/development/cam-coding-standards/index.html create mode 100644 site/development/cam-testing/index.html create mode 100644 site/development/debugging/index.html create mode 100644 site/development/figures/fork_button.png create mode 100644 site/development/git-basics/index.html create mode 100644 site/development/git-faq/index.html create mode 100644 site/development/git-fleximod/index.html rename site/{design/ccpp-physics => development/tool-recommendations}/index.html (62%) create mode 100644 site/usage/creating-a-case/index.html create mode 100644 site/usage/history/index.html diff --git a/site/404.html b/site/404.html index a5d736a9..209fde14 100644 --- a/site/404.html +++ b/site/404.html @@ -242,7 +242,7 @@ - Welcome to MkDocs + CAM-SIMA developer documentation @@ -360,7 +360,7 @@ - CAM-SIMA Build Process + Build process @@ -381,7 +381,7 @@ - CAM-SIMA Run Process + Run process @@ -398,11 +398,11 @@
  • - + - CCPP Physics + CCPP in CAM-SIMA @@ -444,7 +444,301 @@ - History & Model Output + History & model output + + + + +
    • + + + + + + + + + + +
  • + + + + + Design goals & features + + + + +
    • + + + + + + + + + + + + + + + + + + + + + +
  • + + + + + + + + + + +
    • + + + + + + + + + + + + + +
  • + + + + + + + + + + +
    • + + + + + + + + + + + + + +
  • + + + + + + + + + + +
    • + + + + + + + + + + + + + +
  • + + + + + + + + + + +
    • + + + + + + + + + + + + + +
  • + + + + + + + + + + +
    • + + + + + + + + + + + + + +
  • + + + + + + + + + +
    • + +
  • + + + Static run sequence images + + +
    • @@ -536,11 +545,11 @@
  • - + - CCPP Physics + CCPP in CAM-SIMA @@ -582,7 +591,301 @@ - History & Model Output + History & model output + + + + +
    • + + + + + + + + + + +
  • + + + + + Design goals & features + + + + +
    • + + + + + + + + + + + + + + + + + + + + + +
  • + + + + + + + + + + +
    • + + + + + + + + + + + + + +
  • + + + + + + + + + +
    • + +
  • + + + Static run sequence images + + +
    • @@ -727,14 +1039,18 @@ -

    CAM-SIMA Run Process

    -

    text

    +

    Run process

    +
    + text +
    CAM-SIMA run sequence*
    +
    +

    *Static images can be found at the bottom of this page

    CAM-SIMA API

    Upon running ./case.submit the core CAM-SIMA driver code* is in $CAM-SIMA/src/control/cam_comp.F90. This section lays out each of the subroutines within cam_comp.F90. The subroutines in cam_comp.F90 are set up to mirror the phases of the Common Community Physics Package (CCPP).

    * cam_comp.F90 subroutines are called by the NUOPC cap: $CAM-SIMA/src/cpl/nuopc/atm_comp_nuopc.F90

    cam_init

    -

    cam_init sets up the metadata and configuration objects/modules for the run. It is called by atm_comp_nuopc.F90 at startup. Below are the variables passed in:

    +

    cam_init sets up the metadata and configuration objects/modules for the run. It is called once at startup. Below are the variables passed in:

    @@ -902,15 +1218,15 @@

    cam_init

  • cam_ctrl_init (src/control/cam_control_mod.F90): Sets the module-level run configuration variables; logs configurations to the atm log
  • cam_ctrl_set_orbit (src/control/cam_control_mod.F90): Sets the module-level orbital variables
  • timemgr_init (src/utils/time_manager.F90`): Initializes the time manager; logs configurations to the atm log
    • -
  • read_namelist (src/control/runtime_opts.F90): Reads all namelists for the run, including auto-generated scheme namelists (see build process)
    • +
  • read_namelist (src/control/runtime_opts.F90): Reads all namelists for the run, including auto-generated scheme namelists (see build process)
  • cam_ctrl_set_physics_type (src/control/cam_control_mod.F90): sets module-level configuration for variables for simple physics and moist physics schemes; logs configurations to atm log
  • cam_initfiles_open (src/control/cam_initfiles.F90): Opens initial or restart file, and topography file if specified
    • -
  • cam_register_constituents (src/control/cam_comp.F90): Sets the total number and advected number of constituents; currently ALWAYS adds water vapor as constituent (expected by the SE dycore)
    • +
  • cam_register_constituents (src/control/cam_comp.F90): Sets the total number and advected number of constituents; currently ALWAYS adds water vapor as constituent (expected by the SE dycore)
  • air_composition_init (src/data/air_composition.F90): Initializes air-composition-dependent model constants
  • model_grid_init (src/dynamics/<dycore>/dyn_grid.F90): Initializes model grids and decompositions
  • cam_ccpp_initialize_constituents ($CASE/bld/atm/obj/ccpp/cam_ccpp_cap.F90): initializes the constituent data array; after this point, we cannot add new constituents
  • dyn_init (src/dynamics/<dycore>/dyn_comp.F90): Initializes the dynamical core
    • -
  • atm2hub_alloc and hub2atm_alloc (src/control/camsrfexch.F90): Allocate and set up surface exchange data
    • +
  • atm2hub_alloc and hub2atm_alloc (src/control/camsrfexch.F90): Allocates and sets up surface exchange data
  • phys_init (src/physics/utils/phys_comp.F90): Initializes physics (includes call to CCPP cap to run init phases of schemes in the Suite Definition File (SDF)
  • stepon_init (src/dynamics/<dycore>/stepon.F90): Initializes dynamics <--> physics coupling
    • @@ -920,18 +1236,22 @@

    cam_timestep_init

    1. stepon_timestep_init (src/dynamics/<dycore>/stepon.F90): First phase of dynamics (couple from dynamics to physics); also returns timestep for physics
    2. phys_timestep_init (src/physics/utils/phys_comp.F90):
        -
      1. Read data from initial data file (for the null dycore, this means we're reading most physics input variables (as defined in src/data/registry.xml) from the ncdata file; for the SE dycore, we are reading in any variables not marked as initialized by the SE dycore initialization)
        2. -
      2. Call the CCPP cap to run timestep_init phases of all schemes in the user-specified suite definition file (SDF)
        3. +
      3. Read un-initialized data from initial data file
          +
        • For the null dycore, this means we're reading most physics input variables (as defined in src/data/registry.xml) from the ncdata file
          • +
        • For the SE dycore, we are reading in any variables not marked as initialized by the SE dycore initialization
          • +
        +
        4. +
      4. Calls the CCPP cap to run timestep_init phases of all schemes in the user-specified SDF

    cam_run1

    -

    cam_run1 is the first "run" phase called in the physics loop. It is called BEFORE the mediator/coupler and calls the following subroutine (location):

    +

    cam_run1 is the first "run" phase called in the physics loop. It is called every timestep BEFORE the mediator/surface coupler and calls the following subroutine (location):

      -
    1. phys_run1 (src/physics/utils/phys_comp.F90): Calls the run phase for all physics schemes in the "physics_before_coupler" group in the suite definition file (SDF)
      2. +
    2. phys_run1 (src/physics/utils/phys_comp.F90): Calls the run phase for all physics schemes in the "physics_before_coupler" group in the SDF

    cam_run2

    -

    cam_run2 is the second "run" phase called in the physics loop. It is called AFTER the mediator/coupler. Input/output variables:

    +

    cam_run2 is the second "run" phase called in the physics loop. It is called every timestep AFTER the mediator/coupler. Input/output variables:

    @@ -955,11 +1275,11 @@

    cam_run2

    cam_run2 calls these subroutines (locations):

      -
    1. phys_run2 (src/physics/utils/phys_comp.F90): Calls the run phase for all physics schemes in the "physics_after_coupler" group in the suite definition file (SDF)
      2. +
    2. phys_run2 (src/physics/utils/phys_comp.F90): Calls the run phase for all physics schemes in the "physics_after_coupler" group in the SDF
    3. stepon_run2 (src/dynamics/<dycore>/stepon.F90): The second phase of dynamics (couple from physics to dynamics)

    cam_run3

    -

    cam_run3 is the third "run" phase called in the physics loop. It is called AFTER cam_run3 and BEFORE cam_run4 (unsurprisingly). In/out variables:

    +

    cam_run3 is the third "run" phase called in the physics loop. It is called every timestep AFTER cam_run3 and BEFORE cam_run4 (unsurprisingly). In/out variables:

    @@ -972,7 +1292,7 @@

    cam_run3

    - +
    cam_out (inout) surface exchange object - output from CAM-SIMAPassed into stepon_run2Passed into stepon_run3
    @@ -981,7 +1301,7 @@

    cam_run3

  • stepon_run3 (src/dynamics/<dycore>/stepon.F90): Calls dyn_run, which runs the dycore

    • cam_run4

    -

    cam_run4 currently does nothing!

    +

    cam_run4 currently does nothing! (but it is called every timestep)

    cam_timestep_final

    cam_timestep_final runs at the end of each timestep. In/out variables:

    @@ -1007,19 +1327,19 @@

    cam_timestep_final

    cam_timestep_final calls the following subroutines (locations):

      -
    1. History routines. If it's not the last (half) timestep,
        +
      1. History routines. If it's not the last (half) timestep,
        1. history_write_files (src/history/cam_history.F90): Writes fields to user-configured history files (if applicable)
        2. history_wrap_up (src/history/cam_history.F90): Closes files and zeros buffers as necessary
      2. phys_timestep_final (src/physics/utils/phys_comp.F90):
          -
        1. Calls the timestep_final phase for all physics schemes in the suite definition file (SDF)
          2. +
        2. Calls the timestep_final phase for all physics schemes in the SDF
        3. If ncdata_check is set in user_nl_cam, calls physics_check_data ($CASE/bld/atm/obj/phys_init/physics_inputs.F90) to perform snapshot checking

      cam_final

      -

      cam_final is called at the end of the model execution. In/out variables:

      +

      cam_final is called once at the end of the model execution. In/out variables:

      @@ -1043,10 +1363,15 @@

      cam_final

      cam_final calls the following subroutines (locations):

        -
      1. phys_final (src/physics/utils/phys_comp.F90): calls "final" phase of all schemes in the suite definition file (SDF)
        2. -
      2. stepon_final (src/dynamics/<dycore>/stepon.F90): finalize dycore (doesn't currently do anything)
        3. +
      3. phys_final (src/physics/utils/phys_comp.F90): calls "final" phase of all schemes in the SDF
        4. +
      4. stepon_final (src/dynamics/<dycore>/stepon.F90): finalizes dycore (doesn't currently do anything)
      5. atm2hub_deallocate and hub2atm_deallocate (src/control/camsrfexch.F90): deallocate cam_in/cam_out objects
      +

      Static run sequence images

      +

      text +text +text +text

      @@ -1093,7 +1418,7 @@

      cam_final

      - + diff --git a/site/design/ccpp-in-cam-sima/index.html b/site/design/ccpp-in-cam-sima/index.html new file mode 100644 index 00000000..80752d5f --- /dev/null +++ b/site/design/ccpp-in-cam-sima/index.html @@ -0,0 +1,1261 @@ + + + + + + + + + + + + + + + + + + + + + + + CCPP in CAM-SIMA - CAM-SIMA + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
      + +
      + + + + + + +
      + + +
      + +
      + + + + + + +
      +
      + + + +
      +
      +
      + + + + + +
      +
      +
      + + + +
      +
      +
      + + + +
      +
      +
      + + + +
      +
      + + + + +

      CCPP in CAM-SIMA

      +

      Overview

      +

      The core Common Community Physics Package (CCPP) documentation can be found here. This section details the code structure and implementation of the CCPP Framework within CAM-SIMA. That said here's a quick overview of the CCPP:

      +
        +
      • A CCPP-enabled model consists of the following components (depicted in the diagram below):
          +
        • Host model and dynamical core
          • +
        • CCPP-compliant physics schemes (organized into Suite Definition Files [SDFs])
          • +
        • Caps generated by the CCPP framework to connect the host model to the physics schemes
          • +
        +
        • +
      +

      text

      +

      CCPP-compliant physics schemes must adhere to the following criteria:

      +
        +
      • Must be broken up into only the following phases:
          +
        • register (anything in the scheme that must be done before the grid is initialized) - run once at startup
          • +
        • init - run once at startup
          • +
        • timestep_init - run at beginning of every physics timestep
          • +
        • run - run on every timestep
          • +
        • timestep_final - run at the end of every timestep
          • +
        • final - run once at the end of model execution
          • +
        +
        • +
      • Must contain metadata for all input/output/inout variables passed into each phase (see metadata example below)
        • +
      • Must not have use statements outside of the following allowed CCPP use statements (with the exception of dependencies, but we're not getting into that now):
          +
        • ccpp_kinds
          • +
        • ccpp_constituent_prop_mod
          • +
        +
        • +
      +

      Metadata example +(snippet taken from kessler.meta)

      +
      [ precl ]
+  standard_name = total_precipitation_rate_at_surface
+  long_name = Total precipitation rate at surface
+  units = m s-1
+  dimensions = (horizontal_loop_extent)
+  type = real | kind = kind_phys
+  intent = out
+[ relhum ]
+  standard_name = relative_humidity
+  long_name = Relative humidity
+  units = percent
+  dimensions = (horizontal_loop_extent, vertical_layer_dimension)
+  type = real | kind = kind_phys
+  intent = out
+[ scheme_name ]
+  standard_name = scheme_name
+  units = none
+  type = character | kind = len=64
+  dimensions = ()
+  intent = out
+
+
      +

      CCPP-compliant physics schemes are organized into suite definition files (SDFs). An SDF tells the framework which schemes will be run in what order. Separating schemes into "groups" also allows the run phases of those groups to be called separately by the host model. Here's an example SDF (from suite_kessler.xml):

      +
      <?xml version="1.0" encoding="UTF-8"?>
+
+<suite name="kessler" version="1.0">
+  <group name="physics_before_coupler">
+    <scheme>calc_exner</scheme>
+    <scheme>temp_to_potential_temp</scheme>
+    <scheme>calc_dry_air_ideal_gas_density</scheme>
+    <scheme>wet_to_dry_water_vapor</scheme>
+    <scheme>wet_to_dry_cloud_liquid_water</scheme>
+    <scheme>wet_to_dry_rain</scheme>
+    <scheme>kessler</scheme>
+    <scheme>potential_temp_to_temp</scheme>
+    <scheme>dry_to_wet_water_vapor</scheme>
+    <scheme>dry_to_wet_cloud_liquid_water</scheme>
+    <scheme>dry_to_wet_rain</scheme>
+    <scheme>kessler_update</scheme>
+    <scheme>qneg</scheme>
+    <scheme>geopotential_temp</scheme>
+    <scheme>cam_state_diagnostics</scheme>
+    <scheme>kessler_diagnostics</scheme>
+  </group>
+  <group name="physics_after_coupler">
+    <scheme>cam_tend_diagnostics</scheme>
+  </group>
+</suite>
+
      +

      The framework code is primarily python code that generates Fortran caps. The class structure looks like:

      +
      +text +
      +

      Given CCPP-compliant physics schemes and one or more SDF, the framework generates caps for the host model to call at the appropriate time. The core files generated by the framework are:

      +
        +
      • <host>_ccpp_cap.F90: contains the interface layer between the host and the suite(s)
        • +
      • ccpp_<suite>_cap.F90: contains one subroutine per phase (including one run phase per group) in which the phases of the schemes within the suite are called in order
          +
        • There is one suite cap generated for each suite being run
          • +
        +
        • +
      • ccpp_datatable.xml: consolidates metadata into an XML file to be used as desired by the host model
        • +
      +

      Code Structure

      +

      How CAM-SIMA and the CCPP come together: +text

      +

      Host Model

      +

      The core host model code is what is held in the CAM-SIMA github repository, plus code that is generated at build-time (or preview_namelists-time) based on the registry (src/data/registry.xml).

      +

      Physics

      +

      The CCPP physics scheme code exists in the atmospheric_physics repository, which exists as a submodule of CAM-SIMA in the following location: +$CAM-SIMA/src/physics/ncar_ccpp

      +

      SDFs are located in the root directory of the repository and scheme source code is in the relevant subdirectories.

      +

      The diagnostics directory contains all diagnostic schemes (the global ones used for state and tendency output, as well as the scheme-specific diagnostic schemes).

      +

      The to_be_ccppized directory contains physics schemes and utilties that have not been CCPP-ized, but were needed by an CCPP-ized scheme.

      +

      The utilities directory contains schemes that are used regularly, such as tendency applicators and state converters. See below for more.

      +

      Generated caps

      +

      The caps generated by the CCPP Framework at model build time (or preview_namelists-time) can be found in the following location: +$CASE/bld/atm/obj/ccpp/

      +

      Implementation

      +

      All CCPP phases are called from the physics driver (src/physics/utils/phys_comp.F90). You can see the order of these calls more thoroughly in the documented run sequence.

      +

      Host-side variables and metadata

      +

      For a CCPP-ized physics scheme to work, the framework needs to be able to find a matching variable on the host side for each input variable for the suite(s) in question. This means that CAM-SIMA needs to allocate, initialize, and provide metadata for these variables. We do this in two ways:

      +
        +
      • Adding metadata to existing CAM modules (such as src/data/physconst.F90) so that a scheme can access an existing host model variable
          +
        • All static host-side metadata schemes are included at the top of the registry
          • +
        +
        • +
      • Adding a variable to the registry (src/data/registry.xml)
          +
        • You can learn more about how the registry is used to generate a Fortran module and corresponding metadata here
          • +
        +
        • +
      +

      The registry-based code generator is run before the CCPP framework does its magic, so, when it's time, the framework can connect the dots between the host model and the physics.

      +

      State and tendency variables

      +

      Two of the most commonly used and referred-to objects in CAM-SIMA are:

      +
        +
      • physics_state
        • +
      • physics_tend
        • +
      +

      The Fortran for both objects is auto-generated by CAM-SIMA based on the registry ($CAM-SIMA/src/data/registry.xml). The generated code can be found here: $CASE/bld/atm/obj/cam_registry/physics_types.F90. The objects are used by the host model (CAM-SIMA), while the CCPP physics take the individual component variables as inputs.

      +

      Physics state

      +

      The physics_state object in CAM-SIMA contains the current values for a select set of variables that describe the atmosphere, at the resolution specified by the input grid.

      +

      Some examples of these core "state" variables include temperature (T), eastward wind (U), and northward wind (V)

      +

      As a rule, CAM-SIMA physics schemes do not update the state directly and instead return tendencies (see below) which are then applied to the state later in the run phase. This is called time splitting, which means that all physics since the last state update get the same input state. The alternative, process splitting, means that the output state of one scheme serves as the input state of the next.

      +
      +text +
      +

      NOTE: Although constituents are handled independently of the physics_state object (they are handled by the CCPP framework), they ARE considered state variables.

      +

      Physics tendencies

      +

      The physics tendencies represent how a given scheme (or schemes) changes the state in a single model timestep. The tendencies are accumulated until it is time to apply them to the state. There is one tendency for each state variable being "updated" by the scheme. Some of these tendency variables are held within the physics_tend object, but others are internal to the physics.

      +

      The module $CAM-SIMA/src/physics/ncar_ccpp/utilities/physics_tendency_updaters.F90 includes the schemes to apply the tendencies to the relevant state variables. These schemes are added to the SDF whenever the state should be updated. Each calculation looks like: state_var = state_var + tend*dt where dt is the timestep size.

      + + + + + + + + + + + + + +
      +
      + + + +
      + +
      + + + +
      +
      +
      +
      + + + + + + + + + + \ No newline at end of file diff --git a/site/design/constituents/index.html b/site/design/constituents/index.html index c6270489..b18ed003 100644 --- a/site/design/constituents/index.html +++ b/site/design/constituents/index.html @@ -9,7 +9,7 @@ - + @@ -251,7 +251,7 @@ - Welcome to MkDocs + CAM-SIMA developer documentation @@ -371,7 +371,7 @@ - CAM-SIMA Build Process + Build process @@ -392,7 +392,7 @@ - CAM-SIMA Run Process + Run process @@ -409,11 +409,11 @@
    2. - + - CCPP Physics + CCPP in CAM-SIMA @@ -579,7 +579,301 @@ - History & Model Output + History & model output + + + + +
      3. + + + + + + + + + + +
    3. + + + + + Design goals & features + + + + +
      4. + + + + + + + + + + + + + + + + + + + + + +
    4. + + + + + + + + + + +
      5. + + + + + + + + + + + + + +
    5. + + + + + + + + + + +
      6. + + + + + + + + + + + + + +
    6. + + + + + + + + + + +
      7. + + + + + + + + + + + + + +
    7. + + + + + + + + + @@ -743,7 +1039,7 @@ -

      History & Model Output

      +

      History & model output

      CAM-SIMA history is the mechanism for configuring and generating diagnostic output from a model run. It is also used to generate initial-data files and aids in the model-restart process by saving the state of diagnostic fields whose processing window (e.g., averaging, standard deviation) crosses a restart-write cycle. This page describes the implementation of CAM-SIMA history in CAM-SIMA.

      History Initialization

      Reading and processing the history configuration

      @@ -781,7 +1077,7 @@

      Reading and processing hist_output_frequency -Specifies the frequency of writes to the volume. The syntax is “*” where “time_period” can be: steps, seconds, minutes, hours, days, months, years. The closest CAM7 equivalent is “nhtfrq”. +Specifies the frequency of writes to the volume. The syntax is "<integer>*<time period>" where “time_period” can be: steps, seconds, minutes, hours, days, months, years. The closest CAM7 equivalent is “nhtfrq”. &hist_file_config_nl
       hist_output_frequency @@ -791,7 +1087,12 @@

      Reading and processing hist_write_nstep0 -Specifies the template for the filename for the volume.
      Defaults to "%c.cam.%u.%y-%m-%d-%s.nc" where "%c" is the case name, "%u" is the volume, "%y" is the year, "%m" is the month, "%d" is the day, and "%s" is the seconds +Indicates whether or not to write the nstep=0 sample to the volume.
      Defaults to .false. +&hist_file_config_nl
       hist_write_nstep0 + + +hist_filename_template +Specifies the template for the filename for the volume.
      Defaults to "%c.cam.%u.%y-%m-%d-%s.nc" where "%c" is the case name, "%u" is the volume, "%y" is the year, "%m" is the month, "%d" is the day, and "%s" is the number of seconds since midnight GMT, with the timestamp itself representing the model time when the file is created. &hist_file_config_nl
       hist_filename_spec @@ -801,64 +1102,7 @@

      Reading and processing
    8. The HistoryConfig object is created in buildnml out of entries in user_nl_cam and written to run/atm_in.
    9. In order to ensure that all relevant runtime (namelist) values show up in atm_in, the HistoryConfig object must contain all the logic in setting default values.
      10. -

      text

      -

      Example -Take the following sample user_nl_cam:

      -
      hist_output_frequency;h1: 5*ndays
-hist_max_frames;h1: 3
-hist_add_inst_fields;h1: U
-hist_add_inst_fields;h1: V, Q
-hist_precision;h1: REAL64
-hist_filename_spec;h1: my-history-file%m-%d
-hist_write_nstep0;h1: .false.
-
      -

      It will be parsed by hist_config.py and this will be the relevant section of atm_in:

      -
      &hist_config_arrays_nl
-    hist_num_inst_fields = 3
-    hist_num_avg_fields = 2
-    hist_num_min_fields = 0
-    hist_num_max_fields = 0
-    hist_num_var_fields = 0
-/
-
-&hist_file_config_nl
-    hist_volume = 'h0'
-    hist_avg_fields = 'T', 'Q'
-    hist_max_frames = 1
-    hist_output_frequency = '1*month'
-    hist_precision = 'REAL32'
-    hist_file_type = 'history'
-    hist_filename_spec = '%c.cam.%u.%y-%m-%d-%s.nc'
-    hist_write_nstep0 = .false.
-/
-
-&hist_file_config_nl
-    hist_volume = 'h1'
-    hist_inst_fields = 'U', ‘V’, ‘Q’
-    hist_max_frames = 3
-    hist_output_frequency = '5*ndays'
-    hist_precision = 'REAL64'
-    hist_file_type = 'history'
-    hist_filename_spec = 'my-history-file%m-%d'
-    hist_write_nstep0 = .false.
-/
-
      -

      In plain English, a one-month run with these history configuration will result in a total of three files that will look something like these: -- my-history-file01-06.nc - - This file will contain instantaneous output for U, V, and Q (eastward_wind, northward_wind, and water vapor) - - It will contain three frames, one at each of the following times: - - 0001-01-06 (time=5) - - 0001-01-11 (time=10) - - 0001-01-16 (time=15) -- my-history-file01-21.nc - - This file will contain instantaneous output for U, V, and Q (eastward_wind, northward_wind, and water vapor) - - It will contain three frames, one at each of the following times: - - 0001-01-21 (time=20) - - 0001-01-26 (time=25) - - 0001-01-31 (time=30) -- .cam.h0a.0001-02-01-00000.nc - - This file will contain averaged output for T and Q (air_temperature and water vapor) - - It will have one frame with the time calculated at the midpoint of the month

      +

      text

      Setting up the history data structures

      • History namelist information is read and history data structures are set up in src/history/cam_hist_file.F90
        • @@ -867,7 +1111,7 @@

        Setting up the history data stru
      • The hist_file_t object contains information about the configuration options for a given history volume. This includes the maximum number of frames that can be written to the file, the current number of frames on the file, the name of the file, and all of the history fields to be written to the file. It also contains methods to populate the field lists (config_set_up_fields), set up the metadata of the file (config_define_file), and write history fields to the file (config_write_time_dependent_variables).
      • Each hist_file_t object contains both a hash table and allocatable field list to keep track of the fields written to the file. The core class for each of these is the hist_field_info_t (in src/history/buffers/src/hist_field.F90), which contains information about a history field. This includes the field names, the accumulate flag (average, instantaneous, minimum, etc), units, type, dimensions, and fill value. It also includes the buffer(s) (type is/are hist_buffer_t) that will and do hold the actual data.

      -

      text

      +

      text

      Populating the possible field list

      The possible fields to be output by the history infrastructure are tracked in cam_history.F90 via the possible_field_list hash table. It is populated during init time by calls to the subroutine history_add_field (found in src/history/cam_history.F90). “Init time,” means that all calls to history_add_field must occur during the execution of cam_init (found in src/control/cam_comp.F90).

        @@ -941,7 +1185,7 @@

        Writing a history file

        The bolded step #3 above occurs any time the criteria for #1 is satisfied. At this point, the following call is made to write the history fields (whose data has been stored in their buffers via calls to history_out_field):

          call hist_configs(file_idx)%write_time_dependent_variables(file_idx, restart)
        -

        It is during this call that we increment the number of samples written for this volume and actual write the data held within the buffer(s) to the netcdf file (as well as the time-dependent metadata for the fields).

        +

        It is during this call that we increment the number of samples written for this volume and actually write the data held within the buffer(s) to the netcdf file (as well as the time-dependent metadata for the fields).

        Defining history restart files

        Restarts not yet implemented in CAM-SIMA

        Writing a history restart file

        @@ -994,7 +1238,7 @@

        Reading a history restart file

        - + diff --git a/site/design/sima-design-goals/index.html b/site/design/sima-design-goals/index.html new file mode 100644 index 00000000..5a454b24 --- /dev/null +++ b/site/design/sima-design-goals/index.html @@ -0,0 +1,1067 @@ + + + + + + + + + + + + + + + + + + + + + + + Design goals & features - CAM-SIMA + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
        + +
        + + + + + + +
        + + +
        + +
        + + + + + + +
        +
        + + + +
        +
        +
        + + + + + +
        +
        +
        + + + +
        +
        +
        + + + +
        +
        +
        + + + +
        +
        + + + + +

        Design goals & features

        +

        Motivated by the Singletrack project (a precursor to SIMA), the CAM-SIMA project was created to build a new CAM infrastructure to meet the SIMA science and computational needs. This page documents those needs and some of the features that implement them.

        +

        CAM needs to be more run-time configurable

        +
          +
        • To make CAM and CESM more available, e.g., for usage in containers and the cloud, a build of CAM should be more configurable than it is at present.
          • +
        • One feature that makes CAM more run-time configurable is moving physics suites to the CCPP. By allowing CAM to compile in more than one physics suite, the physics suite can be selected at run time (e.g., as a namelist variable).
          • +
        • Another feature needed to make CAM more run-time configurable is making dycores themselves more run-time configurable. For instance, the SE dycore will no longer require the number of advected constituents to be specified at compile time.
          • +
        +

        Remove obstacles to use of specialized high-performance processing units (e.g., GPUs, FPGAs)

        +

        The chunk structure in CAM physics served its purpose when threading was the only way to accelerate code. However, to make the best use of both threading and modern accelerators, a flexible chunking mechanism is required. The new infrastructure enables this by using flat arrays for all fields.

        +
          +
        • Moving to flexible precision of data is important for being able to test both performance improvements and the affect on model quality. The CCPP is explicitly designed to allow for compile-time selection of precision at the physics suite level as well as for individual fields. In addition, the new infrastructure is explicitly designed to handle the case where the dycore is running at a different precision than the physcs (i.e., by using proper field promotion and demotion primitives).
          • +
        • Pointers in Fortran are less efficient because they prevent some optimization techniques. The new infrastructure avoids pointers as much as possible by making use of the automatic data management capability of the CCPP (which does not create pointers).
          • +
        • The new infrastructure provides greater flexibility in that the model can be built with multiple physics suites to increase run-time flexibility. There is a tradeoff in that building more physics suites will often increase build time. Builds with a single suite should be faster than now since only the schemes that are required for the suite are compiled (currently most schemes are compiled all the time even if they will not be used).
          • +
        +

        Modularity

        +

        In order to continue to allow CAM to grow without an ever increasing cost of bringing in new features, CAM must be more modular. A classic example is chemistry which ACOM would like to make modular but which is currently entwined with CAM in many areas (e.g., code in CAM repository, extensive configuration code dedicated to chemistry, extensive namelist building code and data dedicated to chemistry, large number of use cases containing chemistry configuration data). The new CAM infrastructure contains features to increase modularity.

        +
          +
        • Support for multiple namelists. This allows modular components to contain their own namelist (run-time configuration options). The active namelists are combined to produce atm_in.
          • +
        • Flexible handling of constituent information. Modular components can provide constituent information via metadata (if component is a CCPP scheme) or at run time.
          • +
        +

        Modularity will allow CAM to move components to external repositories. This process cuts development costs for both CAM and for the component (e.g., as has happened with PUMAS). Some ways this is accomplished are listed here:

        +
          +
        • Code reviews are more efficient since CAM SEs do not have to review every routine in the external module so they can just focus on the interfaces. The external developers do not have to be involved in CAM modifications.
          • +
        • Externals can develop and maintain they own namelist definition files, they do not have to coordinate with the larger CAM namelist (which itself has been broken into several smaller namelists).
          • +
        • Namelists associated with physics schemes do not have to have separate namelist-reading code. The new infrastructure automatically creates an appropriate Fortran module to read in the runtime data from atm_in. The system then also ensures that all active namelists are called at run time. This process ensures that namelists are always read correctly while not requiring coding or reviews to keep up to date with namelist changes.
          • +
        +

        Use of the CCPP to build physics suites also makes CAM more modular because the CCPP treats physics schemes as modular which allows flexibility in building physics suites. The CCPP takes care of making sure variables are available before they are used and also builds the code currently handled via hand-written code in the various versions of physpkg.F90.

        +

        Run-time data checking

        +

        CAM needs data to run but the data needs vary with the simulation. The new infrastructure facilitates this.

        +
          +
        • Before running physics, the new infrastructures queries the physics suite as to what input fields are required (using a CCPP interface). Then it makes sure that all of these fields have been initialized or reads the values from the initial data file. Any uninitialized fields that are not found on the initial data file will trigger a run-time error.
          • +
        +

        Efficient offline testing and simulation

        +

        CAM currently has a few ways to run offline testing (e.g., SCAM, PORT). The new infrastructure builds these capabilities in for more efficient and flexible use.

        +
          +
        • The new infrastructure has the ability to run without a dycore.
          • +
        • Offline mode (NULL dycore) can be run with any number of columns.
          • +
        • Offline mode does not required gridded input.
          • +
        +

        Software quality control

        +

        To enable efficient quality control, the new infrastructure implements a number of continuous integration (CI) techniques.

        +
          +
        • To implement all the flexibility mentioned above, the new infrastructure makes extensive use of python scripts.
            +
          • Python scripts make extensive use of python doctests (for simpler tests).
            • +
          • There are python unit tests to cover more complicated situations.
            • +
          • The GitHub CI also runs a static analysis tool (pylint) to provide feedback on potential coding issues.
            • +
          • Python tests can easily be run by hand but are also automatically run on GitHub
            • +
          +
          • +
        • The new infrastructure will use offline mode (see above) to run many physics configurations quickly without requiring large machines.
            +
          • This will enable quick testing during development on a laptop.
            • +
          • We hope many of these tests can also be run automatically on GitHub.
            • +
          +
          • +
        + + + + + + + + + + + + + +
        +
        + + + +
        + +
        + + + +
        +
        +
        +
        + + + + + + + + + + \ No newline at end of file diff --git a/site/development/cam-coding-standards/index.html b/site/development/cam-coding-standards/index.html new file mode 100644 index 00000000..76bf8cd3 --- /dev/null +++ b/site/development/cam-coding-standards/index.html @@ -0,0 +1,1241 @@ + + + + + + + + + + + + + + + + + + + + + + + Coding standards - CAM-SIMA + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
        + +
        + + + + + + +
        + + +
        + +
        + + + + + + +
        +
        + + + +
        +
        +
        + + + + + +
        +
        +
        + + + +
        +
        +
        + + + +
        +
        +
        + + + +
        +
        + + + + +

        Coding standards

        +

        The standards in this document are largely prescriptive, i.e., they should be followed.

        +
          +
        • MUST: Exceptions must be discussed and agreed to by the CAM SEs and noted in this document.
          • +
        • SHOULD: Exceptions must be approved by the CAM SEs and documented in the ChangeLog.
          • +
        +

        While some legacy code will not follow these rules, efforts SHOULD be made to improve the code whenever you are working on it (e.g., bug fixes, enhancements).

        +

        General coding standards

        +

        MUST

        +
          +
        • Always use spaces instead of tabs
          • +
        • No trailing spaces (i.e., no spaces at the end of a line)
          • +
        +

        See tips for configuring editors

        +

        SHOULD

        +
          +
        • Use comments to explain the purpose of the following code and/or include any important but non-obvious information. When working with code, always check the comments to make sure they are still correct and useful.
          • +
        • Do not use comments to 'save code for later in case it might be useful'.
          • +
        • Do not include a comment that merely restates the following code logic (e.g., 'Loop over variables')
          • +
        +

        Python coding standards

        +

        We expect all python code to pass with a perfect score (10) using pylint with this version of pylintrc. However, external repos may have their own pylintrc version depending on their needs.

        +

        We also expect all python files to follow the black code style and format.

        +

        Finally, one can also follow the Google Python Style Guide.

        +

        Fortran coding standards

        +

        The standards described in this section represent the CAM Fortran standards. Other Fortran standards:

        + +

        MUST

        +
          +
        • No naked use statements
          • +
        • No continued single-line if statements (i.e., all if statements should have a then if the statement is on more than one line)
          • +
        • Every namelist variable in each active namelist group is present in the namelist file. An active namelist group is one which may be read during the current run.
          • +
        • All namelist variables except for logical quantities are initialized to invalid values (integer: -HUGE(1), real: NaN, character: 'UNSET').
          • +
        • Functions may not have side effects, and should include the pure keyword.
          • +
        • Do not combine statements on a single line (i.e., avoid use of the semi-colon to combine statements).
          • +
        • Use intent for dummy arguments except for pointers.
          • +
        • All variables of type real must have a specified kind, including literals. For example, use 1.5_r8, not 1.5 or 1.5D0. Literals must also include the decimal point.
          • +
        • All character declarations must use Fortran 90+ syntax (e.g., character(len=*) or character(len=CL)).
          • +
        • All variable declarations must use Fortran 90+ syntax (i.e., must include the double colon between the attributes and the variable name).
          • +
        • All type and procedure declarations must use Fortran 90+ syntax (i.e., must include the double colon before the type or procedure name).
          • +
        • All modules should include an implicit none statement in the preamble (after the use statements). Module routines then do not need this statement.
          • +
        • All optional arguments must be passed via keyword (e.g. use call subroutine(x, optional_y=y) instead of call subroutine(x, y) for the optional variable optional_y).
          • +
        • Initialize local (non-parameter) variables in subroutines and functions at the top of the executable code, NOT on a variable declaration lines.
            +
          • Initializing a local variable on a declaration line invokes the SAVE attribute and is not thread safe.
            • +
          • Local pointer variables MUST be initialized before other (non-initialization) statements. By default, use the nullify statement.
            • +
          +
          • +
        • All variables that are on the physics grid must have their horizontal dimension declared with pcols, even if only a subset of the variable is used in the subroutine or function.
          • +
        +

        SHOULD

        +
          +
        • Avoid use of preprocessor directives (e.g., #if, #ifdef). Always try for runtime variable logic instead.
          • +
        • Keep formula statements relatively short. Use temporary variables to break long formulas into easier-to-read sections.
          • +
        • Use subroutines to avoid repeated (cut and paste) code logic.
          • +
        • Avoid side effects in subroutines. Pass variables to routines instead of 'using' them from elsewhere.
          • +
        • Use the pure keyword if a subroutine has no side effects.
          • +
        • List dummy arguments one per line, however, related items may be grouped.
          • +
        • Dummy argument order should match the order in the argument list.
          • +
        • Use symbolic numerical comparison operators (e.g., ==, /=, <, >=) not old character versions (e.g., .eq.).
          • +
        • Avoid the use of pointers as dummy arguments (exceptions must be discussed in design or code review)
          • +
        • Modules should be default private. Public interfaces are declared after the private declaration.
          • +
        • private module interfaces (i.e., subroutines and functions) should be declared private in the module header.
          • +
        • Module names should conform to their filename (i.e., the module name should be the filename without the .F90).
          • +
        • Functions should use the pure attribute. If they cannot, the reason should be included as a comment in the function's preamble.
          • +
        • All functions and subroutines should avoid un-necessary statements (e.g. a blank return at the end of a subroutine).
          • +
        • use statements should be brought in at the smallest scope possible (e.g. inside individual subroutines instead of at the module level).
          • +
        +

        Indentation and style

        +
          +
        • Scoping: Indentation should follow scope. That is, whenever entering a new scope (e.g., module, subroutine, if, do), indent that scope relative to the scoping statement (recommended 3 spaces but each module should at least be self consistent).
          • +
        • A single line should be less than 133 characters long.
          • +
        • Continue lines: Indent continue lines 5 spaces or align with similar lines in statement.
          • +
        • Use spaces to ease reading statements (e.g., before and after operators, after commas except in a dimensions list)
          • +
        • Include a space after if, else, end, do, and while.
          • +
        • Include a space before and after ::
          • +
        • No space after only, i.e., only:, not only :.
          • +
        • When aligning code for readability, commas go immediately after a symbol (no space).
          • +
        +

        Tips for configuring editors

        +

        emacs (add to your .emacs file)

        +
          +
        • To automatically remove trailing spaces whenever you save a file:
          • +
        +
        (add-hook 'before-save-hook 'delete-trailing-whitespace)
+
        +
          +
        • To automatically indent with spaces instead of tabs:
          • +
        +
        (setq-default indent-tabs-mode nil)
+
        +
          +
        • To use 4 spaces for each indent:
          • +
        +
        (setq tab-width 4)
+
        +

        vi (add to your .vimrc file)

        +
          +
        • To automatically remove trailing spaces whenever you save a file:
          • +
        +
         autocmd BufWritePre * :%s/\s\+$//e
+
        + + + + + + + + + + + + + +
        +
        + + + +
        + +
        + + + +
        +
        +
        +
        + + + + + + + + + + \ No newline at end of file diff --git a/site/development/cam-testing/index.html b/site/development/cam-testing/index.html new file mode 100644 index 00000000..866ec483 --- /dev/null +++ b/site/development/cam-testing/index.html @@ -0,0 +1,1112 @@ + + + + + + + + + + + + + + + + + + + + + + + Testing - CAM-SIMA + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
        + +
        + + + + + + +
        + + +
        + +
        + + + + + + +
        +
        + + + +
        +
        +
        + + + + + +
        +
        +
        + + + +
        +
        +
        + + + +
        +
        +
        + + + +
        +
        + + + + +

        Testing

        +

        This page describes the various automated and manual tests that are run for CAM-SIMA whenever the code is modified, as well as instructions for how to add new tests.

        +

        Python unit testing

        +

        CAM-SIMA supports two kinds of python unit tests, doctest and unittest tests, both of which are part of the standard python library.

        +

        All unittest tests should be in:

        +

        CAM-SIMA/test/unit

        +

        while all files used by the tests should be in:

        +

        CAM-SIMA/test/unit/sample_files

        +

        All unittest tests are automatically run via Github Actions whenever a Pull Request (PR) is opened, modified, or merged.

        +

        All doctest tests are also run automatically as long as the scripts they are located in are under CAM-SIMA/cime_config or CAM-SIMA/src/data.

        +

        To manually run all of the unit tests at any time, simply run the following shell script:

        +

        CAM-SIMA/test/run_tests.sh

        +

        Finally, when adding new tests, determine if the test can be done in only a few lines with minimal interaction with external files or variables. If so, then it would likely be best as a doctest. Otherwise it should be a unittest test. Failure to follow this rule of thumb could result in test failures in the Github Actions workflow. Also remember to add your new tests to the run_tests.sh script so future users can easily run the tests manually.

        +

        Static Source Code Analysis

        +

        Python

        +

        Any python script which is added or modified via a PR will automatically be analyzed using pylint, and must have a score of 9.5 or greater in order to not be marked as a test failure. The hidden pylintrc file used for the analysis can be found here:

        +

        CAM-SIMA/test/.pylintrc

        +

        Users can also manually run pylint against the core python build scripts by running the following shell script:

        +

        CAM-SIMA/test/pylint_test.sh

        +

        Please note that pylint is not part of the standard python library, and so it may need to be installed before being able to run the shell script.

        +

        Regression Testing

        +

        Running the regression tests (manual)

        +

        NOTE: Regression testing on Derecho should be done for every PR before merging!

        +

        Users can manually run regression tests on Derecho to ensure that the model builds correctly in various configurations. The tests can be run with a local copy of CAM-SIMA by using the test_driver.sh script under $CAM-SIMA/test/system. To run the tests associated with a particular compiler option one can do the following commands:

        +

        For running GNU tests*:

        +
        env CAM_FC=gnu ./test_driver.sh -f
+
        +

        For running Intel tests*:

        +
        env CAM_FC=intel ./test_driver.sh -f
+
        +

        Running the script will produce a directory in your scratch space labeled aux_sima_<CAM_FC>_<timestamp>, where <CAM_FC> is the compiler you chose, and <timestamp> is the timestamp (starting with the date) of when the tests were started, along with a job submitted to the local batch system.

        +

        Inside the directory you should see an executable labeled cs.status.*. Running that command after the submitted job has finished will display the test results. Currently for all tests everything should be labeled PASS except the SUBMIT step, which should be labeled PEND. Any other label indicates that a test may have failed, and should be investigated.

        +

        Finally, the tests themselves are listed in <CAM-SIMA>/cime_config/testdefs/testlist_cam.xml. Any files that need to be included in order for the tests to run properly are located in <CAM-SIMA/cime_config/testdefs/testmods_dirs/cam/outfrq_XXX, where XXX is the name of the test. Additional information on the CIME testing system, which is what this testing infrastructure is built on, can be found online here.

        +

        *Note: you may also have to include the environment variable CAM_ACCOUNT on derecho, which points to your account key

        +

        Adding a new regression test

        +

        The test list can be found here: $CAM-SIMA/cime_config/testdefs/testlist_cam.xml

        +
          +
        • If you are adding a new machine, compiler or category for an existing test, add a new <machine> XML entry
          • +
        • If you are adding a fully new test, add a new <test> XML entry with the following structure:
          • +
        +
        <test compset="<COMPSET_NAME>" grid="<GRID_ALIAS>" name="<TEST_TYPE>_<TEST_MOD>" testmods="<RELPATH_TO_TESTMODS_DIR>">
+  <machines>
+    <machine name="<MACH_NAME>" compiler="<COMPILER>" category="<TEST_CATEGORY>"/>
+  </machines>
+  <options>
+    <option name="comment">COMMENT HERE</option>
+    <option name="wallclock">WALLCLOCK_TIME</option>
+  </options>
+</test>
+
        +
          +
        • <COMPSET_NAME>: component set alias (or long name) - you can see more about compsets here
          • +
        • <GRID_ALIAS>: model grid/resolution you'd like to run on - you can see more about grids here
          • +
        • <TEST_TYPE>: type of test to be run. You can find the testing types here.
          • +
        • <TEST_MOD>: test modifier that changes the default behavior of the test type. More here
          • +
        • <RELPATH_TO_TESTMODS_DIR>: relative path to the testmods directory for this run; usually looks something like "cam/some_directory_name/"
            +
          • The testmods directory will contain any namelist mods and XML configuration variable changes for the test (user_nl_cam and/or shell_commands)
            • +
          • testmods directories can be found in $CAM-SIMA/cime_config/testdefs/testmods_dirs/cam/
            • +
          +
          • +
        • <MACH_NAME>: machine name - will almost definitely be either derecho or izumi
          • +
        • <COMPILER>: compiler to be used (options: gnu, nag, intel, nvhpc)
          • +
        • <TEST_CATEGORY>: group of tests that this test belongs to - the default run by test_driver.sh is aux_sima (which is run for each PR to CAM-SIMA)
          • +
        • WALLCLOCK_TIME: maximum amount of time that the job will be allowed to run
          • +
        +

        Here is an example test entry for a 2-timestep smoke test of kessler physics on the MPAS grid, run with both intel and gnu

        +
          <test compset="FKESSLER" grid="mpasa480_mpasa480" name="SMS_Ln2" testmods="cam/outfrq_kessler_mpas_derecho_nooutput/">
+    <machines>
+      <machine name="derecho" compiler="intel" category="aux_sima"/>
+      <machine name="derecho" compiler="gnu" category="aux_sima"/>
+    </machines>
+    <options>
+      <option name="wallclock">00:10:00</option>
+      <option name="comment">GNU build test for MPAS dycore (with Kessler physics)</option>
+    </options>
+  </test>
+
        + + + + + + + + + + + + + +
        +
        + + + +
        + +
        + + + +
        +
        +
        +
        + + + + + + + + + + \ No newline at end of file diff --git a/site/development/debugging/index.html b/site/development/debugging/index.html new file mode 100644 index 00000000..66302a11 --- /dev/null +++ b/site/development/debugging/index.html @@ -0,0 +1,1095 @@ + + + + + + + + + + + + + + + + + + + + + + + Debugging techniques - CAM-SIMA + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
        + +
        + + + + + + +
        + + +
        + +
        + + + + + + +
        +
        + + + +
        +
        +
        + + + + + +
        +
        +
        + + + +
        +
        +
        + + + +
        +
        +
        + + + +
        +
        + + + + +

        Debugging techniques

        +

        Start with the CAM wiki.

        +

        CAM-SIMA-specific debugging

        +

        Build errors

        +

        Debugging tips if you get build errors:

        +
          +
        • If the output indicates that the error message or failure is coming from somewhere within $CAM-SIMA/ccpp_framework:
            +
          • If you're getting a clear error message, it's likely that you have something wrong with your metadata
            • +
          • If you're getting an error message that indicates that something is breaking in the framework code itself (something went uncaught) - consult the AMP SEs
            • +
          +
          • +
        • If the error happens during the atm build, you can see the full output of the atm build in the build log here: bld/atm.bldlog.*
          • +
        +

        Run-time errors

        +
          +
        • Start with the atm.log* - if the issue occurred during the execution of the CAM code, it will hopefully have a clear and concise error message
          • +
        • Move to the cesm.log* - it will hopefully include a stack trace for the error in question; if the error did not occur in the CAM code (or CAM did not properly trap the error), it will help you identify the source of the issue.
          • +
        • If neither log file contains helpful information, a few first steps:
            +
          • Resubmit the case; it could be a machine hiccup
            • +
          • Turn on DEBUG mode (if it's not on already) and rebuild/rerun
            • +
          • Look in your run directory for any log files called PETXXX - if there was an issue on the ESMF side of things, it will show up in one of these (there will be one PET file per processor)
            • +
          • Try a different compiler - maybe it'll give you a more helpful error message
            • +
          • set NTASKS=1 (./xmlchange NTASKS=1), do a clean rebuild (as instructed), and run again; maybe running in serial will identify the error
            • +
          • Look for the ***************** HISTORY FIELD LIST ****************** in the atm.log* file; if it's not there, the error occurred at init time
              +
            • If the error occurred during init time, try a new case with a different grid and/or dycore
              • +
            • If the model ran for a few timesteps before dying (look for the CAM-SIMA time step advanced message in the atm.log* file), it's likely that one or more variable that you introduced or modified has gone off the rails (value has become very large or very small or zero)
                +
              • Update your user_nl_cam to output all possible suspected variables to a history file at some point shortly before the model dies, then inspect the output to see if any are obviously wrong
                • +
              +
              • +
            • If the model completed all timesteps, try running a shorter case to see if the problem persists; if so, it's an error during the model finalization
              • +
            +
            • +
          • Run the TotalView debugger on izumi
            • +
          • Use the old standard - print statements - to narrow down where the code is stopping
            • +
          • Ask for help!
            • +
          +
          • +
        +

        Unexpected answer changes

        +
          +
        • Two paths here:
            +
          • You're getting unexpected DIFFs from the regression testing
              +
            • Consult with a scientist about whether differences are expected and for which configurations (compsets, resolutions, namelists parameters, etc)
              • +
            • If the differences are very small (look like round-off), consult with the other AMP SEs on whether we're ok with this
              • +
            • If the differences are indeed unexpected and larger than round-off, create a case using the code from the head of development and:
                +
              • place print statements in both code bases (your development branch and the head of development) to identify where the numbers are going awry OR
                • +
              • run the TotalView debugger OR
                • +
              • use the comparison tool described below ($CAM-SIMA/tools/find_max_nonzero_index.F90)
                • +
              +
              • +
            +
            • +
          • You're getting unexpected answer changes compared with CAM
              +
            • Consult with other AMP SEs about whether the differences appear to be due to round-off error
              • +
            • Use the comparison tool (LINK ONCE IT EXISTS): $CAM-SIMA/tools/find_max_nonzero_index.F90
                +
              • This tool can help you narrow down where the issue begins by printing out values at a specific index and comparing those with the "truth" (from CAM)
                • +
              +
              • +
            +
            • +
          +
          • +
        +

        TotalView

        +

        (COURTNEY - ASK CHERYL TO HELP WITH THIS)

        +
          +
        • Create and configure a new case (using gnu and only 1 task)
          • +
        • Build the case (./case.build)
          • +
        • Run command bash to change to bash (if not already)
          • +
        • Run the following commands:
          • +
        +
        np=1
+nthreads=1
+
+source .env_mach_specific.sh
+
+RUNDIR=`./xmlquery RUNDIR -value`
+EXEROOT=`./xmlquery EXEROOT -value`
+LID=`date '+%y%m%d-%H%M%S'`
+
+cd $RUNDIR
+mkdir timing
+mkdir timing/checkpoints
+echo `pwd`
+export OMP_NUM_THREADS=$nthreads
+totalview ${EXEROOT}/cesm.exe
+
        +
          +
        • exit to exit the totalview window and give up the node
          • +
        + + + + + + + + + + + + + +
        +
        + + + +
        + +
        + + + +
        +
        +
        +
        + + + + + + + + + + \ No newline at end of file diff --git a/site/development/figures/fork_button.png b/site/development/figures/fork_button.png new file mode 100644 index 0000000000000000000000000000000000000000..061b2cc8c6dd3f86b2dc24d0a1f2bb6526d9424f GIT binary patch literal 12719 zcmeHscUV)+ws$}&B2592-a)CM214k)gkGcwgb{WRf7KD>*nO@Z{&_R%@ca^_q_KY${ z@^!_uM%pS(G+VWM4*gR|__c}G6YcL_Y@EN5@ z3mQBQN>eyWy(oVg@RKuUrS7fxPV-M6Sb5s3{nOa}i~Bp14Rt)J7S@P5PASe@>a>Qv z?U*pezFLp%pRv;|B^D;+PxXpDFB2Pnge`U5WIYG_?HXB}Du8E$>S+ot;&S^Ab^L1H zHaPM$$i^(YMEpFN@0bdk170xs9dA^Q%?=d}Vy@e!$@q#c?uS!>*1P-0C?!(oIyWzW zsy@Y@qSnxURV_1Jo2b-&0%fr=Txa!us}6qts3mKk6Ui(wCcn+M0(s|grcyn4kYA3anck}&>w>VTv-anbo9&)4;w$uC zA<3U*op;#y?j?UYC-w6a{y=eaGkUcNLw|bbZcPqdc`RLuOQFA!iqAQV>jkG|_;}4- z)lPb1qfWtc(i%neFbLrf16l5>c5Htg88Q~-hly(6_*#xvMa9I9m1B?j4ls!YL6epq zH!$=bmJ6KOYe;hp8>R&)ns1`&<>XZs#=avxhY0MEMOFN216pit+;~DDAtS^_MpUxsNaH z!i1#d0qqXIuK;sndx$5VsYjstUtslQ07EZeu{Old^z0xT-q&OD_s+Ib>*Vm9@ zg;TY=e9tL}Zk@t5e!q@Cyr}^K$kc{JQzUX`IXLFkObaTLY%F8$Y`-1~nmT5HWckXLpwmTni&2bC+>^bAX2UwLI;lwDec(i-T0JYj!+P0>*d(uloXXC(sqJR3 zi7?xOF-Um?BdL*;d%lTE#&Zufp_W?%-%jzTvR_#bsTbF~eLKikxP;-#%J%nX8n zNUv0akB3Xu?Kic)3ThdfyQbCUStJQ0#+#H$si?w9cIb_0rX3Vf?bR_}3{wg=nk$}# zqqJ&*;}xn26bBqDQtqM1@F9-(0>>1NwF_O(W*k1e)fC$8SL*CiDewLW9qdJ6jtlZ( zJLJYC!|%td6E8ifn6MF&UaPdrD`4XxuW6{V`Fe#P(|%{a_1Ho6qTQT*+_MXg=0rC!OWS0=_{#$xZKd zm-avbgQA4LtGXEJswzCj#B#$YufjBH?g}p7sAM5aKDSLb>ONu7RVd+7gLFMi-1GAP zY?Zc#>-#i(QL&2cLC|E5&XLRoy(qotCNCG$N1R%{qN5Q}L^_t1i0`V9&C)=P3)?V6 zRVJhna_d=PqeRcX9Z^8QP>*3lxaHK8yz{C`*uIV8?FVn;tv^kwruI@qQ$J?elq2E= z$VPu3HM;xNXIRFt$oPHB-1ZRAP#OzS?vOKy5oQgYBPshpq?yJ%rL^)+8MDl6H1MIm=x1}H zaLa`c$Mf@EfEUXdA?JCXw%gtv>7;DA&^z?uF@9_W4D2t7s@Ou4$df}A73iYx@wGG3 zqHZp9zNx+CGW`RW#r!~c>aYfw>NYlXA(ZlraPWrHljHCka!~j#p_xMh<_(cL;@*0d z=<_~a89is2=V23gp@f)J7G{jBt?wLn`!BC{Y z>vdhC{G1~8y;+9N(^1Y9b_dG51D=d$5jd(3_wNpfmmLlsVH^$!Lq(#`EaknQq0qb2Lf&=^LV;g)h(yLyZlD46M{)HGS5YwrwJ=^JPRvCKuDyOac*mZaQn-GU<9K-BakJecbc&bHrv!}!%6NDY+?{!G)DM;(1;4+V}3%=n%nUXphxZ>Hi0B&SWYbEw<{{U zM@fh^ESY>>NhZSh)T)#vL#odFK0_Cq-#2sLe0&04=_XX@80`^Saq!Qc~UHorv{dk*=zc#V>cG$q2;nnJgz`N($x0gCrCW>%647Rp=**b};54FdC0Ff!lH7FD$(A#V2$k7Tn; zEF~*5TI26#!cu8|NAOPl*kVG3Br`1m(-?zaW*9C7_3ahAB=5-CE47$+4o z^}aYiPgJ*W2QydFmWU-Pp!fxUtfXsv335%en%3s`TP%-aJuAxibV5oWFloJU8Rbct_DNE#cbl0MFAxnc>r$4g6nyfjnSHY3_NSI9TA zl?c%3;p5B8is~xCYtp}YNA5mcQL%e?O_rCSZ`YNG&U9xH(se$a2A6!8fdAuGOPMN& zbCF=NB`4cumhIJtk6FZNB#~4osYQk2F75_rZjr6E#5+vMiEOWkZ)bDI)g;6c#CjwN z=e(y}Id|daNU0Dm7y&<-i*5kw^y|e{skz*L$Z39y@n-w`%m8-I7Ml@8-q-0DSs@Y| zCvwP+<@{%_;TVGNqo^fL=zeaGa;ibAmC~8}&&{Xkm?WLZtp)AQ&&NL-u}RKTR1ShK z4M`GCnd=4y%oQ5vg`Avc2A0Le1+I+xm^D z?BYRCV-o|=Fk{O#(*4-=_-H55%}-)N1KHo3?7t5H5H;JNQ>{l(RWRJqo*QQ2XpZ3a zuy;bIVgLX{!ovv$e}X{KnnM(@VL9XbG!3%J0h)kctu1+cz}F7 ze0*GJ1ec4a0}AHB<>10}P4NeZEW!njv~fb&I6BZ@bHdCWT~T6;jOcp$KgrRxa5)cj z;hzewpMTN2pe%Tl&<_IW{-6Ood_bTu7m$yOPlV@pd2}rl`j@nW%b!+6+mpuw=ETFx z4dk)6{|5;dl$_h&_5D)`7aerQ%cF&Gadbt(5pr$_2NcuqN}cRnU4GZ;>Vmi~`qgeb z3rik!qkd`rT}D9(s_~c1wHd8!?45o|T%&(STEPFpIk_V3eqk)&JP134J-Q(-XlCAj zz+Y$Pe@Xm9f39o(TSL(5{yF+@_WIS9Uv>q{I>KGAl`6@KFk+N6%hg= zga!G{xy*Ud7XeEPAue+P7(W++PejCM*c#vVaQ<3&Y^NAasj(NEfu1Uz73zxq111iCe(I z@@PsJ+S_dGVO9tpCkLxv71!ee*0AwF*gcX(vpf7D$}hzMpADy3moJii_xJpT&(-3`_df5AciaA`%f ztRvF%H~rcO=RcnQn6&I{ewm7%{?|kShQY6`D8}dlb3<7C8XvSB|EPjn!yK#-=o$IX zaQs`j%|CHV!`dMX1eT|2C{tCN$R4FdU(#`>G{{9kat$^Yr7|C9Ld zus@__9GyJT!ETLGcX#-g>i-Gw4+d2mI6AHz|5ej}hx}od-_|R1oBt?7uY>4imFKT@ z^-ov1j;8;`&z~;%zo-FC{U0U&mVN(YuK$?p-?G5J1^$nB{l{GYmIeMT@PEAP|C_n+ z{+^j59MGFTcXZzEs<1+V&e^feRTN|aSJ&Tnb-B^#l3PxS1}*>qH`DbG!y{Y54PA(X zQi96iEa6a+Z~=BYchb>sUNMzqrF1+dzo)uC(it25>c>CY2+%jjw=KT!El1Ihi!CKb zV5X9x&pX~o`NGIy!*;<8#g&#^wc^{Z6E3;o6|6r)9s&X79Eep^YZllV$?dz&lD%^bD7mYtDDhhXS zcqPjyMSPQ2&>waDc4oZXPP5o_erwBaaBw>+E-vTznYxk^3kOHU+2N*UHWC3%i&B&< zh%BLsY=|*K`G%eKN1BEDy`(ZE^dH1fi{66{Z_gOld0_=fhZ%FX70KX0>L8_2 z9r*0(%YFO$Li6vo1K%CW@;>J=y%Q|U$iN3F)H&@6OfLR%u+vwa2KrzzfJtoj?Zayn z$_1-vGXPNEEI-)Mb4b=75*9+U$G{uf)_M`?ABfG*A)HWIDNKr&eda}-9&vAXmo1H$ zoLqln5Bi!;!Z~1ntuJ2tg6Gx2LCx~=a)OV~uCXd7Z>e)iMA(Y4YN6C=tTGE;kn{^s z+{X*grAGi#yr9*mv#!HqFCRAKs;7N)zrrXsZP3g)Ywze_5D^LL?(XK}s}Ind*6$Or zJ$8`y+g7)=Wfu|>>P{66x!NA@8EXJ@DJAm+spos5o-PWp<4xc65awch8! z!5hY=pR9T5ghloRn40I^Vp%WNb6ry)uF55SA>798aa=}SS&y^B3}))QBVkMzHsgh4 zcM8UAi+5&g(rD?9v+I&Y3BMcTCW%L+P%qhk9&0I4Ni07149qD^t$8}aF}Kcdb)UDB z&WaqA@)*pA{3LVNe8)Y{L?(x`01vDVTE%JtW1wJJ#yY)XnkWKFtXO` zqxC|--IBFz*uugmyYHc?b(w00lp^9jCkbKExlVfriR6PJ48lkO$`Cj1O4r?Y_THfS zTS=FL)Un}b;uI>%k<_B{PVII8bl)FUY_oBGI>H?*G$8S=O7Xve^Exi>y3*p_y-K%l z8rGgwRn^I;n^M5{VmwLQM)fK6zGw5x%Q`uS8_63H1!I>PCC8FQ7`f~WOiY1|HvG;p zQzQpH%!EJPqS8hkTk5nGri~xJ*sxJLP=%YDYd#=jW;KyY*Kj&%*My4HJbn3PaPaQ! z+Z0erkh~>B=F7l9wae~qRrwctZIhEpJ8QlnH*d}LCblJ%E(v>`(8qG}mR zH(B96So1hIC)j*+OQg`K;f{;TKHsj-;kv@o*OwaXx+V_MUtbavhu0N%l-zrEwU+te z<0n}_*4>WVU8S4{={lFqKL+5|gOztk>hb57^M(pdYF@o^C`P&NSWJHiw9E$f{UB@7 z*3!bgML4ibCBY52o%mvWT(}yw9_}4q+|t_0zz?$eS?QeAhlLXomGtJ!*i&vKjqNZi zi&^~pPzW?xXkl|RBqW!-;7fByZ!aq&MM~}N9g_)c|c;jEuNB@$6Zl3=>21Y9!Ypy3z zVJx!t+rT-AUZHWeSDneT9rEXk60FZo(<7WN`(tnK?w%dCNs>_e?cVK*<7~+^B4?s9 zxj6I(yrdGmX|{1dk#sKVE{{PU-4WhR^uxL2?ZM_){N>epm*3gZ!1P?h70AnPGG6b< z{s)CX%*javZa`y^k&?&JH#ylB$}`V%pwN7mk4S!(Y2A@RZT0c$a)-88`J9H>bd6uy zQLmcn2_HZ|o`~W&NCGg>rXHi)U14!Eq+33{JMyA^B|r9?QZ!qp2D>JEEM-ZhW^=Hi z`aPDWGAk-$Q$0PQj*f?%hPBUEdy=(l^tkj&Oq#P*;@@lM5{{M3&42AlW@ce|?R3f) zLU_?Tl&h|!S}ToI#{2yFGkPk6tq;;!Bdpp0j*f0y<4vPQQvfFy?V|C?$;rjlRqV!Y zgC=V5f#~BJRC;j{*KKKX3jo&b>kA#656Hu9}0%)mUwM*xk*ABynl;D$1y<^py5CLZ*S%%OKgKT_ z)Y-|(KOd;nlyTb*H=lfwn54rRT?&R2j@MF9sCTrt(=##2kcie1)AhAsh3!N& z{X9H2eL?-buQ`OMqqlSQ=(|N6w~^XQ%ANkE6`9n%qwf-p8LcY2dwY!?d8wLbMq{4| zpD^hy<`4qxA1!ICsLXGFhx7|z>wbeK_I+)MLAp7wPmBqk1gkdi?nKZ<E=j8#>b zhg8FB>qAyaM>++1FN4z46|=P*rfWMEnnO;#zUYJZC|7g$-ZL@jZ?-q~jgd>3n%1q@ zE8-7)k=EO~s5?L6cfXVd1QXIKzI4y0nwyfoC>DET?pZ=RgI=T)z`i+_wnoH!i7({R z_nlEuPdq`VJ9{C$X>DIxRVF?!k$?Vvg7sw6_5~feBoU9gPY6D2_YLBc-9SLP40a%3 z3A5hJxbYFYA`8UlAhc-ev&+vGCG^J0X2bY+iPCl~KMyP_8F_i1(hEIzayDauE)HP~ zW%P_bEKi#iN?r6xMcw5XqB}zIi83`agZsKnqv^a3E@ic?+k6CO zY7hqi^h-=eD1@~GF>iAMmP>%PWAUl6hsNg{V~+2_*w(}zKN)^6Rb*JMsHjN4q=e1Q z&HeM#igE2qxG!7f)+d{xU_3H*r=8iyaa^WNO@TOqD31^v&CXKL89LH3iRfH2uF zRyVM}eX2l2+*4%E4|{H(3O25BE7I^J*x$c_!Khz={~;}n=jjG8q<-&d;(f3Oz3;it zmjU>IkDxPN06nqr>64|R!Zob9gJqf-4&>yt`1!#DBbHs4&tbeNimW9`&>k1B z8?aCX<<>lwkM6R%dyVR>W64(-I*8r>;tR4A3-(4r&OaJw ze@+-QO(gkW@Ip6@z`pID?hR52I_PYz?e|x?DAYp4zYGlp>J@DfKXcD7urzSr&SC<9 z9A|INf3KJuH8A6TWK!YqI8T4aH>WQ_Zt(jQcEud=wBL_sQuWOpo!#aak*{m*?Jovf zOr^W^uf%d`z$khR`)AtP+7u8A+<}2MB`B0dP%!q}H*0Jn3T5=x7U1vikJ8vdJ$UrC zrbcvqecgwjC2Gd|2*?7AiisfwyKX#}Z9C+s6+_~PgoTA=<>n53M5rtyz0Kip&%Nba zMI-8iR()?V^6SDH#6eRMwVk=}ul2ltiJf?%ZljDb9PD-sG39y-T znBFWjD9f?Tx#1-DgjE;l*NR7GfZk?Av#P_c{O3--&og{}zjQ{hF?NHwR=-9lS3OnJ zTl+5Zqf2{t_wsC=_sMzBnnUCJPbx(e_ z*S7OEnwsEs@2vo+-b%`GojvIqX^Uo4|3S4YGUhnlGL%Whrk)BLinHI6aQeXcg)qR< zFL*<-d$B4v1V`}6B=KFolY5a?sE+BT@jo+`vaM$3VF-@I}Ic>Gz5~l zYN_{^c1p?ulE_`Uyc?B=c{yw1+>(I)-O}>S!7l+ow5xBQyWY9^Z77%M)~#ErDZ;OF zpvjCC6>Q&oX`rVvA%uvYBtg|ANFqsFB)`k5YL0sH*EcLGXu+kW**a%;_pz3!EUmnp z3+aa|R=6OK1hP>5%Bt!ghw&J8lhvu{#Df*~Gta~6H*>ssm+uKH70A(Y55(TqIMC!@YWgDG>u>`X7oTcX zK~|P|so^Z?s`V{mYn+!8_i_9CzPA1{1^4crZd9)px3JKGjb9!#v!(KD&8M>jAvkNY zP@xRt>;Ar}5HU?rm8}MPh^c7^AswIBu%cce{t|j?osgV}9trpB@$|d$eav+uZ~KuJ z(ZSi_2CwJfSa;&N7&`4ioB$>2CLP!TwVt#rPL1`|O- zR!gHaKEvOcil9hOv4@EqWXiAP6w0iQ9g0nAg0=EmW@l;1$;oSLA+Hm=4%pT6G@N!W z#qt{-(QoW{g!7G(L@dQ;}aU7ZgtRa*)>dq`vO_zQDU3 z6{_bIj%;xFJ~tT$erruErl z3+F!E!BN1WUfDD6s`0xLN2e-!T$1;Y5+Jq^^rAwKi&+&Qsb|TmTcGFrFvQbTS@GRt zic~qx6mj$!NX|aLboM|9JuW_-NK*glev~3JJ{4~`lxnOY$_I(%-xf)$S!5`fNtg3< zcadeX{Na>~)E-JpF(k6y4(HY5}XviUsrG!QY$5LaFexb^Ziqu}Aw8T68VM6Ui8 zg3&`Yd|HztXHq{+itlJRNmGcI5~oLlE{ zrlt-sz3k~jO{eIqH=Ib*w%_HJDClZE5 zg8h2g1nia%fxiyiZv1%@`b&mUc8Np!`Xuhf^DCT7eQS=jdu&1I144k3oSJO0v{}Ia E0gUikKmY&$ literal 0 HcmV?d00001 diff --git a/site/development/git-basics/index.html b/site/development/git-basics/index.html new file mode 100644 index 00000000..f705791d --- /dev/null +++ b/site/development/git-basics/index.html @@ -0,0 +1,1600 @@ + + + + + + + + + + + + + + + + + + + + + + + Working with git and GitHub - CAM-SIMA + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
        + +
        + + + + + + +
        + + +
        + +
        + + + + + + +
        +
        + + + +
        +
        +
        + + + + + +
        +
        +
        + + + +
        +
        +
        + + + +
        +
        +
        + + + +
        +
        + + + + +

        Working with git and GitHub

        +

        Getting started with git

        +
          +
        • For a quick, visual introduction to git, try a 5 Minute Overview of Git on Youtube
          • +
        • Need an introduction to version control along with git and GitHub? Try Git and GitHub for Poets, a Youtube series
          • +
        • For a glossary of common terms used in git and GitHub documentation, see the GitHub glossary.
          • +
        • GitHub's Git Handbook offers a brief overview of git and GitHub, including example workflows and links to more in-depth resources.
          • +
        • Software Carpentry offers a hands-on introduction to git.
          • +
        • A highly-regarded, comprehensive git reference is the book Pro Git (available as a book or a free pdf).
            +
          • The first three chapters of Pro Git, along with sections 6.1 and 6.2 on GitHub, are particularly useful if you're getting started with git. [1]
            • +
          • The site also contains links to git reference pages along with introductory videos on git.
            • +
          • You can skip the last section of chapter 3, on rebasing, until you have more experience with git. It's easy to cause problems for yourself or others with rebasing so only use rebasing in CAM-SIMA following the workflow instructions below as rebasing rewrites history, which can make it impossible to reproduce exactly what you have done before.
            • +
          +
          • +
        • For a clear introduction to git internals, try Dissecting Git Guts by Emily Xie.
          • +
        • Git from the inside out is a blog post with accommpaning video from Mary Rose Cook that looks under the covers a bit to show what happens when you execute git commands.
          • +
        • Git from the Bottom Up by Josh Wiegley provides a quick glossary of the main terms describing a git repository with links to detailed descriptions. Good site for reminders about a git concept.
          • +
        • Need to really understand the science behind how git works? Try Advanced Git: Graphs, Hashes, and Compression, Oh My! by Matthew McCullough
          • +
        +

        Git also offers extensive built-in (i.e., command line) help, although it can be hard to understand until you are familiar with basic git concepts:

        +

        git help

        +

        git help COMMAND

        +

        man gittutorial

        +

        man giteveryday

        +

        How to set up your git development environment

        +

        There are several stages in setting up your git and GitHub development environment. Below, they are broken into three sections which represent the different stages:

        + +

        One time GitHub setup

        +
          +
        1. Set-up personal github account (if one does not already exist): https://github.com
          2. +
        2. +

          Create a new personal CAM-SIMA fork of the ESCOMP/CAM-SIMA repo:

          +

          text

          +
          3. +
        3. +

          Set up SSH keys on GitHub (optional):

          +

          If you will be pushing changes to GitHub (either to your own fork or to shared forks), or if you will be pulling changes from private repositories on GitHub, then it's worth the time to set up ssh keys for each machine you'll be using. By doing so, you won't have to enter your password when pushing to or pulling from GitHub.

          +

          See here for instructions. After doing this, you can use the ssh form of GitHub URLs (e.g., git@github.com:ESCOMP/cam-sima.git) in place of the https form.

          +
          4. +
        4. +

          Configure GitHub notifications (optional): It is important to keep track of activity on GitHub but there are ways to manage how you are notified.

          +
            +
          • Controlling how you receive notifications: Click on your profile picture in the upper-right of any GitHub page, then click on "Settings", and then on "Notifications". You will see several options, here are a few recommended settings:
              +
            • Automatically watch repositories: Yes (check box)
              • +
            • Participating: Email
              • +
            • Watching: Web
              • +
            +
            • +
          • To see and manage Web notifications: Click on the bell in the upper right of any GitHub page to see Web notifications and to manage which repositories you are watching. There will be a blue dot on the bell if there are new (unread) notifications).
            • +
          +
          5. +
        +

        Set up git environment on new machine

        +

        git has global settings which apply to all repository clones on your machine. These settings reside in a file called .gitconfig in your home directory. Below are some required and some optional but recommended global git settings to apply to any new machine where you will do CAM-SIMA development (e.g., Derecho, Izumi, personal laptop). Apply the settings below or simply copy a .gitconfig file from a machine that is already configured.

        +

        Required git global configuration settings

        +
        git config --global user.name "Your Name"
+git config --global user.email <GitHub email address>
+
        +

        We recommend that you use your UCAR email address as your GitHub email address but if you use another address, you can add your UCAR email address by clicking on your profile picture in the upper-right of any GitHub page, then clicking on "Settings", and then on "Emails".

        + +

        You can set which editor to use for log messages, etc., with:

        +
        git config --global core.editor <editor of your choice: emacs, vi, vim, etc>
+
        +

        (See http://swcarpentry.github.io/git-novice/02-setup for specific settings to use for many common editor choices.)

        +

        The following setting generates better patches than the default:

        +
        git config --global diff.algorithm histogram
+
        +

        The following setting makes it easier to resolve conflicts if you're doing conflict resolution by hand as opposed to with a dedicated conflict resolution tool. Without this setting, you'll just see your version and the version you're merging in delimited by conflict markers. With this setting, you'll also see the common ancestor of the two sides of the merge, which can make it much easier to figure out how to resolve the conflict:

        +
        git config --global merge.conflictstyle diff3
+
        +

        Alternatively, look into using a graphical conflict-resolution tool such as kdiff3 or the Emacs built-in M-x vc-resolve-coflicts.

        +

        We recommend that you set git to not push anything by default:

        +
        git config --global push.default nothing
+
        +

        This can help prevent you from accidentally pushing work to the wrong repository.

        +

        Configuring git on shared machines

        +

        If using git on shared resources, such as on the login nodes for CISL machines, then one may find their git commands being killed by sys admins due to git spawning too many threads and thus blocking (or at least slowing down) other users. To avoid this situation, you can limit the number of threads git spawns for various activities by setting the following git config variables:

        +
        git config --global --add index.threads 8
+git config --global --add grep.threads 8
+git config --global --add pack.threads 8
+
        +

        Please note that a limit of 8 threads was chosen specifically for CISL machines. If you are using a separate shared system you may find it beneficial to choose a different thread limit.

        +

        git tools for Bash

        +

        There are two helpful things you can do to streamline your use of git if you use the bash shell. These are completely optional, but improve your git experience. These are documented in the appendix of the excellent Pro Git book.

        +

        Updating your login to a GitHub token

        +

        If you have been denied access to push to a GitHub repo, note that the old password system has been deprecated and no longer works. If you do not make changes, you will no longer be able to work with GitHub on personal forks and private repos.

        +

        If you only ever git clone from public repositories like ESCOMP and ESMCI, you may ignore the rest of this wiki page.

        +

        If you've already created and are using a GitHub token, you may ignore the rest of this wiki page.

        +

        GitHub will soon be requiring that you use a GitHub generated token (basically a long password that they autogenerate for you).

        +
          +
        1. If you are on a Mac and do not need to use your password, but still receive an error message, you may have stored your password in the keychain. Remove a password on Macs
          2. +
        2. Create a GitHub password token. Make sure you copy it to a safe place as they only show it to you once.
          3. +
        3. +

          Store your credentials on every machine you use GitHub on.

          +
            +
          • +

            On a private / secure Linux system:

            +

            git config --global credential.helper store

            +
            • +
          • +

            Note that the store option to git's credential helper stores your GitHub token in plain text. If you are on a public machine, you should store your GitHub token in a password manager and use the credential cache mechanism to open command-line access to your GitHub repositories:

            +

            git config --global credential.helper cache --timeout <seconds>

            +

            where <seconds> is the number of seconds git 'remembers' your token. For example, to only have to enter your token once per day:

            +

            git config --global credential.helper cache --timeout 86400

            +
            • +
          • +

            On Windows:

            +

            git config --global credential.helper wincred + - On Macs (make sure you select the Mac tab in the middle of the window): see this documentation

            +
            • +
          +
          4. +
        4. +

          On each machine, do a git clone of your personal repo. It will ask for your username and password. The password needs to be your saved token. You should only need to do this once.

          +
          5. +
        5. To test that your credentials are successfully cached, do a new git clone of your personal repo. If it clones without asking for your username/password you are done.
          6. +
        +

        Working with clones

        +

        Here are some commands for creating and working with clones:

        +

        Create a new clone

        +
        git clone https://github.com/<GitHub userid>/CAM-SIMA
+cd CAM-SIMA
+
        +

        or

        +
        git clone git@github.com:<GitHub userid>/CAM-SIMA
+cd CAM-SIMA
+
        +

        where <GitHub userid> is your GitHub account login ID. Some useful options to the clone command are:

        +
          +
        • --origin <origin name>: A clone knows where it came from and by default, calls that location, "origin". You can change that name with the --origin option. This can come in handy when dealing with multiple upstream repositories. +Change the clone directory name: By default, the clone is created in a directory with the same name as the repository. You can change this by adding a directory name to the clone command. Use the appropriate example below:
          • +
        +
        git clone https://github.com/<GitHub userid>/CAM-SIMA  <clone_dir_name>
+cd <clone_dir_name>
+
        +

        or

        +
        git clone git@github.com:<GitHub userid>/CAM-SIMA <clone_dir_name>
+cd <clone_dir_name>
+
        +

        Checking out a tag or switching to a new tag

        +
          +
        • To check out a tag:
          • +
        +
        git checkout <tag>
+
        +

        note that <tag> can also be the name of a branch or a commit hash. If you specify the name of a branch, you will check out the head of the branch. If you name a remote branch (e.g., origin/branch_name), you will create a detached HEAD but you can still use the code. Please note that if you plan on changing the code, first create a branch (see Working with branches

        +

        Working with branches

        +

        When you create a clone, your clone will contain pointers all the branches that existed at the clone's origin (e.g., the repository at GitHub). While you can check out these branches, however, before attempting to make any changes, you should first create a local version branch (so git can keep track of the local commits).

        +
          +
        • To create a new local branch that starts at a certain point:
          • +
        +
        git branch <new branch name> <tag or branch name>
+
        +

        for example

        +
        git branch new_feature cam6_2_024
+
        +
          +
        • To check out a local branch:
          • +
        +
        git checkout <new branch name>
+
        +
          +
        • If you are working with a repository that uses git-fleximod (e.g., CAM-SIMA, CESM), always run that tool after checking out a new branch or tag:
          • +
        +
        bin/git-fleximod update
+
        +

        Working with remotes (upstream repository locations)

        +

        While working with clones created using the methods above will be sufficient for most if not all of your development needs, there may be times when you will want to access or compare your code with code from a different repository. git has no problem storing revisions from multiple repositories in a single clone!

        +

        To begin, your clone probably has a single remote (also known as an upstream repository). To see the current status of which upstream repositories are configured for your clone, use the git remote command:

        +
        git remote
+
        +

        To see the location of the remote repositories in your current directory:

        +
        git remote -v
+
        +

        You should see something like:

        +
        origin  https://github.com/gituser/CAM-SIMA (fetch)
+origin  https://github.com/gituser/CAM-SIMA (push)
+
        +

        This tells you the "upstream" location from where new code is downloaded (when you run git fetch origin) or where code is uploaded (when you run git push origin <branch>). Note that most git commands are purely local, using only information in the .git directory of your clone.

        +

        You can rename an existing remote:

        +
        git remote rename origin ESCOMP
+
        +

        You can set the remote name as part of a clone command (the default is 'origin'):

        +
        git clone -o ESCOMP https://github.com/ESCOMP/cam-sima
+
        +

        Adding remote (new upstream repository locations)

        +

        To add a new upstream repository, use the remote add command. For example:

        +
        git remote add ESCOMP https://github.com/ESCOMP/CAM-SIMA
+git fetch --tags ESCOMP
+
        +

        You should see messages much like a new clone when you execute the git fetch command. Note that you can call the new remote anything, in this example we are calling it ESCOMP.

        +

        Updating your branch to latest development

        +

        Note that while this section explains how to update your local branch to the ESCOMP/CAM-SIMA/development branch, the instructions can easily be generalized for any branch from any upstream remote.

        +

        Before starting, you should have either:

        +
          +
        • A fresh clone of your fork with the branch you wish to update checked out (see Create a new clone and Working with branches.
          • +
        • An existing clone with the branch you wish to update checked out and in a clean state (i.e., make sure you do a git commit and that git status shows no modified files).
          • +
        +

        Add the upstream remote, if you have not already done so (see Adding remotes).

        +

        Merge the specific remote/branch into your branch. In this example, it is ESCOMP/development

        +
        git fetch ESCOMP
+git merge ESCOMP/development
+
        +

        Comparing differences using git diff

        +

        If you have a git clone, you can view differences between commits or tags. As far as git diff is concerned, a commit hash is the same as a tag so in the examples below will use <tag>.

        +
          +
        • To see the full difference between two tags (i.e., a changeset):
          • +
        +
          git diff <tag1> <tag2>
+
        +
          +
        • To see the full difference between the current checkout (sandbox) and a tag:
          • +
        +
          git diff <tag>
+
        +
          +
        • To see only the names of files that are different:
          • +
        +
          git diff --name-only <tag> [ <tag2> ]
+
        +
          +
        • To see the difference in one or more specific files:
          • +
        +
          git diff <tag> [ <tag2> ] -- <path_to_file1> <path_to_file2>
+
        +

        Configuring and using a graphical difference tool

        +

        git has a command, difftool, that can run a graphical tool on each file that is different between two commits.

        +
          +
        • To configure opendiff as the graphical difference tool:
          • +
        +
          git config --global diff.tool opendiff
+
        +
          +
        • To see the available graphical difference tools:
          • +
        +
          git difftool --tool-help
+
        +
          +
        • To run difftool on <file1> and <file2>
          • +
        +
          git difftool <tag> [ <tag2> ] -- <path_to_file1> <path_to_file2>
+
        +
          +
        • To optionally run difftool on all files in a changeset (answer 'y' or 'n' for each file):
          • +
        +
          git difftool <tag> [ <tag2> ]
+
        +
          +
        • To run difftool on all files in a changeset (i.e., same as 'y' for every file):
          • +
        +
          yes | git difftool <tag> [ <tag2> ]
+
        +

        Using diffmerge

        +

        This section contains the complete modifications needed for using the graphical tool diffmerge for "git difftool" and "git mergetool"

        +

        There is a tool called diffmerge which enables both side-by-side comparison of edited files in git as well as providing a three way-pane for editing merge conflicts. This tool is available for download at: https://sourcegear.com/diffmerge/. It has been installed on izumi and derecho by the system administrators in public areas, so you don't need to download it for those machines.

        +

        To use the differencing tool type: git difftool

        +

        If after a git merge the git command says there are conflicts, then you may type git mergetool to allow you to resolve the conflicts and complete the merge. The mergetool with diffmerge properly installed will display three panes. From left to right these panes are:

        +
        Code being merged in          Merged code               Your code
+
        +

        The panes are opened in a visual editor, and any changes you make in the middle pane, may be saved by selecting the save icon at the top of the screen and then exiting the window. This will finalize the merge for that particular file.

        +

        For a presentation which can be used as a tutorial, you may refer to: Presentation on diffmerge tool for git

        +

        The following modifications may be copy/pasted into the user's .gitconfig file which resides in the home directory. Since the potential for making an editing mistake is possible, it is recommended that a copy be made of the .gitconfig file prior to these edits in case an error is discovered.

        +

        If you have problems, check out the tips at the bottom of this page

        +
        [diff]
+        tool = diffmerge
+        algorithm = histogram
+[difftooldiffmerge]
+        cmd = diffmerge \"$LOCAL\" \"$REMOTE\"
+[difftool "diffmerge"]
+        cmd = diffmerge \"$LOCAL\" \"$REMOTE\"
+[difftool]
+        prompt = false
+[push]
+        default = simple
+[merge]
+        tool = diffmerge
+        ff = false
+[mergetool "diffmerge"]
+        cmd = diffmerge --merge --result=$MERGED $REMOTE $BASE $LOCAL
+[mergetool]
+        keepBackup = false
+
        +

        Useful tips

        +
          +
        • On derecho, if you get an error message about diffmerge not being loaded, make sure you are using ncarenv/23.09 or newer (to check say "module list")
          • +
        • If you do not get a response when clicking on the "git mergetool" window, make sure you find the "conflict" popup and hit "exit" on it
          • +
        • If your git mergetool gives blank windows and says: "Files are identical or equivalent under the current RuleSet", hit OK and then go to the File dropdown menu and select "Reload". This filled in the blank windows for the user with this error.
            +
          • CAUTION: Check your file carefully if you do this, as a second user who did this discovered that all of the changes which were automatically merged did not reside in the final saved version.
            • +
          +
          • +
        + + + + + + + + + + + + + +
        +
        + + + +
        + +
        + + + +
        +
        +
        +
        + + + + + + + + + + \ No newline at end of file diff --git a/site/development/git-faq/index.html b/site/development/git-faq/index.html new file mode 100644 index 00000000..c32e80e7 --- /dev/null +++ b/site/development/git-faq/index.html @@ -0,0 +1,1038 @@ + + + + + + + + + + + + + + + + + + + + + + + git & GitHub FAQ - CAM-SIMA + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
        + +
        + + + + + + +
        + + +
        + +
        + + + + + + +
        +
        + + + +
        +
        +
        + + + + + +
        +
        +
        + + + +
        +
        +
        + + + +
        +
        +
        + + + +
        +
        + + + + +

        git & GitHub FAQ

        +

        Q: How can I clone someone's git clone to my directory on the same machine?

        +

        A: git clone <path_name> [<local_dir>] where <path_name> is the location of the source git clone and the optional <local_dir> is the name of the clone (default is a local directory of the same name as the original).

        +

        Q: How can I clone someone's git clone to my directory on a different machine?

        +

        A: git clone ssh://<user>@<machine>:<path_name> [<local_dir>] where <user> is your user name on the remote machine, <machine> is the machine name (e.g., derecho.hpc.ucar.edu), <path_name> is the location of the source git clone on the remote machine, and the optional <local_dir> is the name of the clone (default is a local directory of the same name as the original).

        +

        Q: How can I look at someone's PR code?

        +

        A: There a a few ways to do this:

        +
          +
        • On GitHub (like looking at any other code on GitHub)
          • +
        • Add the PR fork to my remote (allows using tools such as git diff or git difftool with your existing branches or development)
          • +
        • As a clone (standalone clone on your machine).
          • +
        +

        A first step is to find the link to the fork's branch. Just below the PR is a line that starts with a colored oval (e.g., "Open" in green) and looks something like:

        +
        Octocat wants to merge 123 commits into ESCOMP:development from Octocat:amazing-new-feature
+
        +

        Clicking on the last part (Octocat:amazing-new-feature) will take you to the correct branch where you can browse the code (the first method above). If you want to download that code, click the green "Code" button and then click the clipboard icon. Be sure to take note of the branch name.

        +
          +
        • To load this code into your clone, cd to your clone directory, add the PR fork as a new remote, and checkout the branch. For instance:
          • +
        +
            git remote add octocat https://github.com/octocat/CAM-SIMA.git
+    git fetch --no-tags octocat
+    git checkout octocat/amazing-new-feature
+
        +

        Instead of the checkout you can also do diffs:

        +
            git difftool origin/development octocat/amazing-new-feature
+
        +
          +
        • If you want to make a new clone with the PR code, simply do:
          • +
        +
            git clone -b amazing-new-feature octocat https://github.com/octocat/CAM-SIMA.git octocat_cam
+    cd octocat_cam-sima
+
        +

        Q: Why do Pull Request (PR) code review take so long?

        +

        A: A code review must fulfill three purposes:

        +
          +
        • Code reviewers must make sure that new and/or changed code does not affect any currently-supported functionality (i.e., it cannot break anything).
            +
          • While regression tests will catch many of these issues, reviewers must also check for usage of or reliance on deprecated code, and also for any code which is not supported on all platforms and compilers used by the CESM community.
            • +
          +
          • +
        • Code reviewers must make sure that any new functionality or changes are implemented correctly and at least somewhat efficiently. They must also ensure that important changes are tested against future regressions.
          • +
        • The CAM SE team is almost always engaged in several projects to implement new CAM functionality along with supporting infrastructure. Each CAM SE usually looks at each PR in order to prevent new code from interfering with those plans.
          • +
        +

        The first two steps are usually completed by a single SE although SEs engaged in a final review will often find missed errors. This is similar to peer reviewers finding problems with a paper even after reviews done by colleagues.

        +

        Q: How do I update my branch to the current cam_development?

        +

        A: see this section

        + + + + + + + + + + + + + +
        +
        + + + +
        + +
        + + + +
        +
        +
        +
        + + + + + + + + + + \ No newline at end of file diff --git a/site/development/git-fleximod/index.html b/site/development/git-fleximod/index.html new file mode 100644 index 00000000..fc9fbf20 --- /dev/null +++ b/site/development/git-fleximod/index.html @@ -0,0 +1,959 @@ + + + + + + + + + + + + + + + + + + + + + + + git-fleximod - CAM-SIMA + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
        + +
        + + + + + + +
        + + +
        + +
        + + + + + + +
        +
        + + + +
        +
        +
        + + + + + +
        +
        +
        + + + +
        +
        +
        + + + +
        +
        +
        + + + +
        +
        + + + + +

        git-fleximod

        +

        Populate and/or update your externals

        +

        Run bin/git-fleximod update

        +

        The process will either:

        +
          +
        • complete and you're good to go
          • +
        • indicate to you that modifications have been made in certain modules
            +
          • this means you should either go commit those mods or remove them
            • +
          • you can also (if you're sure!) run bin/git-fleximod update -f to force overwrite all externals
            • +
          +
          • +
        +

        Update external repo or tag

        +

        To add or update an external repo to CAM-SIMA, the following steps must be done:

        +
          +
        1. Modify the .gitmodules file at the head of CAM-SIMA to add or update the external. Explanations for what each entry in the .gitmodules file is can be found on Github here
          2. +
        2. Once the .gitmodules file has been updated, go to the head of CAM-SIMA and run bin/git-fleximod update. This will bring in the new external code into your local repo (but will not commit them).
          3. +
        3. Once you are ready to officially commit the changes, then make sure to commit both the modified .gitmodules file, and the updated submodule itself. An easy way to make sure you have commited everything is to run git status and make sure there are no files or directories that have been modified but are still un-staged.
          4. +
        +

        Once all of that is done then congrats! A new external has been successfully added/updated.

        + + + + + + + + + + + + + +
        +
        + + + +
        + +
        + + + +
        +
        +
        +
        + + + + + + + + + + \ No newline at end of file diff --git a/site/design/ccpp-physics/index.html b/site/development/tool-recommendations/index.html similarity index 62% rename from site/design/ccpp-physics/index.html rename to site/development/tool-recommendations/index.html index 0ae4bc1b..6e29a564 100644 --- a/site/design/ccpp-physics/index.html +++ b/site/development/tool-recommendations/index.html @@ -9,10 +9,10 @@ - + - + @@ -20,7 +20,7 @@ - CCPP Physics - CAM-SIMA + Tool recommendations - CAM-SIMA @@ -82,7 +82,7 @@
        - + Skip to content @@ -118,7 +118,7 @@
        - CCPP Physics + Tool recommendations
        @@ -251,7 +251,7 @@ - Welcome to MkDocs + CAM-SIMA developer documentation @@ -327,19 +327,17 @@ - - -
      • +
      • - + -
      + + +
      10. + + + + + + + + + + + + + + + +
    10. + + + + + + + + + + +
      11. + + + + + + + + + + + + + +
    11. + + + + + + + + + @@ -515,7 +919,18 @@ -

      CCPP Physics

      +

      Tool recommendations

      +

      This page lists recommendations for various tools one might use during CAM-SIMA development.

      +

      Please note that these standards are currently in development, and thus any of them could change at any time.

      +

      Version Control

      +

      The recommended version control tool is git.

      +

      Repository Hosting

      +

      We recommend Github for hosting software repositories

      +

      CI/CD

      +

      When possible, we recommend running any CI/CD workflows using Github Actions.

      +

      Container Software

      +

      We recommend using Docker for building and running containers.

      +

      More to Come!

      @@ -562,7 +977,7 @@

      CCPP Physics

      - + diff --git a/site/index.html b/site/index.html index 7a66bf18..11b6d681 100644 --- a/site/index.html +++ b/site/index.html @@ -80,7 +80,7 @@
      - + Skip to content @@ -116,7 +116,7 @@
      - Welcome to MkDocs + CAM-SIMA developer documentation
      @@ -254,64 +254,16 @@ - - - Welcome to MkDocs + CAM-SIMA developer documentation - - - -
      12. @@ -425,7 +377,7 @@ - CAM-SIMA Build Process + Build process @@ -446,7 +398,7 @@ - CAM-SIMA Run Process + Run process @@ -463,11 +415,11 @@
    12. - + - CCPP Physics + CCPP in CAM-SIMA @@ -509,7 +461,301 @@ - History & Model Output + History & model output + + + + +
      13. + + + + + + + + + + +
    13. + + + + + Design goals & features + + + + +
      14. + + + + + + + + + + + + + + + + + + + + + +
    14. + + + + + + + + + + +
      15. + + + + + + + + + + + + + +
    15. + + + + + + + + + @@ -585,21 +805,7 @@ -

      Welcome to MkDocs

      -

      For full documentation visit mkdocs.org.

      -

      Commands

      -
        -
      • mkdocs new [dir-name] - Create a new project.
        • -
      • mkdocs serve - Start the live-reloading docs server.
        • -
      • mkdocs build - Build the documentation site.
        • -
      • mkdocs -h - Print help message and exit.
        • -
      -

      Project layout

      -
      mkdocs.yml    # The configuration file.
-docs/
-    index.md  # The documentation homepage.
-    ...       # Other markdown pages, images and other files.
-
      +

      CAM-SIMA developer documentation

      @@ -646,7 +852,7 @@

      Project layout

      - + diff --git a/site/search/search_index.json b/site/search/search_index.json index ea722e58..4cdfca9e 100644 --- a/site/search/search_index.json +++ b/site/search/search_index.json @@ -1 +1 @@ -{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Welcome to MkDocs","text":"

      For full documentation visit mkdocs.org.

      "},{"location":"#commands","title":"Commands","text":"
      • mkdocs new [dir-name] - Create a new project.
      • mkdocs serve - Start the live-reloading docs server.
      • mkdocs build - Build the documentation site.
      • mkdocs -h - Print help message and exit.
      "},{"location":"#project-layout","title":"Project layout","text":"
      mkdocs.yml    # The configuration file.\ndocs/\n    index.md  # The documentation homepage.\n    ...       # Other markdown pages, images and other files.\n
      "},{"location":"conversion/ccpp-conversion-guide/","title":"CAM Parameterization to CCPP","text":""},{"location":"conversion/ccpp-conversion-guide/#background","title":"Background","text":""},{"location":"conversion/ccpp-conversion-guide/#running-jobs-in-cam-and-cam-sima-primer-experienced-users-can-jump-to-the-next-section","title":"Running jobs in CAM and CAM-SIMA primer (experienced users can jump to the next section)","text":"

      To make runs in either CAM or CAM-SIMA the commands are identical:

      ./create_newcase\u2026 #(the commands will differ and are specified later in this document)\ncd YourCaseDirectory\n./xmlchange\u2026 #(specific commands are specified later in this document)\n./case.setup\n./case.build\n./case.submit\n

      Depending on which machine you are on, you may prefer to run the ./case.build command on a compute node instead of the login node due to user resource utilization limits on the login nodes.

      For more detailed information on case creation and building, see https://ncar.github.io/CAM/doc/build/html/users_guide/building-and-running-cam.html

      "},{"location":"conversion/ccpp-conversion-guide/#prep-work","title":"Prep Work","text":""},{"location":"conversion/ccpp-conversion-guide/#conversion-spreadsheet","title":"Conversion Spreadsheet","text":"

      Put the parameterization that you are going to convert into the conversion spreadsheet https://docs.google.com/spreadsheets/d/1_1TTpnejam5jfrDqAORCCZtfkNhMRcu7cul37YTr_WM/edit#gid=0

      "},{"location":"conversion/ccpp-conversion-guide/#create-github-issues","title":"Create Github Issues","text":"

      Create a Github Issue in the ESCOMP/CAM repo that states which physics parameterization you are planning to convert to the CCPP framework. Then create another issue in the ESCOMP atmospheric physics repo describing the same physics parameterization that you are now planning to add to the collection of NCAR CCPP physics suites. Doing this allows the software engineers to keep track of which physics routines are being worked on, and which still need to be assigned. The goal of converting the physics parameterization is to ultimately have the CCPP-ized physics package reside in ESCOMP atmospheric physics and be removed from ESCOMP/CAM.

      "},{"location":"conversion/ccpp-conversion-guide/#setting-up-your-sandbox","title":"Setting up your sandbox","text":"

      Make sure you have github forks for both ESCOMP/CAM-SIMA and ESCOMP/atmospheric_physics. If needed see https://github.com/ESCOMP/Cam/wiki/CAM-Development-Workflow-in-GitHub#how-to-makestore-revisions-to-your-personal-cam-repository-github-fork

      To begin, fork ESCOMP/CAM-SIMA: 

      git clone  https://github.com/<GithubUserID>/CAM-SIMA\ncd CAM-SIMA\n\n

      NOTE: As you make changes and want to commit them to your github repos, you\u2019ll be managing two separate repos. When you issue git commands, be aware of where you are in your code tree. If you want to see changes in CAM-SIMA, you can issue a \u201cgit status\u201d in the main CAM-SIMA directory. If you want to see changes in the atmospheric_physics repo, make sure you are in src/physics/ncar_ccpp before you issue the \u201cgit status\u201d command. All other git commands will be relative to your current working directory as well.

      "},{"location":"design/cam-build-process/","title":"CAM-SIMA Build Process","text":"

      In order to describe the build process, we need to define several source and build directories:

      • <srcroot> : The CAM-SIMA sandbox being built. From a case directory, this can be found with ./xmlquery SRCROOT.
      • <caseroot>: The location of the case being built.
      • <src_mods>: The location of the CAM-SIMA source mods, located at <caseroot>/SourceMods/src.cam.
      • <bldroot> : The location of CAM-SIMA generated code and compiled code (.o and .mod). See atm/obj under ./xmlquery OBJROOT.
      "},{"location":"design/cam-build-process/#build-sequence","title":"Build Sequence","text":"

      Given the context above, the following is the CAM-SIMA build sequence:

      1. Create a ConfigCAM object, used to store build configuration information. Code can be found in <srcroot>/cime_config/cam_config.py
        • Inputs:
          • <case>: A CIME case object
          • <case_log>: A python logger, which is usually created by CIME itself
        • Outputs:
          • config_dict: A dictionary of configure options, with the dictionary keys being the name of the config variable/option, and the dictionary values being special config variable objects that contain the following properties:
            • name: The name of the config option/variable. This is what is used as the key in the config_dict dictionary
            • desc: A text description of what this particular config option is, and what sort of values the config option should have.
            • value: The value of that config option/variable. Currently can only be an integer, string, or list of integers or strings.
            • valid_vals: Optional property that contains either a list or range of valid values for this particular config entry
            • valid_type: Optional property for a list-type config option that states what data type the list elements can have.
            • is_nml_attr: Logical that states whether this config option can be used as an XML attribute in a namelist definition file.
        • Additional info:
          • The ConfigCAM object also has various build-in methods that are used to call other steps in the build workflow, change the value of a given config variable, or print extra config info to the provided python logger.
          • The ConfigCAM object currently has no uses outside buildnml and buildlib, and so is not saved after those steps are complete.
      2. Generate CAM-SIMA source code. This sequence has several steps, each of which is performed if any of its inputs has changed.
        • Create or read in a BuildCacheCAM object (<srcroot>/cime_config/cam_build_cache.py).
          • <build_cache>: An optional cache file (created by previous build). This file is created in the case build directory (bld/atm/obj).
        • Create the physics derived data types using the registry (if required).
        • Find all scheme metadata files for configured suites.
        • Find any active schemes with namelist XML variables
          • Create metadata file with namelist variables
          • Create namelist-reading module (to go with metadata file).
        • Call the CCPP Framework to generate glue code (CAPS), (if required).
      "},{"location":"design/cam-build-process/#cam-sima-source-and-namelist-generation-buildnml-workflow","title":"CAM-SIMA source and namelist generation (buildnml) workflow","text":"

      The diagram above displays everything that occurs when CAM-SIMA's buildnml is called by CIME, which occurs after the user calls preview_namelists, case.build, or case.submit.

      All blue boxes represent source code, and are listed as \"source file: function/object used\", and all objects or functions that have a \"+\" are automatically tested whenever a Pull Request is opened, updated, or merged.

      All orange boxes are input files that are used by the code, while all green boxes are output files that are generated. It is important to note that additional files are generated as well, specifically a build cache and a CCPP datatable, but those files are used internally in the build scripts shown here and not by the final model executable, and thus aren't listed here.

      Finally, the arrows show the order of operations, starting with buildnml, with the top two source code boxes representing python classes that are used by the functions/objects directly below them.

      "},{"location":"design/cam-run-process/","title":"CAM-SIMA Run Process","text":""},{"location":"design/cam-run-process/#cam-sima-api","title":"CAM-SIMA API","text":"

      Upon running ./case.submit the core CAM-SIMA driver code* is in $CAM-SIMA/src/control/cam_comp.F90. This section lays out each of the subroutines within cam_comp.F90. The subroutines in cam_comp.F90 are set up to mirror the phases of the Common Community Physics Package (CCPP).

      * cam_comp.F90 subroutines are called by the NUOPC cap: $CAM-SIMA/src/cpl/nuopc/atm_comp_nuopc.F90

      "},{"location":"design/cam-run-process/#cam_init","title":"cam_init","text":"

      cam_init sets up the metadata and configuration objects/modules for the run. It is called by atm_comp_nuopc.F90 at startup. Below are the variables passed in:

      Variable (intent) Definition How it's Used caseid (in) case name (as input to ./create_newcase) metadata for cam_control_mod and cam_history ctitle (in) case title (matches caseid) metadata for cam_control_mod and cam_history model_doi_url (in) CESM model DOI (currently hard-coded to \"non_set\") metadata for cam_history initial_run_in (in) logical that is TRUE if this is a startup run** determines whether to call dyn_init or read restart restart_run_in (in) logical that is TRUE if this is a restart run** metadata for cam_control_mod branch_run_in (in) logical that is TRUE if this is a branch run** metadata for cam_control_mod post_assim_in (in) logical that is TRUE if data assimilation mode (DART) is on metadata for cam_control_mod calendar (in) calendar type (NOLEAP or GREGORIAN) input to time manager init brnch_retain_casename (in) flag to allow a branch to use the same caseid as the run being branched from metadata for cam_control_mod aqua_planet (in) flag to run model in aqua planet mode metadata for cam_control_mod single_column (in) flag to run the single column model (SCAM) passed into scam_readnl (not yet enabled in CAM-SIMA scmlat (in) SCAM latitude passed into scam_readnl (not yet enabled in CAM-SIMA) scmlon (in) SCAM longitude passed into scam_readnl (not yet enabled in CAM-SIMA) eccen (in) Earth's eccentricity factor used to set module-level eccen in cam_control_mod obliqr (in) Earth's obliquity in radians used to set module-level obliqr in cam_control_mod lambm0 (in) Mean longitude of perihelion at the vernal equinox (radians) used to set module-level lambm0 in cam_control_mod mvelpp (in) Earth's moving vernal equinox longitude of perihelion plus pi (radians) used to set module-level mvelpp in cam_control_mod perpetual_run (in) flag to determine if perpetual mode is enabled passed to time manager init perpetual_ymd (in) perpetual year, month, day (YYYYMMDD) used to determine the sun position and interpolate boundary data sets passed to time manager init dtime (in) model timestep size in seconds passed to time manager init start_ymd (in) start date (YYYYMMDD) passed to time manager init start_tod (in) start time of day (sec) passed to time manager init ref_ymd (in) reference date (YYYYMMDD) (defaults to start_ymd) passed to time manager init ref_tod (in) reference time of day (sec) (defaults to start_tod) passed to time manager init stop_ymd (in) stop date (YYYYMMDD) passed to time manager init stop_tod (in) stop time of day (sec) passed to time manager init curr_ymd (in) current date (YYYYMMDD) (same as start date) passed to time manager init curr_tod (in) current time of day (sec) (same as start tod) passed to time manager init cam_in (inout) surface exchange object - coupler to CAM-SIMA allocated if this is an initial run cam_out (inout) surface exchange object - CAM-SIMA to coupler allocated if this is an initial run

      ** For additional information on run types, see the CESM Tutorial

      cam_init calls the following key subroutines (locations) in this order:

      1. cam_ctrl_init (src/control/cam_control_mod.F90): Sets the module-level run configuration variables; logs configurations to the atm log
      2. cam_ctrl_set_orbit (src/control/cam_control_mod.F90): Sets the module-level orbital variables
      3. timemgr_init (src/utils/time_manager.F90`): Initializes the time manager; logs configurations to the atm log
      4. read_namelist (src/control/runtime_opts.F90): Reads all namelists for the run, including auto-generated scheme namelists (see build process)
      5. cam_ctrl_set_physics_type (src/control/cam_control_mod.F90): sets module-level configuration for variables for simple physics and moist physics schemes; logs configurations to atm log
      6. cam_initfiles_open (src/control/cam_initfiles.F90): Opens initial or restart file, and topography file if specified
      7. cam_register_constituents (src/control/cam_comp.F90): Sets the total number and advected number of constituents; currently ALWAYS adds water vapor as constituent (expected by the SE dycore)
      8. air_composition_init (src/data/air_composition.F90): Initializes air-composition-dependent model constants
      9. model_grid_init (src/dynamics/<dycore>/dyn_grid.F90): Initializes model grids and decompositions
      10. cam_ccpp_initialize_constituents ($CASE/bld/atm/obj/ccpp/cam_ccpp_cap.F90): initializes the constituent data array; after this point, we cannot add new constituents
      11. dyn_init (src/dynamics/<dycore>/dyn_comp.F90): Initializes the dynamical core
      12. atm2hub_alloc and hub2atm_alloc (src/control/camsrfexch.F90): Allocate and set up surface exchange data
      13. phys_init (src/physics/utils/phys_comp.F90): Initializes physics (includes call to CCPP cap to run init phases of schemes in the Suite Definition File (SDF)
      14. stepon_init (src/dynamics/<dycore>/stepon.F90): Initializes dynamics <--> physics coupling
      "},{"location":"design/cam-run-process/#cam_timestep_init","title":"cam_timestep_init","text":"

      cam_timestep_init is called at the start of each timestep. It has no input/output/inout variables.

      The routine calls the following subroutines (locations) in this order:

      1. stepon_timestep_init (src/dynamics/<dycore>/stepon.F90): First phase of dynamics (couple from dynamics to physics); also returns timestep for physics
      2. phys_timestep_init (src/physics/utils/phys_comp.F90):
        1. Read data from initial data file (for the null dycore, this means we're reading most physics input variables (as defined in src/data/registry.xml) from the ncdata file; for the SE dycore, we are reading in any variables not marked as initialized by the SE dycore initialization)
        2. Call the CCPP cap to run timestep_init phases of all schemes in the user-specified suite definition file (SDF)
      "},{"location":"design/cam-run-process/#cam_run1","title":"cam_run1","text":"

      cam_run1 is the first \"run\" phase called in the physics loop. It is called BEFORE the mediator/coupler and calls the following subroutine (location):

      1. phys_run1 (src/physics/utils/phys_comp.F90): Calls the run phase for all physics schemes in the \"physics_before_coupler\" group in the suite definition file (SDF)
      "},{"location":"design/cam-run-process/#cam_run2","title":"cam_run2","text":"

      cam_run2 is the second \"run\" phase called in the physics loop. It is called AFTER the mediator/coupler. Input/output variables:

      Variable (intent) Definition How it's Used cam_in (inout) surface exchange object - input to CAM-SIMA Passed into stepon_run2 cam_out (inout) surface exchange object - output from CAM-SIMA Passed into stepon_run2

      cam_run2 calls these subroutines (locations):

      1. phys_run2 (src/physics/utils/phys_comp.F90): Calls the run phase for all physics schemes in the \"physics_after_coupler\" group in the suite definition file (SDF)
      2. stepon_run2 (src/dynamics/<dycore>/stepon.F90): The second phase of dynamics (couple from physics to dynamics)
      "},{"location":"design/cam-run-process/#cam_run3","title":"cam_run3","text":"

      cam_run3 is the third \"run\" phase called in the physics loop. It is called AFTER cam_run3 and BEFORE cam_run4 (unsurprisingly). In/out variables:

      Variable (intent) Definition How it's Used cam_out (inout) surface exchange object - output from CAM-SIMA Passed into stepon_run2

      cam_run3 calls the following subroutine (location):

      1. stepon_run3 (src/dynamics/<dycore>/stepon.F90): Calls dyn_run, which runs the dycore
      "},{"location":"design/cam-run-process/#cam_run4","title":"cam_run4","text":"

      cam_run4 currently does nothing!

      "},{"location":"design/cam-run-process/#cam_timestep_final","title":"cam_timestep_final","text":"

      cam_timestep_final runs at the end of each timestep. In/out variables:

      Variable (intent) Definition How it's Used rstwr (in) flag to write a restart file Passed into history_wrap_up nlend (in) flag to indicate whether this is the final timestep Passed into history_wrap_up

      cam_timestep_final calls the following subroutines (locations):

      1. History routines. If it's not the last (half) timestep,
        1. history_write_files (src/history/cam_history.F90): Writes fields to user-configured history files (if applicable)
        2. history_wrap_up (src/history/cam_history.F90): Closes files and zeros buffers as necessary
      2. phys_timestep_final (src/physics/utils/phys_comp.F90):
        1. Calls the timestep_final phase for all physics schemes in the suite definition file (SDF)
        2. If ncdata_check is set in user_nl_cam, calls physics_check_data ($CASE/bld/atm/obj/phys_init/physics_inputs.F90) to perform snapshot checking
      "},{"location":"design/cam-run-process/#cam_final","title":"cam_final","text":"

      cam_final is called at the end of the model execution. In/out variables:

      Variable (intent) Definition How it's Used cam_in (inout) surface exchange object - input to CAM-SIMA Deallocated cam_out (inout) surface exchange object - output from CAM-SIMA Deallocated

      cam_final calls the following subroutines (locations):

      1. phys_final (src/physics/utils/phys_comp.F90): calls \"final\" phase of all schemes in the suite definition file (SDF)
      2. stepon_final (src/dynamics/<dycore>/stepon.F90): finalize dycore (doesn't currently do anything)
      3. atm2hub_deallocate and hub2atm_deallocate (src/control/camsrfexch.F90): deallocate cam_in/cam_out objects
      "},{"location":"design/ccpp-physics/","title":"CCPP Physics","text":""},{"location":"design/constituents/","title":"Constituents","text":""},{"location":"design/constituents/#introductionoverview","title":"Introduction/Overview","text":"

      Some definitions to start (as written by a non-scientist, so there is more nuance than this!):

      • A constituent is a physical quantity or substance that exists in the atmosphere
      • A constituent can be advected, which means it is moved through the atmosphere by some sort of dynamical method
        • The dynamical core (dycore) is the part of an atmosphere model (like CAM-SIMA) that advects the quantities over the underlying grid. Dynamical cores include:
          • null/none - the null dycore does nothing and is used in CAM-SIMA to validate physics schemes
          • spectral element (SE): the only dycore currently implemented in CAM-SIMA
          • finite-volume cubed-sphere (FV3): not currently implemented in CAM-SIMA
          • model for prediction across scales (MPAS): in progress
      • A constituent had additional properties, such as:
        • water type: can be \"dry\", \"wet\", or \"moist\" (we can convert between any of these quantities - \"dry\" means that it's the \"amount\" of that constituent with respect to dry air
        • mixing ratio type: can be \"volume\" or \"mass\"
          • volume mixing ratio: constituent values are a ratio of how much of the constituent exists per mole of air (units = mol mol-1)
          • mass mixing ratio: constituent values are the mass of the constituent per unit volume (kg m-3)
        • molar mass: Molar mass of a given quantity; used in converting between mass mixing ratio and volume mixing ratio
        • thermodynamically active: thermodynamically active constituents in CAM-SIMA can be found in src/data/air_composition.F90
        • minimum value: The scheme qneg will set constituent values that are less than the minimum to the minimum
      "},{"location":"design/constituents/#cam-sima-constituent-handling","title":"CAM-SIMA constituent handling","text":""},{"location":"design/constituents/#determining-constituents","title":"Determining constituents","text":"

      There are three ways to identify a quantity as a constituent in CAM-SIMA:

      1. Constituent is provided by host (CAM-SIMA):
        • Host model constituents allow for constituents to be added as a constituent independent of the physics being run
        • Host constituents are added in cam_register_constituents (in src/control/cam_comp.F90). Currently, we are always adding water vapor as a host constituent because it is expected by the SE dycore.
      2. Constituent is a build-time physics constituent in a CCPP-ized scheme:
        • If a quantity is known to be a constituent at build-time, it is identified in the metadata for the scheme with: advected = True
      3. Constituent is a run-time physics constituent in a CCPP-ized scheme:
        • Sometimes, a scheme does not know what constituents it will require until run-time. In this case, an array of constituent properties (one for each needed constituent) is returned from the cam register phase. An example of how this works can be found in src/physics/ncar_ccpp/musica/micm/micm.F90
      "},{"location":"design/constituents/#registering-initializing-constituents","title":"Registering & Initializing Constituents","text":"

      The registration and initializaiton of the constituent data array and the constituent properties object are done through calls to the generated CCPP cap.

      • cam_ccpp_register_constituents: combines the three sources of constituents into one ccpp_model_constituents_t object
        • Called before the physics grid (which requires the number of constituents) is initialized
      • cam_ccpp_initialize_constituents: initializes the data array within the ccpp_model_constituents_t object
        • Called after the physics grid is initialized (so we know the size of the array allocate)
        • The array is initialized to (columns, levels, number of constituents)
      "},{"location":"design/constituents/#constituent-usage","title":"Constituent Usage","text":"

      Constituent values and properties can be accessed from the host side and from the physics in the following ways:

      • Host side: constituents and properties can be accessed via the host model and dycore by way of the cam_constituents.F90 module, which is an interface to the CCPP cap, which is in turn an interface to the constituents object
      • Physics: the constituent array and/or the constituent properties object are passed into a scheme via the following metadata (local name and intent may vary):
      [ q ]\n  standard_name = ccpp_constituents\n  units = none\n  type = real | kind = kind_phys\n  dimensions = (horizontal_loop_extent,vertical_layer_dimension,number_of_ccpp_constituents)\n  intent = inout\n[ const_props ]\n  standard_name = ccpp_constituent_properties\n  units = None\n  type = ccpp_constituent_prop_ptr_t\n  dimensions = (number_of_ccpp_constituents)\n  intent = in\n
      "},{"location":"design/constituents/#ccpp-framework-constituent-handling","title":"CCPP Framework constituent handling","text":"

      This section can be removed when constituents are documented in the CCPP Framework documentation.

      "},{"location":"design/constituents/#constituent-object-fortran","title":"Constituent object (Fortran)","text":"

      The constituent object (found in $CAM-SIMA/ccpp_framework/src/ccpp_constituent_prop_ mod.F90) is a flexible and extendable means of containing necessary constituent data for the framework. The primary object is ccpp_model_constituents_t.

      This object, importantly, contains the following properties (for which there is metadata; CCPP standard name in parenthesis):

      • const_metadata (ccpp_constituent_properties)
      • num_layer_vars (number_of_ccpp_constituents)
      • num_advected_vars (number_of_ccpp_advected_constituents)
      • vars_layer (ccpp_constituents)

      The const_metadata property is of type ccpp_constituent_prop_ptr_t, which contains a pointer to a ccpp_constituent_properties_t object, as depicted above. This object contains all of the constituent properties for the constituents in the constituents array, with the same indices as the constituents array.

      The ccpp_model_constituents_t type also contains a hash table of constituent properties for more efficient searching, as well as several methods used by the generated cap code. Some methods are highlighted below:

      • new_field: add a new constituent\u2019s set of metadata fields to the hash table
      • lock_table: lock the constituent hash table and initialize the constituent array
        • Initializes the constituent array to the default value specified for each constituent (min value held in constituent props array for each constituent)
        • Packs the advected constituents at the front of the constituents array
      • const_index: retrieves the constituent index for a constituent (provided the standard name)
      "},{"location":"design/constituents/#code-generation-python","title":"Code generation (python)","text":"

      The constituents-related code generation routines provide an interface to the constituents object. These routines can be found in $CAM-SIMA/ccpp_framework/scripts/constituents.py, primarily within the \u201cwrite_host_routines\u201d function. The (most often used) generated routines related to constituents are:

      • <hostname>_ccpp_register_constituents
      • <hostname>_ccpp_initialize_constituents
      • <hostname>_ccpp_number_constituents
      • <hostname>_constituents_array
      • <hostname>_model_const_properties
      • <hostname>_const_get_index

      The routines above are generated during ./preview_namelists or ./case.build and can be found here: $CASE/bld/atm/obj/ccpp/cam_ccpp_cap.F90

      "},{"location":"design/history/","title":"History & Model Output","text":"

      CAM-SIMA history is the mechanism for configuring and generating diagnostic output from a model run. It is also used to generate initial-data files and aids in the model-restart process by saving the state of diagnostic fields whose processing window (e.g., averaging, standard deviation) crosses a restart-write cycle. This page describes the implementation of CAM-SIMA history in CAM-SIMA.

      "},{"location":"design/history/#history-initialization","title":"History Initialization","text":""},{"location":"design/history/#reading-and-processing-the-history-configuration","title":"Reading and processing the history configuration","text":"
      • The allowable history configuration keywords are defined in cime_config/hist_config.py in _HIST_CONFIG_ENTRY_TYPES.
        • Each of the keywords is configurable via the namelist (user_nl_cam)
        • The syntax is <keyword>;<volume>: <value> (see examples below)
        • Currently, these configuration keywords are as follows (the atm_in equivalent indicates how hist_config.py parses these into a namelist to be read by SIMA):
      Configuration Keyword Description atm_in equivalent hist_add_avg_fields hist_add_inst_fieldshist_add_min_fields hist_add_max_field hist_add_var_field hist_remove_fields These configuration keywords add/remove fields to the specified volume with the indicated accumulation flag (average, instantaneous, minimum, maximum, standard deviation). The closest CAM7 equivalent is \u201cfinclX\u201d &hist_config_arrays_nl\u2002hist_num_avg_fields\u2002hist_num_inst_fields\u2002hist_num_min_fields\u2002hist_num_max_fields\u2002hist_num_var_fields&hist_file_config_nl\u2002hist_avg_fields\u2002hist_inst_fields\u2002hist_min_fields\u2002hist_max_fields\u2002hist_var_fields hist_file_type This keyword determines the type of file. Options are: \u201chistory,\u201d \u201csatellite,\u201d and \u201cinitial_value\u201dDefaults to \u201chistory\u201d &hist_file_config_nl\u2002hist_file_type hist_max_frames Indicates the maximum number of samples/frames that can be written to a file before that file is considered \u201cfull\u201d. The CAM7 equivalent is \u201cmfilt\u201d.Defaults to 1 for h0 and 30 for all other volumes. &hist_file_config_nl\u2002hist_max_frames hist_output_frequency Specifies the frequency of writes to the volume. The syntax is \u201c*\u201d where \u201ctime_period\u201d can be: steps, seconds, minutes, hours, days, months, years. The closest CAM7 equivalent is \u201cnhtfrq\u201d. &hist_file_config_nl\u2002hist_output_frequency hist_precision Denotes the precision for the volume. Options are \"REAL32\" and \"REAL64\".Defaults to \"REAL32\" &hist_file_config_nl\u2002hist_precision hist_write_nstep0 Specifies the template for the filename for the volume. Defaults to \"%c.cam.%u.%y-%m-%d-%s.nc\" where \"%c\" is the case name, \"%u\" is the volume, \"%y\" is the year, \"%m\" is the month, \"%d\" is the day, and \"%s\" is the seconds &hist_file_config_nl\u2002hist_filename_spec
      • hist_config.py also contains the HistoryVolConfig class (all the info pertaining to a single history file), the HistoryConfig class (all the history configuration information including a dict of HistoryVolConfig objects), and helper classes.
      • The HistoryConfig object is created in buildnml out of entries in user_nl_cam and written to run/atm_in.
      • In order to ensure that all relevant runtime (namelist) values show up in atm_in, the HistoryConfig object must contain all the logic in setting default values.

      Example Take the following sample user_nl_cam:

      hist_output_frequency;h1: 5*ndays\nhist_max_frames;h1: 3\nhist_add_inst_fields;h1: U\nhist_add_inst_fields;h1: V, Q\nhist_precision;h1: REAL64\nhist_filename_spec;h1: my-history-file%m-%d\nhist_write_nstep0;h1: .false.\n

      It will be parsed by hist_config.py and this will be the relevant section of atm_in:

      &hist_config_arrays_nl\n    hist_num_inst_fields = 3\n    hist_num_avg_fields = 2\n    hist_num_min_fields = 0\n    hist_num_max_fields = 0\n    hist_num_var_fields = 0\n/\n\n&hist_file_config_nl\n    hist_volume = 'h0'\n    hist_avg_fields = 'T', 'Q'\n    hist_max_frames = 1\n    hist_output_frequency = '1*month'\n    hist_precision = 'REAL32'\n    hist_file_type = 'history'\n    hist_filename_spec = '%c.cam.%u.%y-%m-%d-%s.nc'\n    hist_write_nstep0 = .false.\n/\n\n&hist_file_config_nl\n    hist_volume = 'h1'\n    hist_inst_fields = 'U', \u2018V\u2019, \u2018Q\u2019\n    hist_max_frames = 3\n    hist_output_frequency = '5*ndays'\n    hist_precision = 'REAL64'\n    hist_file_type = 'history'\n    hist_filename_spec = 'my-history-file%m-%d'\n    hist_write_nstep0 = .false.\n/\n

      In plain English, a one-month run with these history configuration will result in a total of three files that will look something like these: - my-history-file01-06.nc - This file will contain instantaneous output for U, V, and Q (eastward_wind, northward_wind, and water vapor) - It will contain three frames, one at each of the following times: - 0001-01-06 (time=5) - 0001-01-11 (time=10) - 0001-01-16 (time=15) - my-history-file01-21.nc - This file will contain instantaneous output for U, V, and Q (eastward_wind, northward_wind, and water vapor) - It will contain three frames, one at each of the following times: - 0001-01-21 (time=20) - 0001-01-26 (time=25) - 0001-01-31 (time=30) - .cam.h0a.0001-02-01-00000.nc - This file will contain averaged output for T and Q (air_temperature and water vapor) - It will have one frame with the time calculated at the midpoint of the month"},{"location":"design/history/#setting-up-the-history-data-structures","title":"Setting up the history data structures","text":"

      • History namelist information is read and history data structures are set up in src/history/cam_hist_file.F90
      • The driving function in cam_hist_file.F90 is hist_read_namelist_config, which is called by history_readnl in src/history/cam_history.F90. This function reads in the hist_config_arrays_nl namelist group, allocates the history field arrays, and then uses those arrays to read in the hist_file_config_nl namelist group (via a call to read_namelist_entry).
      • The history configuration namelist options are used to populate cam_history.F90\u2019s module-level hist_configs array of hist_file_t objects (the size of this array is the number of user-configured volumes).
      • The hist_file_t object contains information about the configuration options for a given history volume. This includes the maximum number of frames that can be written to the file, the current number of frames on the file, the name of the file, and all of the history fields to be written to the file. It also contains methods to populate the field lists (config_set_up_fields), set up the metadata of the file (config_define_file), and write history fields to the file (config_write_time_dependent_variables).
      • Each hist_file_t object contains both a hash table and allocatable field list to keep track of the fields written to the file. The core class for each of these is the hist_field_info_t (in src/history/buffers/src/hist_field.F90), which contains information about a history field. This includes the field names, the accumulate flag (average, instantaneous, minimum, etc), units, type, dimensions, and fill value. It also includes the buffer(s) (type is/are hist_buffer_t) that will and do hold the actual data.
      "},{"location":"design/history/#populating-the-possible-field-list","title":"Populating the possible field list","text":"

      The possible fields to be output by the history infrastructure are tracked in cam_history.F90 via the possible_field_list hash table. It is populated during init time by calls to the subroutine history_add_field (found in src/history/cam_history.F90). \u201cInit time,\u201d means that all calls to history_add_field must occur during the execution of cam_init (found in src/control/cam_comp.F90).

      • Within the CCPP physics, this means that any diagnostic fields must be added to possible field list during _init (ideally the diagnostics are kept separate from the core scheme physics to keep the physics portable)
      • Within dynamics, the fields must be added during dyn_init or stepon_init
      • For physics variables:
        • State variables are added to the field list in the init phase of the cam_state_diagnostics scheme in src/physics/ncar_ccpp/diagnostics/cam_diagnostics.F90
          • This scheme is included at the end of the physics_before_coupler group in the suite definition file (SDF)
        • Tendency variables are added to the field list in the init phase of the cam_tend_diagnostics scheme in src/physics/ncar_ccpp/diagnotics/cam_diagnostics.F90
          • This scheme is included at the end of the physics_after_coupler group in the SDF
        • Additional variables specific to the scheme are included in the init phase of a _diagnostics scheme in src/physics/ncar_ccpp/diagnostics

          Each call to history_add_field adds a new field to the end of the possible_field_list_head linked list. At the end of cam_init, the possible field list linked list is used to print the list (to the atm.log* file) and then is converted to the possible_field_list hash table. A sample of the history field list is seen below.

            ***************** HISTORY FIELD LIST ******************\n             T          K  avg  air_temperature\n            ZM          m  avg  geopotential_height_wrt_surface\n          PHIS     m2 s-2  ins   surface_geopotential\n          PMID         Pa  avg  air_pressure\n       PDELDRY         Pa  avg  air_pressure_thickness_of_dry_air\n             Q    kg kg-1  avg  water_vapor_mixing_ratio_wrt_moist_air_and_condensed_water\n        CLDLIQ    kg kg-1  avg cloud_liquid_water_mixing_ratio_wrt_moist_air_and_condensed_water          \n        RAINQM    kg kg-1  avg  rain_mixing_ratio_wrt_moist_air_and_condensed_water\n         TTEND      K s-1  avg  tendency_of_air_temperature_due_to_model_physics\n  *************** END HISTORY FIELD LIST ****************\n
          "},{"location":"design/history/#capturing-history-output","title":"Capturing history output","text":"

          Outside of CAM-SIMA init and finalize time, history buffers can be populated with data via a call to history_out_field (found in src/history/cam_history.F90)

          The subroutine history_out_field iterates over the hist_configs array and populates the buffer(s) of the hist_field_info_t object of the hist_file_t object if the field name in the call is active on that file (e.g. the field was configured via the namelist to be output for that volume).

          • Within the CCPP physics, calls to history_out_field can exist anywhere except _init and _final
          • In dynamics, calls to history_out_field can exist anywhere except in dyn_init, stepon_init, and stepon_final
          • For physics variables, history_out_field calls are included in the run phase of the same schemes described in the section above.
          "},{"location":"design/history/#defining-new-history-files","title":"Defining new history files","text":"

          The cam_history.F90 subroutine history_write_files (which is called during cam_timestep_final) does three main actions for each of the user-defined history volumes:

          1. Determine if it\u2019s time to write to the file based on the user-supplied namelist option hist_output_frequency
          2. If it's time to write, determine if we need to open a new file
          3. Write the variables to the file(s)

          The bolded step #2 above is what determines if we need to define a new history file. The situations where we would need to define a new file are:

          1. It's the first sample we're outputting to a specific volume.
          2. The last time we output to the volume, we \"filled\" the file (number of samples written to the file met the user-configured hist_max_frames) and closed it

          We determine if it's time for a new file with the following line of code:

             mod(num_samples, hist_configs(file_idx)%max_frame()) == 0\n

          If it is indeed time to define a new file, we call the config_define_file subroutine (which is found in src/history/cam_hist_file.F90) for the volume:

            call hist_configs(file_idx)%define_file(restart, logname, host, model_doi_url)\n
          • This subroutine opens the netcdf file(s) and writes the necessary time-independent metadata (for the file and for the history fields) and grid information.
            • Note that there will be a maximum of TWO (2) files opened at this stage. If there are both instantaneous fields AND accumulated fields (average, minimum, maximum, standard deviation) on the volume, two files will be created (hXi AND hXa); otherwise, only the relevant file will be created (hXi OR hXa)
          "},{"location":"design/history/#writing-a-history-file","title":"Writing a history file","text":"

          The cam_history.F90 subroutine history_write_files (which is called during cam_timestep_final) does three main actions for each of the user-defined history volumes:

          1. Determine if it\u2019s time to write to the file based on the user-supplied namelist option hist_output_frequency
          2. If it's time to write, determine if we need to open a new file
          3. Write the variables to the file(s)

          The bolded step #3 above occurs any time the criteria for #1 is satisfied. At this point, the following call is made to write the history fields (whose data has been stored in their buffers via calls to history_out_field):

            call hist_configs(file_idx)%write_time_dependent_variables(file_idx, restart)\n

          It is during this call that we increment the number of samples written for this volume and actual write the data held within the buffer(s) to the netcdf file (as well as the time-dependent metadata for the fields).

          "},{"location":"design/history/#defining-history-restart-files","title":"Defining history restart files","text":"

          Restarts not yet implemented in CAM-SIMA

          "},{"location":"design/history/#writing-a-history-restart-file","title":"Writing a history restart file","text":"

          Restarts not yet implemented in CAM-SIMA

          "},{"location":"design/history/#reading-a-history-restart-file","title":"Reading a history restart file","text":"

          Restarts not yet implemented in CAM-SIMA

          "}]} \ No newline at end of file +{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"CAM-SIMA developer documentation","text":""},{"location":"conversion/ccpp-conversion-guide/","title":"CAM Parameterization to CCPP","text":""},{"location":"conversion/ccpp-conversion-guide/#background","title":"Background","text":""},{"location":"conversion/ccpp-conversion-guide/#running-jobs-in-cam-and-cam-sima-primer-experienced-users-can-jump-to-the-next-section","title":"Running jobs in CAM and CAM-SIMA primer (experienced users can jump to the next section)","text":"

          To make runs in either CAM or CAM-SIMA the commands are identical:

          ./create_newcase\u2026 #(the commands will differ and are specified later in this document)\ncd YourCaseDirectory\n./xmlchange\u2026 #(specific commands are specified later in this document)\n./case.setup\n./case.build\n./case.submit\n

          Depending on which machine you are on, you may prefer to run the ./case.build command on a compute node instead of the login node due to user resource utilization limits on the login nodes.

          For more detailed information on case creation and building, see https://ncar.github.io/CAM/doc/build/html/users_guide/building-and-running-cam.html

          "},{"location":"conversion/ccpp-conversion-guide/#prep-work","title":"Prep Work","text":""},{"location":"conversion/ccpp-conversion-guide/#conversion-spreadsheet","title":"Conversion Spreadsheet","text":"

          Put the parameterization that you are going to convert into the conversion spreadsheet https://docs.google.com/spreadsheets/d/1_1TTpnejam5jfrDqAORCCZtfkNhMRcu7cul37YTr_WM/edit#gid=0

          "},{"location":"conversion/ccpp-conversion-guide/#create-github-issues","title":"Create Github Issues","text":"

          Create a Github Issue in the ESCOMP/CAM repo that states which physics parameterization you are planning to convert to the CCPP framework. Then create another issue in the ESCOMP atmospheric physics repo describing the same physics parameterization that you are now planning to add to the collection of NCAR CCPP physics suites. Doing this allows the software engineers to keep track of which physics routines are being worked on, and which still need to be assigned. The goal of converting the physics parameterization is to ultimately have the CCPP-ized physics package reside in ESCOMP atmospheric physics and be removed from ESCOMP/CAM.

          "},{"location":"conversion/ccpp-conversion-guide/#setting-up-your-sandbox","title":"Setting up your sandbox","text":"

          Make sure you have github forks for both ESCOMP/CAM-SIMA and ESCOMP/atmospheric_physics. If needed see https://github.com/ESCOMP/Cam/wiki/CAM-Development-Workflow-in-GitHub#how-to-makestore-revisions-to-your-personal-cam-repository-github-fork

          To begin, fork ESCOMP/CAM-SIMA:

          And select the Create new fork option. This will bring you to the \"Create new fork\" screen:

          !!! note Uncheck the \"Copy the main branch only\" option

          Failure to uncheck this will prevent you from pulling in updates from the development branch easily.

          As you make changes and want to commit them to your github repos, you will be managing two separate repos. When you issue git commands, be aware of where you are in your code tree. If you want to see changes in CAM-SIMA, you can issue a git status in the main CAM-SIMA directory. If you want to see changes in the atmospheric_physics repo, make sure you are in src/physics/ncar_ccpp before you issue the git status command. All other git commands will be relative to your current working directory as well.

          "},{"location":"design/cam-build-process/","title":"Build process","text":"

          In order to describe the build process, we need to define several source and build directories:

          • <srcroot> : The CAM-SIMA sandbox being built. From a case directory, this can be found with ./xmlquery SRCROOT.
          • <caseroot>: The location of the case being built.
          • <src_mods>: The location of the CAM-SIMA source mods, located at <caseroot>/SourceMods/src.cam.
          • <bldroot> : The location of CAM-SIMA generated code and compiled code (.o and .mod). See atm/obj under ./xmlquery OBJROOT.
          "},{"location":"design/cam-build-process/#build-sequence","title":"Build Sequence","text":"

          Given the context above, the following is the CAM-SIMA build sequence:

          1. Create a ConfigCAM object, used to store build configuration information. Code can be found in <srcroot>/cime_config/cam_config.py
            • Inputs:
              • <case>: A CIME case object
              • <case_log>: A python logger, which is usually created by CIME itself
            • Outputs:
              • config_dict: A dictionary of configure options, with the dictionary keys being the name of the config variable/option, and the dictionary values being special config variable objects that contain the following properties:
                • name: The name of the config option/variable. This is what is used as the key in the config_dict dictionary
                • desc: A text description of what this particular config option is, and what sort of values the config option should have.
                • value: The value of that config option/variable. Currently can only be an integer, string, or list of integers or strings.
                • valid_vals: Optional property that contains either a list or range of valid values for this particular config entry
                • valid_type: Optional property for a list-type config option that states what data type the list elements can have.
                • is_nml_attr: Logical that states whether this config option can be used as an XML attribute in a namelist definition file.
            • Additional info:
              • The ConfigCAM object also has various build-in methods that are used to call other steps in the build workflow, change the value of a given config variable, or print extra config info to the provided python logger.
              • The ConfigCAM object currently has no uses outside buildnml and buildlib, and so is not saved after those steps are complete.
          2. Generate CAM-SIMA source code. This sequence has several steps, each of which is performed if any of its inputs has changed.
            • Create or read in a BuildCacheCAM object (<srcroot>/cime_config/cam_build_cache.py).
              • <build_cache>: An optional cache file (created by previous build). This file is created in the case build directory (bld/atm/obj).
            • Create the physics derived data types using the registry (if required).
            • Find all scheme metadata files for configured suites.
            • Find any active schemes with namelist XML variables
              • Create metadata file with namelist variables
              • Create namelist-reading module (to go with metadata file).
            • Call the CCPP Framework to generate glue code (CAPS), (if required).
          "},{"location":"design/cam-build-process/#cam-sima-source-and-namelist-generation-buildnml-workflow","title":"CAM-SIMA source and namelist generation (buildnml) workflow","text":"

          The diagram above displays everything that occurs when CAM-SIMA's buildnml is called by CIME, which occurs after the user calls preview_namelists, case.build, or case.submit.

          All blue boxes represent source code, and are listed as \"source file: function/object used\", and all objects or functions that have a \"+\" are automatically tested whenever a Pull Request is opened, updated, or merged.

          All orange boxes are input files that are used by the code, while all green boxes are output files that are generated. It is important to note that additional files are generated as well, specifically a build cache and a CCPP datatable, but those files are used internally in the build scripts shown here and not by the final model executable, and thus aren't listed here.

          Finally, the arrows show the order of operations, starting with buildnml, with the top two source code boxes representing python classes that are used by the functions/objects directly below them.

          "},{"location":"design/cam-run-process/","title":"Run process","text":"CAM-SIMA run sequence*

          *Static images can be found at the bottom of this page

          "},{"location":"design/cam-run-process/#cam-sima-api","title":"CAM-SIMA API","text":"

          Upon running ./case.submit the core CAM-SIMA driver code* is in $CAM-SIMA/src/control/cam_comp.F90. This section lays out each of the subroutines within cam_comp.F90. The subroutines in cam_comp.F90 are set up to mirror the phases of the Common Community Physics Package (CCPP).

          * cam_comp.F90 subroutines are called by the NUOPC cap: $CAM-SIMA/src/cpl/nuopc/atm_comp_nuopc.F90

          "},{"location":"design/cam-run-process/#cam_init","title":"cam_init","text":"

          cam_init sets up the metadata and configuration objects/modules for the run. It is called once at startup. Below are the variables passed in:

          Variable (intent) Definition How it's Used caseid (in) case name (as input to ./create_newcase) metadata for cam_control_mod and cam_history ctitle (in) case title (matches caseid) metadata for cam_control_mod and cam_history model_doi_url (in) CESM model DOI (currently hard-coded to \"non_set\") metadata for cam_history initial_run_in (in) logical that is TRUE if this is a startup run** determines whether to call dyn_init or read restart restart_run_in (in) logical that is TRUE if this is a restart run** metadata for cam_control_mod branch_run_in (in) logical that is TRUE if this is a branch run** metadata for cam_control_mod post_assim_in (in) logical that is TRUE if data assimilation mode (DART) is on metadata for cam_control_mod calendar (in) calendar type (NOLEAP or GREGORIAN) input to time manager init brnch_retain_casename (in) flag to allow a branch to use the same caseid as the run being branched from metadata for cam_control_mod aqua_planet (in) flag to run model in aqua planet mode metadata for cam_control_mod single_column (in) flag to run the single column model (SCAM) passed into scam_readnl (not yet enabled in CAM-SIMA scmlat (in) SCAM latitude passed into scam_readnl (not yet enabled in CAM-SIMA) scmlon (in) SCAM longitude passed into scam_readnl (not yet enabled in CAM-SIMA) eccen (in) Earth's eccentricity factor used to set module-level eccen in cam_control_mod obliqr (in) Earth's obliquity in radians used to set module-level obliqr in cam_control_mod lambm0 (in) Mean longitude of perihelion at the vernal equinox (radians) used to set module-level lambm0 in cam_control_mod mvelpp (in) Earth's moving vernal equinox longitude of perihelion plus pi (radians) used to set module-level mvelpp in cam_control_mod perpetual_run (in) flag to determine if perpetual mode is enabled passed to time manager init perpetual_ymd (in) perpetual year, month, day (YYYYMMDD) used to determine the sun position and interpolate boundary data sets passed to time manager init dtime (in) model timestep size in seconds passed to time manager init start_ymd (in) start date (YYYYMMDD) passed to time manager init start_tod (in) start time of day (sec) passed to time manager init ref_ymd (in) reference date (YYYYMMDD) (defaults to start_ymd) passed to time manager init ref_tod (in) reference time of day (sec) (defaults to start_tod) passed to time manager init stop_ymd (in) stop date (YYYYMMDD) passed to time manager init stop_tod (in) stop time of day (sec) passed to time manager init curr_ymd (in) current date (YYYYMMDD) (same as start date) passed to time manager init curr_tod (in) current time of day (sec) (same as start tod) passed to time manager init cam_in (inout) surface exchange object - coupler to CAM-SIMA allocated if this is an initial run cam_out (inout) surface exchange object - CAM-SIMA to coupler allocated if this is an initial run

          ** For additional information on run types, see the CESM Tutorial

          cam_init calls the following key subroutines (locations) in this order:

          1. cam_ctrl_init (src/control/cam_control_mod.F90): Sets the module-level run configuration variables; logs configurations to the atm log
          2. cam_ctrl_set_orbit (src/control/cam_control_mod.F90): Sets the module-level orbital variables
          3. timemgr_init (src/utils/time_manager.F90`): Initializes the time manager; logs configurations to the atm log
          4. read_namelist (src/control/runtime_opts.F90): Reads all namelists for the run, including auto-generated scheme namelists (see build process)
          5. cam_ctrl_set_physics_type (src/control/cam_control_mod.F90): sets module-level configuration for variables for simple physics and moist physics schemes; logs configurations to atm log
          6. cam_initfiles_open (src/control/cam_initfiles.F90): Opens initial or restart file, and topography file if specified
          7. cam_register_constituents (src/control/cam_comp.F90): Sets the total number and advected number of constituents; currently ALWAYS adds water vapor as constituent (expected by the SE dycore)
          8. air_composition_init (src/data/air_composition.F90): Initializes air-composition-dependent model constants
          9. model_grid_init (src/dynamics/<dycore>/dyn_grid.F90): Initializes model grids and decompositions
          10. cam_ccpp_initialize_constituents ($CASE/bld/atm/obj/ccpp/cam_ccpp_cap.F90): initializes the constituent data array; after this point, we cannot add new constituents
          11. dyn_init (src/dynamics/<dycore>/dyn_comp.F90): Initializes the dynamical core
          12. atm2hub_alloc and hub2atm_alloc (src/control/camsrfexch.F90): Allocates and sets up surface exchange data
          13. phys_init (src/physics/utils/phys_comp.F90): Initializes physics (includes call to CCPP cap to run init phases of schemes in the Suite Definition File (SDF)
          14. stepon_init (src/dynamics/<dycore>/stepon.F90): Initializes dynamics <--> physics coupling
          "},{"location":"design/cam-run-process/#cam_timestep_init","title":"cam_timestep_init","text":"

          cam_timestep_init is called at the start of each timestep. It has no input/output/inout variables.

          The routine calls the following subroutines (locations) in this order:

          1. stepon_timestep_init (src/dynamics/<dycore>/stepon.F90): First phase of dynamics (couple from dynamics to physics); also returns timestep for physics
          2. phys_timestep_init (src/physics/utils/phys_comp.F90):
            1. Read un-initialized data from initial data file
              • For the null dycore, this means we're reading most physics input variables (as defined in src/data/registry.xml) from the ncdata file
              • For the SE dycore, we are reading in any variables not marked as initialized by the SE dycore initialization
            2. Calls the CCPP cap to run timestep_init phases of all schemes in the user-specified SDF
          "},{"location":"design/cam-run-process/#cam_run1","title":"cam_run1","text":"

          cam_run1 is the first \"run\" phase called in the physics loop. It is called every timestep BEFORE the mediator/surface coupler and calls the following subroutine (location):

          1. phys_run1 (src/physics/utils/phys_comp.F90): Calls the run phase for all physics schemes in the \"physics_before_coupler\" group in the SDF
          "},{"location":"design/cam-run-process/#cam_run2","title":"cam_run2","text":"

          cam_run2 is the second \"run\" phase called in the physics loop. It is called every timestep AFTER the mediator/coupler. Input/output variables:

          Variable (intent) Definition How it's Used cam_in (inout) surface exchange object - input to CAM-SIMA Passed into stepon_run2 cam_out (inout) surface exchange object - output from CAM-SIMA Passed into stepon_run2

          cam_run2 calls these subroutines (locations):

          1. phys_run2 (src/physics/utils/phys_comp.F90): Calls the run phase for all physics schemes in the \"physics_after_coupler\" group in the SDF
          2. stepon_run2 (src/dynamics/<dycore>/stepon.F90): The second phase of dynamics (couple from physics to dynamics)
          "},{"location":"design/cam-run-process/#cam_run3","title":"cam_run3","text":"

          cam_run3 is the third \"run\" phase called in the physics loop. It is called every timestep AFTER cam_run3 and BEFORE cam_run4 (unsurprisingly). In/out variables:

          Variable (intent) Definition How it's Used cam_out (inout) surface exchange object - output from CAM-SIMA Passed into stepon_run3

          cam_run3 calls the following subroutine (location):

          1. stepon_run3 (src/dynamics/<dycore>/stepon.F90): Calls dyn_run, which runs the dycore
          "},{"location":"design/cam-run-process/#cam_run4","title":"cam_run4","text":"

          cam_run4 currently does nothing! (but it is called every timestep)

          "},{"location":"design/cam-run-process/#cam_timestep_final","title":"cam_timestep_final","text":"

          cam_timestep_final runs at the end of each timestep. In/out variables:

          Variable (intent) Definition How it's Used rstwr (in) flag to write a restart file Passed into history_wrap_up nlend (in) flag to indicate whether this is the final timestep Passed into history_wrap_up

          cam_timestep_final calls the following subroutines (locations):

          1. History routines. If it's not the last (half) timestep,
            1. history_write_files (src/history/cam_history.F90): Writes fields to user-configured history files (if applicable)
            2. history_wrap_up (src/history/cam_history.F90): Closes files and zeros buffers as necessary
          2. phys_timestep_final (src/physics/utils/phys_comp.F90):
            1. Calls the timestep_final phase for all physics schemes in the SDF
            2. If ncdata_check is set in user_nl_cam, calls physics_check_data ($CASE/bld/atm/obj/phys_init/physics_inputs.F90) to perform snapshot checking
          "},{"location":"design/cam-run-process/#cam_final","title":"cam_final","text":"

          cam_final is called once at the end of the model execution. In/out variables:

          Variable (intent) Definition How it's Used cam_in (inout) surface exchange object - input to CAM-SIMA Deallocated cam_out (inout) surface exchange object - output from CAM-SIMA Deallocated

          cam_final calls the following subroutines (locations):

          1. phys_final (src/physics/utils/phys_comp.F90): calls \"final\" phase of all schemes in the SDF
          2. stepon_final (src/dynamics/<dycore>/stepon.F90): finalizes dycore (doesn't currently do anything)
          3. atm2hub_deallocate and hub2atm_deallocate (src/control/camsrfexch.F90): deallocate cam_in/cam_out objects
          "},{"location":"design/cam-run-process/#static-run-sequence-images","title":"Static run sequence images","text":""},{"location":"design/ccpp-in-cam-sima/","title":"CCPP in CAM-SIMA","text":""},{"location":"design/ccpp-in-cam-sima/#overview","title":"Overview","text":"

          The core Common Community Physics Package (CCPP) documentation can be found here. This section details the code structure and implementation of the CCPP Framework within CAM-SIMA. That said here's a quick overview of the CCPP:

          • A CCPP-enabled model consists of the following components (depicted in the diagram below):
            • Host model and dynamical core
            • CCPP-compliant physics schemes (organized into Suite Definition Files [SDFs])
            • Caps generated by the CCPP framework to connect the host model to the physics schemes

          CCPP-compliant physics schemes must adhere to the following criteria:

          • Must be broken up into only the following phases:
            • register (anything in the scheme that must be done before the grid is initialized) - run once at startup
            • init - run once at startup
            • timestep_init - run at beginning of every physics timestep
            • run - run on every timestep
            • timestep_final - run at the end of every timestep
            • final - run once at the end of model execution
          • Must contain metadata for all input/output/inout variables passed into each phase (see metadata example below)
          • Must not have use statements outside of the following allowed CCPP use statements (with the exception of dependencies, but we're not getting into that now):
            • ccpp_kinds
            • ccpp_constituent_prop_mod

          Metadata example (snippet taken from kessler.meta)

          [ precl ]\n  standard_name = total_precipitation_rate_at_surface\n  long_name = Total precipitation rate at surface\n  units = m s-1\n  dimensions = (horizontal_loop_extent)\n  type = real | kind = kind_phys\n  intent = out\n[ relhum ]\n  standard_name = relative_humidity\n  long_name = Relative humidity\n  units = percent\n  dimensions = (horizontal_loop_extent, vertical_layer_dimension)\n  type = real | kind = kind_phys\n  intent = out\n[ scheme_name ]\n  standard_name = scheme_name\n  units = none\n  type = character | kind = len=64\n  dimensions = ()\n  intent = out\n\n

          CCPP-compliant physics schemes are organized into suite definition files (SDFs). An SDF tells the framework which schemes will be run in what order. Separating schemes into \"groups\" also allows the run phases of those groups to be called separately by the host model. Here's an example SDF (from suite_kessler.xml):

          <?xml version=\"1.0\" encoding=\"UTF-8\"?>\n\n<suite name=\"kessler\" version=\"1.0\">\n  <group name=\"physics_before_coupler\">\n    <scheme>calc_exner</scheme>\n    <scheme>temp_to_potential_temp</scheme>\n    <scheme>calc_dry_air_ideal_gas_density</scheme>\n    <scheme>wet_to_dry_water_vapor</scheme>\n    <scheme>wet_to_dry_cloud_liquid_water</scheme>\n    <scheme>wet_to_dry_rain</scheme>\n    <scheme>kessler</scheme>\n    <scheme>potential_temp_to_temp</scheme>\n    <scheme>dry_to_wet_water_vapor</scheme>\n    <scheme>dry_to_wet_cloud_liquid_water</scheme>\n    <scheme>dry_to_wet_rain</scheme>\n    <scheme>kessler_update</scheme>\n    <scheme>qneg</scheme>\n    <scheme>geopotential_temp</scheme>\n    <scheme>cam_state_diagnostics</scheme>\n    <scheme>kessler_diagnostics</scheme>\n  </group>\n  <group name=\"physics_after_coupler\">\n    <scheme>cam_tend_diagnostics</scheme>\n  </group>\n</suite>\n

          The framework code is primarily python code that generates Fortran caps. The class structure looks like:

          Given CCPP-compliant physics schemes and one or more SDF, the framework generates caps for the host model to call at the appropriate time. The core files generated by the framework are:

          • <host>_ccpp_cap.F90: contains the interface layer between the host and the suite(s)
          • ccpp_<suite>_cap.F90: contains one subroutine per phase (including one run phase per group) in which the phases of the schemes within the suite are called in order
            • There is one suite cap generated for each suite being run
          • ccpp_datatable.xml: consolidates metadata into an XML file to be used as desired by the host model
          "},{"location":"design/ccpp-in-cam-sima/#code-structure","title":"Code Structure","text":"

          How CAM-SIMA and the CCPP come together:

          "},{"location":"design/ccpp-in-cam-sima/#host-model","title":"Host Model","text":"

          The core host model code is what is held in the CAM-SIMA github repository, plus code that is generated at build-time (or preview_namelists-time) based on the registry (src/data/registry.xml).

          "},{"location":"design/ccpp-in-cam-sima/#physics","title":"Physics","text":"

          The CCPP physics scheme code exists in the atmospheric_physics repository, which exists as a submodule of CAM-SIMA in the following location: $CAM-SIMA/src/physics/ncar_ccpp

          SDFs are located in the root directory of the repository and scheme source code is in the relevant subdirectories.

          The diagnostics directory contains all diagnostic schemes (the global ones used for state and tendency output, as well as the scheme-specific diagnostic schemes).

          The to_be_ccppized directory contains physics schemes and utilties that have not been CCPP-ized, but were needed by an CCPP-ized scheme.

          The utilities directory contains schemes that are used regularly, such as tendency applicators and state converters. See below for more.

          "},{"location":"design/ccpp-in-cam-sima/#generated-caps","title":"Generated caps","text":"

          The caps generated by the CCPP Framework at model build time (or preview_namelists-time) can be found in the following location: $CASE/bld/atm/obj/ccpp/

          "},{"location":"design/ccpp-in-cam-sima/#implementation","title":"Implementation","text":"

          All CCPP phases are called from the physics driver (src/physics/utils/phys_comp.F90). You can see the order of these calls more thoroughly in the documented run sequence.

          "},{"location":"design/ccpp-in-cam-sima/#host-side-variables-and-metadata","title":"Host-side variables and metadata","text":"

          For a CCPP-ized physics scheme to work, the framework needs to be able to find a matching variable on the host side for each input variable for the suite(s) in question. This means that CAM-SIMA needs to allocate, initialize, and provide metadata for these variables. We do this in two ways:

          • Adding metadata to existing CAM modules (such as src/data/physconst.F90) so that a scheme can access an existing host model variable
            • All static host-side metadata schemes are included at the top of the registry
          • Adding a variable to the registry (src/data/registry.xml)
            • You can learn more about how the registry is used to generate a Fortran module and corresponding metadata here

          The registry-based code generator is run before the CCPP framework does its magic, so, when it's time, the framework can connect the dots between the host model and the physics.

          "},{"location":"design/ccpp-in-cam-sima/#state-and-tendency-variables","title":"State and tendency variables","text":"

          Two of the most commonly used and referred-to objects in CAM-SIMA are:

          • physics_state
          • physics_tend

          The Fortran for both objects is auto-generated by CAM-SIMA based on the registry ($CAM-SIMA/src/data/registry.xml). The generated code can be found here: $CASE/bld/atm/obj/cam_registry/physics_types.F90. The objects are used by the host model (CAM-SIMA), while the CCPP physics take the individual component variables as inputs.

          "},{"location":"design/ccpp-in-cam-sima/#physics-state","title":"Physics state","text":"

          The physics_state object in CAM-SIMA contains the current values for a select set of variables that describe the atmosphere, at the resolution specified by the input grid.

          Some examples of these core \"state\" variables include temperature (T), eastward wind (U), and northward wind (V)

          As a rule, CAM-SIMA physics schemes do not update the state directly and instead return tendencies (see below) which are then applied to the state later in the run phase. This is called time splitting, which means that all physics since the last state update get the same input state. The alternative, process splitting, means that the output state of one scheme serves as the input state of the next.

          NOTE: Although constituents are handled independently of the physics_state object (they are handled by the CCPP framework), they ARE considered state variables.

          "},{"location":"design/ccpp-in-cam-sima/#physics-tendencies","title":"Physics tendencies","text":"

          The physics tendencies represent how a given scheme (or schemes) changes the state in a single model timestep. The tendencies are accumulated until it is time to apply them to the state. There is one tendency for each state variable being \"updated\" by the scheme. Some of these tendency variables are held within the physics_tend object, but others are internal to the physics.

          The module $CAM-SIMA/src/physics/ncar_ccpp/utilities/physics_tendency_updaters.F90 includes the schemes to apply the tendencies to the relevant state variables. These schemes are added to the SDF whenever the state should be updated. Each calculation looks like: state_var = state_var + tend*dt where dt is the timestep size.

          "},{"location":"design/constituents/","title":"Constituents","text":""},{"location":"design/constituents/#introductionoverview","title":"Introduction/Overview","text":"

          Some definitions to start (as written by a non-scientist, so there is more nuance than this!):

          • A constituent is a physical quantity or substance that exists in the atmosphere
          • A constituent can be advected, which means it is moved through the atmosphere by some sort of dynamical method
            • The dynamical core (dycore) is the part of an atmosphere model (like CAM-SIMA) that advects the quantities over the underlying grid. Dynamical cores include:
              • null/none - the null dycore does nothing and is used in CAM-SIMA to validate physics schemes
              • spectral element (SE): the only dycore currently implemented in CAM-SIMA
              • finite-volume cubed-sphere (FV3): not currently implemented in CAM-SIMA
              • model for prediction across scales (MPAS): in progress
          • A constituent had additional properties, such as:
            • water type: can be \"dry\", \"wet\", or \"moist\" (we can convert between any of these quantities - \"dry\" means that it's the \"amount\" of that constituent with respect to dry air
            • mixing ratio type: can be \"volume\" or \"mass\"
              • volume mixing ratio: constituent values are a ratio of how much of the constituent exists per mole of air (units = mol mol-1)
              • mass mixing ratio: constituent values are the mass of the constituent per unit volume (kg m-3)
            • molar mass: Molar mass of a given quantity; used in converting between mass mixing ratio and volume mixing ratio
            • thermodynamically active: thermodynamically active constituents in CAM-SIMA can be found in src/data/air_composition.F90
            • minimum value: The scheme qneg will set constituent values that are less than the minimum to the minimum
          "},{"location":"design/constituents/#cam-sima-constituent-handling","title":"CAM-SIMA constituent handling","text":""},{"location":"design/constituents/#determining-constituents","title":"Determining constituents","text":"

          There are three ways to identify a quantity as a constituent in CAM-SIMA:

          1. Constituent is provided by host (CAM-SIMA):
            • Host model constituents allow for constituents to be added as a constituent independent of the physics being run
            • Host constituents are added in cam_register_constituents (in src/control/cam_comp.F90). Currently, we are always adding water vapor as a host constituent because it is expected by the SE dycore.
          2. Constituent is a build-time physics constituent in a CCPP-ized scheme:
            • If a quantity is known to be a constituent at build-time, it is identified in the metadata for the scheme with: advected = True
          3. Constituent is a run-time physics constituent in a CCPP-ized scheme:
            • Sometimes, a scheme does not know what constituents it will require until run-time. In this case, an array of constituent properties (one for each needed constituent) is returned from the cam register phase. An example of how this works can be found in src/physics/ncar_ccpp/musica/micm/micm.F90
          "},{"location":"design/constituents/#registering-initializing-constituents","title":"Registering & Initializing Constituents","text":"

          The registration and initializaiton of the constituent data array and the constituent properties object are done through calls to the generated CCPP cap.

          • cam_ccpp_register_constituents: combines the three sources of constituents into one ccpp_model_constituents_t object
            • Called before the physics grid (which requires the number of constituents) is initialized
          • cam_ccpp_initialize_constituents: initializes the data array within the ccpp_model_constituents_t object
            • Called after the physics grid is initialized (so we know the size of the array allocate)
            • The array is initialized to (columns, levels, number of constituents)
          "},{"location":"design/constituents/#constituent-usage","title":"Constituent Usage","text":"

          Constituent values and properties can be accessed from the host side and from the physics in the following ways:

          • Host side: constituents and properties can be accessed via the host model and dycore by way of the cam_constituents.F90 module, which is an interface to the CCPP cap, which is in turn an interface to the constituents object
          • Physics: the constituent array and/or the constituent properties object are passed into a scheme via the following metadata (local name and intent may vary):
          [ q ]\n  standard_name = ccpp_constituents\n  units = none\n  type = real | kind = kind_phys\n  dimensions = (horizontal_loop_extent,vertical_layer_dimension,number_of_ccpp_constituents)\n  intent = inout\n[ const_props ]\n  standard_name = ccpp_constituent_properties\n  units = None\n  type = ccpp_constituent_prop_ptr_t\n  dimensions = (number_of_ccpp_constituents)\n  intent = in\n
          "},{"location":"design/constituents/#ccpp-framework-constituent-handling","title":"CCPP Framework constituent handling","text":"

          This section can be removed when constituents are documented in the CCPP Framework documentation.

          "},{"location":"design/constituents/#constituent-object-fortran","title":"Constituent object (Fortran)","text":"

          The constituent object (found in $CAM-SIMA/ccpp_framework/src/ccpp_constituent_prop_ mod.F90) is a flexible and extendable means of containing necessary constituent data for the framework. The primary object is ccpp_model_constituents_t.

          This object, importantly, contains the following properties (for which there is metadata; CCPP standard name in parenthesis):

          • const_metadata (ccpp_constituent_properties)
          • num_layer_vars (number_of_ccpp_constituents)
          • num_advected_vars (number_of_ccpp_advected_constituents)
          • vars_layer (ccpp_constituents)

          The const_metadata property is of type ccpp_constituent_prop_ptr_t, which contains a pointer to a ccpp_constituent_properties_t object, as depicted above. This object contains all of the constituent properties for the constituents in the constituents array, with the same indices as the constituents array.

          The ccpp_model_constituents_t type also contains a hash table of constituent properties for more efficient searching, as well as several methods used by the generated cap code. Some methods are highlighted below:

          • new_field: add a new constituent\u2019s set of metadata fields to the hash table
          • lock_table: lock the constituent hash table and initialize the constituent array
            • Initializes the constituent array to the default value specified for each constituent (min value held in constituent props array for each constituent)
            • Packs the advected constituents at the front of the constituents array
          • const_index: retrieves the constituent index for a constituent (provided the standard name)
          "},{"location":"design/constituents/#code-generation-python","title":"Code generation (python)","text":"

          The constituents-related code generation routines provide an interface to the constituents object. These routines can be found in $CAM-SIMA/ccpp_framework/scripts/constituents.py, primarily within the \u201cwrite_host_routines\u201d function. The (most often used) generated routines related to constituents are:

          • <hostname>_ccpp_register_constituents
          • <hostname>_ccpp_initialize_constituents
          • <hostname>_ccpp_number_constituents
          • <hostname>_constituents_array
          • <hostname>_model_const_properties
          • <hostname>_const_get_index

          The routines above are generated during ./preview_namelists or ./case.build and can be found here: $CASE/bld/atm/obj/ccpp/cam_ccpp_cap.F90

          "},{"location":"design/history/","title":"History & model output","text":"

          CAM-SIMA history is the mechanism for configuring and generating diagnostic output from a model run. It is also used to generate initial-data files and aids in the model-restart process by saving the state of diagnostic fields whose processing window (e.g., averaging, standard deviation) crosses a restart-write cycle. This page describes the implementation of CAM-SIMA history in CAM-SIMA.

          "},{"location":"design/history/#history-initialization","title":"History Initialization","text":""},{"location":"design/history/#reading-and-processing-the-history-configuration","title":"Reading and processing the history configuration","text":"
          • The allowable history configuration keywords are defined in cime_config/hist_config.py in _HIST_CONFIG_ENTRY_TYPES.
            • Each of the keywords is configurable via the namelist (user_nl_cam)
            • The syntax is <keyword>;<volume>: <value> (see examples below)
            • Currently, these configuration keywords are as follows (the atm_in equivalent indicates how hist_config.py parses these into a namelist to be read by SIMA):
          Configuration Keyword Description atm_in equivalent hist_add_avg_fields hist_add_inst_fieldshist_add_min_fields hist_add_max_field hist_add_var_field hist_remove_fields These configuration keywords add/remove fields to the specified volume with the indicated accumulation flag (average, instantaneous, minimum, maximum, standard deviation). The closest CAM7 equivalent is \u201cfinclX\u201d &hist_config_arrays_nl\u2002hist_num_avg_fields\u2002hist_num_inst_fields\u2002hist_num_min_fields\u2002hist_num_max_fields\u2002hist_num_var_fields&hist_file_config_nl\u2002hist_avg_fields\u2002hist_inst_fields\u2002hist_min_fields\u2002hist_max_fields\u2002hist_var_fields hist_file_type This keyword determines the type of file. Options are: \u201chistory,\u201d \u201csatellite,\u201d and \u201cinitial_value\u201dDefaults to \u201chistory\u201d &hist_file_config_nl\u2002hist_file_type hist_max_frames Indicates the maximum number of samples/frames that can be written to a file before that file is considered \u201cfull\u201d. The CAM7 equivalent is \u201cmfilt\u201d.Defaults to 1 for h0 and 30 for all other volumes. &hist_file_config_nl\u2002hist_max_frames hist_output_frequency Specifies the frequency of writes to the volume. The syntax is \"<integer>*<time period>\" where \u201ctime_period\u201d can be: steps, seconds, minutes, hours, days, months, years. The closest CAM7 equivalent is \u201cnhtfrq\u201d. &hist_file_config_nl\u2002hist_output_frequency hist_precision Denotes the precision for the volume. Options are \"REAL32\" and \"REAL64\".Defaults to \"REAL32\" &hist_file_config_nl\u2002hist_precision hist_write_nstep0 Indicates whether or not to write the nstep=0 sample to the volume.Defaults to .false. &hist_file_config_nl\u2002hist_write_nstep0 hist_filename_template Specifies the template for the filename for the volume. Defaults to \"%c.cam.%u.%y-%m-%d-%s.nc\" where \"%c\" is the case name, \"%u\" is the volume, \"%y\" is the year, \"%m\" is the month, \"%d\" is the day, and \"%s\" is the number of seconds since midnight GMT, with the timestamp itself representing the model time when the file is created. &hist_file_config_nl\u2002hist_filename_spec
          • hist_config.py also contains the HistoryVolConfig class (all the info pertaining to a single history file), the HistoryConfig class (all the history configuration information including a dict of HistoryVolConfig objects), and helper classes.
          • The HistoryConfig object is created in buildnml out of entries in user_nl_cam and written to run/atm_in.
          • In order to ensure that all relevant runtime (namelist) values show up in atm_in, the HistoryConfig object must contain all the logic in setting default values.
          "},{"location":"design/history/#setting-up-the-history-data-structures","title":"Setting up the history data structures","text":"
          • History namelist information is read and history data structures are set up in src/history/cam_hist_file.F90
          • The driving function in cam_hist_file.F90 is hist_read_namelist_config, which is called by history_readnl in src/history/cam_history.F90. This function reads in the hist_config_arrays_nl namelist group, allocates the history field arrays, and then uses those arrays to read in the hist_file_config_nl namelist group (via a call to read_namelist_entry).
          • The history configuration namelist options are used to populate cam_history.F90\u2019s module-level hist_configs array of hist_file_t objects (the size of this array is the number of user-configured volumes).
          • The hist_file_t object contains information about the configuration options for a given history volume. This includes the maximum number of frames that can be written to the file, the current number of frames on the file, the name of the file, and all of the history fields to be written to the file. It also contains methods to populate the field lists (config_set_up_fields), set up the metadata of the file (config_define_file), and write history fields to the file (config_write_time_dependent_variables).
          • Each hist_file_t object contains both a hash table and allocatable field list to keep track of the fields written to the file. The core class for each of these is the hist_field_info_t (in src/history/buffers/src/hist_field.F90), which contains information about a history field. This includes the field names, the accumulate flag (average, instantaneous, minimum, etc), units, type, dimensions, and fill value. It also includes the buffer(s) (type is/are hist_buffer_t) that will and do hold the actual data.
          "},{"location":"design/history/#populating-the-possible-field-list","title":"Populating the possible field list","text":"

          The possible fields to be output by the history infrastructure are tracked in cam_history.F90 via the possible_field_list hash table. It is populated during init time by calls to the subroutine history_add_field (found in src/history/cam_history.F90). \u201cInit time,\u201d means that all calls to history_add_field must occur during the execution of cam_init (found in src/control/cam_comp.F90).

          • Within the CCPP physics, this means that any diagnostic fields must be added to possible field list during _init (ideally the diagnostics are kept separate from the core scheme physics to keep the physics portable)
          • Within dynamics, the fields must be added during dyn_init or stepon_init
          • For physics variables:
            • State variables are added to the field list in the init phase of the cam_state_diagnostics scheme in src/physics/ncar_ccpp/diagnostics/cam_diagnostics.F90
              • This scheme is included at the end of the physics_before_coupler group in the suite definition file (SDF)
            • Tendency variables are added to the field list in the init phase of the cam_tend_diagnostics scheme in src/physics/ncar_ccpp/diagnotics/cam_diagnostics.F90
              • This scheme is included at the end of the physics_after_coupler group in the SDF
            • Additional variables specific to the scheme are included in the init phase of a _diagnostics scheme in src/physics/ncar_ccpp/diagnostics

              Each call to history_add_field adds a new field to the end of the possible_field_list_head linked list. At the end of cam_init, the possible field list linked list is used to print the list (to the atm.log* file) and then is converted to the possible_field_list hash table. A sample of the history field list is seen below.

                ***************** HISTORY FIELD LIST ******************\n             T          K  avg  air_temperature\n            ZM          m  avg  geopotential_height_wrt_surface\n          PHIS     m2 s-2  ins   surface_geopotential\n          PMID         Pa  avg  air_pressure\n       PDELDRY         Pa  avg  air_pressure_thickness_of_dry_air\n             Q    kg kg-1  avg  water_vapor_mixing_ratio_wrt_moist_air_and_condensed_water\n        CLDLIQ    kg kg-1  avg cloud_liquid_water_mixing_ratio_wrt_moist_air_and_condensed_water          \n        RAINQM    kg kg-1  avg  rain_mixing_ratio_wrt_moist_air_and_condensed_water\n         TTEND      K s-1  avg  tendency_of_air_temperature_due_to_model_physics\n  *************** END HISTORY FIELD LIST ****************\n
              "},{"location":"design/history/#capturing-history-output","title":"Capturing history output","text":"

              Outside of CAM-SIMA init and finalize time, history buffers can be populated with data via a call to history_out_field (found in src/history/cam_history.F90)

              The subroutine history_out_field iterates over the hist_configs array and populates the buffer(s) of the hist_field_info_t object of the hist_file_t object if the field name in the call is active on that file (e.g. the field was configured via the namelist to be output for that volume).

              • Within the CCPP physics, calls to history_out_field can exist anywhere except _init and _final
              • In dynamics, calls to history_out_field can exist anywhere except in dyn_init, stepon_init, and stepon_final
              • For physics variables, history_out_field calls are included in the run phase of the same schemes described in the section above.
              "},{"location":"design/history/#defining-new-history-files","title":"Defining new history files","text":"

              The cam_history.F90 subroutine history_write_files (which is called during cam_timestep_final) does three main actions for each of the user-defined history volumes:

              1. Determine if it\u2019s time to write to the file based on the user-supplied namelist option hist_output_frequency
              2. If it's time to write, determine if we need to open a new file
              3. Write the variables to the file(s)

              The bolded step #2 above is what determines if we need to define a new history file. The situations where we would need to define a new file are:

              1. It's the first sample we're outputting to a specific volume.
              2. The last time we output to the volume, we \"filled\" the file (number of samples written to the file met the user-configured hist_max_frames) and closed it

              We determine if it's time for a new file with the following line of code:

                 mod(num_samples, hist_configs(file_idx)%max_frame()) == 0\n

              If it is indeed time to define a new file, we call the config_define_file subroutine (which is found in src/history/cam_hist_file.F90) for the volume:

                call hist_configs(file_idx)%define_file(restart, logname, host, model_doi_url)\n
              • This subroutine opens the netcdf file(s) and writes the necessary time-independent metadata (for the file and for the history fields) and grid information.
                • Note that there will be a maximum of TWO (2) files opened at this stage. If there are both instantaneous fields AND accumulated fields (average, minimum, maximum, standard deviation) on the volume, two files will be created (hXi AND hXa); otherwise, only the relevant file will be created (hXi OR hXa)
              "},{"location":"design/history/#writing-a-history-file","title":"Writing a history file","text":"

              The cam_history.F90 subroutine history_write_files (which is called during cam_timestep_final) does three main actions for each of the user-defined history volumes:

              1. Determine if it\u2019s time to write to the file based on the user-supplied namelist option hist_output_frequency
              2. If it's time to write, determine if we need to open a new file
              3. Write the variables to the file(s)

              The bolded step #3 above occurs any time the criteria for #1 is satisfied. At this point, the following call is made to write the history fields (whose data has been stored in their buffers via calls to history_out_field):

                call hist_configs(file_idx)%write_time_dependent_variables(file_idx, restart)\n

              It is during this call that we increment the number of samples written for this volume and actually write the data held within the buffer(s) to the netcdf file (as well as the time-dependent metadata for the fields).

              "},{"location":"design/history/#defining-history-restart-files","title":"Defining history restart files","text":"

              Restarts not yet implemented in CAM-SIMA

              "},{"location":"design/history/#writing-a-history-restart-file","title":"Writing a history restart file","text":"

              Restarts not yet implemented in CAM-SIMA

              "},{"location":"design/history/#reading-a-history-restart-file","title":"Reading a history restart file","text":"

              Restarts not yet implemented in CAM-SIMA

              "},{"location":"design/sima-design-goals/","title":"Design goals & features","text":"

              Motivated by the Singletrack project (a precursor to SIMA), the CAM-SIMA project was created to build a new CAM infrastructure to meet the SIMA science and computational needs. This page documents those needs and some of the features that implement them.

              "},{"location":"design/sima-design-goals/#cam-needs-to-be-more-run-time-configurable","title":"CAM needs to be more run-time configurable","text":"
              • To make CAM and CESM more available, e.g., for usage in containers and the cloud, a build of CAM should be more configurable than it is at present.
              • One feature that makes CAM more run-time configurable is moving physics suites to the CCPP. By allowing CAM to compile in more than one physics suite, the physics suite can be selected at run time (e.g., as a namelist variable).
              • Another feature needed to make CAM more run-time configurable is making dycores themselves more run-time configurable. For instance, the SE dycore will no longer require the number of advected constituents to be specified at compile time.
              "},{"location":"design/sima-design-goals/#remove-obstacles-to-use-of-specialized-high-performance-processing-units-eg-gpus-fpgas","title":"Remove obstacles to use of specialized high-performance processing units (e.g., GPUs, FPGAs)","text":"

              The chunk structure in CAM physics served its purpose when threading was the only way to accelerate code. However, to make the best use of both threading and modern accelerators, a flexible chunking mechanism is required. The new infrastructure enables this by using flat arrays for all fields.

              • Moving to flexible precision of data is important for being able to test both performance improvements and the affect on model quality. The CCPP is explicitly designed to allow for compile-time selection of precision at the physics suite level as well as for individual fields. In addition, the new infrastructure is explicitly designed to handle the case where the dycore is running at a different precision than the physcs (i.e., by using proper field promotion and demotion primitives).
              • Pointers in Fortran are less efficient because they prevent some optimization techniques. The new infrastructure avoids pointers as much as possible by making use of the automatic data management capability of the CCPP (which does not create pointers).
              • The new infrastructure provides greater flexibility in that the model can be built with multiple physics suites to increase run-time flexibility. There is a tradeoff in that building more physics suites will often increase build time. Builds with a single suite should be faster than now since only the schemes that are required for the suite are compiled (currently most schemes are compiled all the time even if they will not be used).
              "},{"location":"design/sima-design-goals/#modularity","title":"Modularity","text":"

              In order to continue to allow CAM to grow without an ever increasing cost of bringing in new features, CAM must be more modular. A classic example is chemistry which ACOM would like to make modular but which is currently entwined with CAM in many areas (e.g., code in CAM repository, extensive configuration code dedicated to chemistry, extensive namelist building code and data dedicated to chemistry, large number of use cases containing chemistry configuration data). The new CAM infrastructure contains features to increase modularity.

              • Support for multiple namelists. This allows modular components to contain their own namelist (run-time configuration options). The active namelists are combined to produce atm_in.
              • Flexible handling of constituent information. Modular components can provide constituent information via metadata (if component is a CCPP scheme) or at run time.

              Modularity will allow CAM to move components to external repositories. This process cuts development costs for both CAM and for the component (e.g., as has happened with PUMAS). Some ways this is accomplished are listed here:

              • Code reviews are more efficient since CAM SEs do not have to review every routine in the external module so they can just focus on the interfaces. The external developers do not have to be involved in CAM modifications.
              • Externals can develop and maintain they own namelist definition files, they do not have to coordinate with the larger CAM namelist (which itself has been broken into several smaller namelists).
              • Namelists associated with physics schemes do not have to have separate namelist-reading code. The new infrastructure automatically creates an appropriate Fortran module to read in the runtime data from atm_in. The system then also ensures that all active namelists are called at run time. This process ensures that namelists are always read correctly while not requiring coding or reviews to keep up to date with namelist changes.

              Use of the CCPP to build physics suites also makes CAM more modular because the CCPP treats physics schemes as modular which allows flexibility in building physics suites. The CCPP takes care of making sure variables are available before they are used and also builds the code currently handled via hand-written code in the various versions of physpkg.F90.

              "},{"location":"design/sima-design-goals/#run-time-data-checking","title":"Run-time data checking","text":"

              CAM needs data to run but the data needs vary with the simulation. The new infrastructure facilitates this.

              • Before running physics, the new infrastructures queries the physics suite as to what input fields are required (using a CCPP interface). Then it makes sure that all of these fields have been initialized or reads the values from the initial data file. Any uninitialized fields that are not found on the initial data file will trigger a run-time error.
              "},{"location":"design/sima-design-goals/#efficient-offline-testing-and-simulation","title":"Efficient offline testing and simulation","text":"

              CAM currently has a few ways to run offline testing (e.g., SCAM, PORT). The new infrastructure builds these capabilities in for more efficient and flexible use.

              • The new infrastructure has the ability to run without a dycore.
              • Offline mode (NULL dycore) can be run with any number of columns.
              • Offline mode does not required gridded input.
              "},{"location":"design/sima-design-goals/#software-quality-control","title":"Software quality control","text":"

              To enable efficient quality control, the new infrastructure implements a number of continuous integration (CI) techniques.

              • To implement all the flexibility mentioned above, the new infrastructure makes extensive use of python scripts.
                • Python scripts make extensive use of python doctests (for simpler tests).
                • There are python unit tests to cover more complicated situations.
                • The GitHub CI also runs a static analysis tool (pylint) to provide feedback on potential coding issues.
                • Python tests can easily be run by hand but are also automatically run on GitHub
              • The new infrastructure will use offline mode (see above) to run many physics configurations quickly without requiring large machines.
                • This will enable quick testing during development on a laptop.
                • We hope many of these tests can also be run automatically on GitHub.
              "},{"location":"development/cam-coding-standards/","title":"Coding standards","text":"

              The standards in this document are largely prescriptive, i.e., they should be followed.

              • MUST: Exceptions must be discussed and agreed to by the CAM SEs and noted in this document.
              • SHOULD: Exceptions must be approved by the CAM SEs and documented in the ChangeLog.

              While some legacy code will not follow these rules, efforts SHOULD be made to improve the code whenever you are working on it (e.g., bug fixes, enhancements).

              "},{"location":"development/cam-coding-standards/#general-coding-standards","title":"General coding standards","text":""},{"location":"development/cam-coding-standards/#must","title":"MUST","text":"
              • Always use spaces instead of tabs
              • No trailing spaces (i.e., no spaces at the end of a line)

              See tips for configuring editors

              "},{"location":"development/cam-coding-standards/#should","title":"SHOULD","text":"
              • Use comments to explain the purpose of the following code and/or include any important but non-obvious information. When working with code, always check the comments to make sure they are still correct and useful.
              • Do not use comments to 'save code for later in case it might be useful'.
              • Do not include a comment that merely restates the following code logic (e.g., 'Loop over variables')
              "},{"location":"development/cam-coding-standards/#python-coding-standards","title":"Python coding standards","text":"

              We expect all python code to pass with a perfect score (10) using pylint with this version of pylintrc. However, external repos may have their own pylintrc version depending on their needs.

              We also expect all python files to follow the black code style and format.

              Finally, one can also follow the Google Python Style Guide.

              "},{"location":"development/cam-coding-standards/#fortran-coding-standards","title":"Fortran coding standards","text":"

              The standards described in this section represent the CAM Fortran standards. Other Fortran standards:

              • CTSM Fortran Standards
              • MOM Fortran Standards
              "},{"location":"development/cam-coding-standards/#must_1","title":"MUST","text":"
              • No naked use statements
              • No continued single-line if statements (i.e., all if statements should have a then if the statement is on more than one line)
              • Every namelist variable in each active namelist group is present in the namelist file. An active namelist group is one which may be read during the current run.
              • All namelist variables except for logical quantities are initialized to invalid values (integer: -HUGE(1), real: NaN, character: 'UNSET').
              • Functions may not have side effects, and should include the pure keyword.
              • Do not combine statements on a single line (i.e., avoid use of the semi-colon to combine statements).
              • Use intent for dummy arguments except for pointers.
              • All variables of type real must have a specified kind, including literals. For example, use 1.5_r8, not 1.5 or 1.5D0. Literals must also include the decimal point.
              • All character declarations must use Fortran 90+ syntax (e.g., character(len=*) or character(len=CL)).
              • All variable declarations must use Fortran 90+ syntax (i.e., must include the double colon between the attributes and the variable name).
              • All type and procedure declarations must use Fortran 90+ syntax (i.e., must include the double colon before the type or procedure name).
              • All modules should include an implicit none statement in the preamble (after the use statements). Module routines then do not need this statement.
              • All optional arguments must be passed via keyword (e.g. use call subroutine(x, optional_y=y) instead of call subroutine(x, y) for the optional variable optional_y).
              • Initialize local (non-parameter) variables in subroutines and functions at the top of the executable code, NOT on a variable declaration lines.
                • Initializing a local variable on a declaration line invokes the SAVE attribute and is not thread safe.
                • Local pointer variables MUST be initialized before other (non-initialization) statements. By default, use the nullify statement.
              • All variables that are on the physics grid must have their horizontal dimension declared with pcols, even if only a subset of the variable is used in the subroutine or function.
              "},{"location":"development/cam-coding-standards/#should_1","title":"SHOULD","text":"
              • Avoid use of preprocessor directives (e.g., #if, #ifdef). Always try for runtime variable logic instead.
              • Keep formula statements relatively short. Use temporary variables to break long formulas into easier-to-read sections.
              • Use subroutines to avoid repeated (cut and paste) code logic.
              • Avoid side effects in subroutines. Pass variables to routines instead of 'using' them from elsewhere.
              • Use the pure keyword if a subroutine has no side effects.
              • List dummy arguments one per line, however, related items may be grouped.
              • Dummy argument order should match the order in the argument list.
              • Use symbolic numerical comparison operators (e.g., ==, /=, <, >=) not old character versions (e.g., .eq.).
              • Avoid the use of pointers as dummy arguments (exceptions must be discussed in design or code review)
              • Modules should be default private. Public interfaces are declared after the private declaration.
              • private module interfaces (i.e., subroutines and functions) should be declared private in the module header.
              • Module names should conform to their filename (i.e., the module name should be the filename without the .F90).
              • Functions should use the pure attribute. If they cannot, the reason should be included as a comment in the function's preamble.
              • All functions and subroutines should avoid un-necessary statements (e.g. a blank return at the end of a subroutine).
              • use statements should be brought in at the smallest scope possible (e.g. inside individual subroutines instead of at the module level).
              "},{"location":"development/cam-coding-standards/#indentation-and-style","title":"Indentation and style","text":"
              • Scoping: Indentation should follow scope. That is, whenever entering a new scope (e.g., module, subroutine, if, do), indent that scope relative to the scoping statement (recommended 3 spaces but each module should at least be self consistent).
              • A single line should be less than 133 characters long.
              • Continue lines: Indent continue lines 5 spaces or align with similar lines in statement.
              • Use spaces to ease reading statements (e.g., before and after operators, after commas except in a dimensions list)
              • Include a space after if, else, end, do, and while.
              • Include a space before and after ::
              • No space after only, i.e., only:, not only :.
              • When aligning code for readability, commas go immediately after a symbol (no space).
              "},{"location":"development/cam-coding-standards/#tips-for-configuring-editors","title":"Tips for configuring editors","text":""},{"location":"development/cam-coding-standards/#emacs-add-to-your-emacs-file","title":"emacs (add to your .emacs file)","text":"
              • To automatically remove trailing spaces whenever you save a file:
              (add-hook 'before-save-hook 'delete-trailing-whitespace)\n
              • To automatically indent with spaces instead of tabs:
              (setq-default indent-tabs-mode nil)\n
              • To use 4 spaces for each indent:
              (setq tab-width 4)\n
              "},{"location":"development/cam-coding-standards/#vi-add-to-your-vimrc-file","title":"vi (add to your .vimrc file)","text":"
              • To automatically remove trailing spaces whenever you save a file:
               autocmd BufWritePre * :%s/\\s\\+$//e\n
              "},{"location":"development/cam-testing/","title":"Testing","text":"

              This page describes the various automated and manual tests that are run for CAM-SIMA whenever the code is modified, as well as instructions for how to add new tests.

              "},{"location":"development/cam-testing/#python-unit-testing","title":"Python unit testing","text":"

              CAM-SIMA supports two kinds of python unit tests, doctest and unittest tests, both of which are part of the standard python library.

              All unittest tests should be in:

              CAM-SIMA/test/unit

              while all files used by the tests should be in:

              CAM-SIMA/test/unit/sample_files

              All unittest tests are automatically run via Github Actions whenever a Pull Request (PR) is opened, modified, or merged.

              All doctest tests are also run automatically as long as the scripts they are located in are under CAM-SIMA/cime_config or CAM-SIMA/src/data.

              To manually run all of the unit tests at any time, simply run the following shell script:

              CAM-SIMA/test/run_tests.sh

              Finally, when adding new tests, determine if the test can be done in only a few lines with minimal interaction with external files or variables. If so, then it would likely be best as a doctest. Otherwise it should be a unittest test. Failure to follow this rule of thumb could result in test failures in the Github Actions workflow. Also remember to add your new tests to the run_tests.sh script so future users can easily run the tests manually.

              "},{"location":"development/cam-testing/#static-source-code-analysis","title":"Static Source Code Analysis","text":""},{"location":"development/cam-testing/#python","title":"Python","text":"

              Any python script which is added or modified via a PR will automatically be analyzed using pylint, and must have a score of 9.5 or greater in order to not be marked as a test failure. The hidden pylintrc file used for the analysis can be found here:

              CAM-SIMA/test/.pylintrc

              Users can also manually run pylint against the core python build scripts by running the following shell script:

              CAM-SIMA/test/pylint_test.sh

              Please note that pylint is not part of the standard python library, and so it may need to be installed before being able to run the shell script.

              "},{"location":"development/cam-testing/#regression-testing","title":"Regression Testing","text":""},{"location":"development/cam-testing/#running-the-regression-tests-manual","title":"Running the regression tests (manual)","text":"

              NOTE: Regression testing on Derecho should be done for every PR before merging!

              Users can manually run regression tests on Derecho to ensure that the model builds correctly in various configurations. The tests can be run with a local copy of CAM-SIMA by using the test_driver.sh script under $CAM-SIMA/test/system. To run the tests associated with a particular compiler option one can do the following commands:

              For running GNU tests*:

              env CAM_FC=gnu ./test_driver.sh -f\n

              For running Intel tests*:

              env CAM_FC=intel ./test_driver.sh -f\n

              Running the script will produce a directory in your scratch space labeled aux_sima_<CAM_FC>_<timestamp>, where <CAM_FC> is the compiler you chose, and <timestamp> is the timestamp (starting with the date) of when the tests were started, along with a job submitted to the local batch system.

              Inside the directory you should see an executable labeled cs.status.*. Running that command after the submitted job has finished will display the test results. Currently for all tests everything should be labeled PASS except the SUBMIT step, which should be labeled PEND. Any other label indicates that a test may have failed, and should be investigated.

              Finally, the tests themselves are listed in <CAM-SIMA>/cime_config/testdefs/testlist_cam.xml. Any files that need to be included in order for the tests to run properly are located in <CAM-SIMA/cime_config/testdefs/testmods_dirs/cam/outfrq_XXX, where XXX is the name of the test. Additional information on the CIME testing system, which is what this testing infrastructure is built on, can be found online here.

              *Note: you may also have to include the environment variable CAM_ACCOUNT on derecho, which points to your account key

              "},{"location":"development/cam-testing/#adding-a-new-regression-test","title":"Adding a new regression test","text":"

              The test list can be found here: $CAM-SIMA/cime_config/testdefs/testlist_cam.xml

              • If you are adding a new machine, compiler or category for an existing test, add a new <machine> XML entry
              • If you are adding a fully new test, add a new <test> XML entry with the following structure:
              <test compset=\"<COMPSET_NAME>\" grid=\"<GRID_ALIAS>\" name=\"<TEST_TYPE>_<TEST_MOD>\" testmods=\"<RELPATH_TO_TESTMODS_DIR>\">\n  <machines>\n    <machine name=\"<MACH_NAME>\" compiler=\"<COMPILER>\" category=\"<TEST_CATEGORY>\"/>\n  </machines>\n  <options>\n    <option name=\"comment\">COMMENT HERE</option>\n    <option name=\"wallclock\">WALLCLOCK_TIME</option>\n  </options>\n</test>\n
              • <COMPSET_NAME>: component set alias (or long name) - you can see more about compsets here
              • <GRID_ALIAS>: model grid/resolution you'd like to run on - you can see more about grids here
              • <TEST_TYPE>: type of test to be run. You can find the testing types here.
              • <TEST_MOD>: test modifier that changes the default behavior of the test type. More here
              • <RELPATH_TO_TESTMODS_DIR>: relative path to the testmods directory for this run; usually looks something like \"cam/some_directory_name/\"
                • The testmods directory will contain any namelist mods and XML configuration variable changes for the test (user_nl_cam and/or shell_commands)
                • testmods directories can be found in $CAM-SIMA/cime_config/testdefs/testmods_dirs/cam/
              • <MACH_NAME>: machine name - will almost definitely be either derecho or izumi
              • <COMPILER>: compiler to be used (options: gnu, nag, intel, nvhpc)
              • <TEST_CATEGORY>: group of tests that this test belongs to - the default run by test_driver.sh is aux_sima (which is run for each PR to CAM-SIMA)
              • WALLCLOCK_TIME: maximum amount of time that the job will be allowed to run

              Here is an example test entry for a 2-timestep smoke test of kessler physics on the MPAS grid, run with both intel and gnu 

                <test compset=\"FKESSLER\" grid=\"mpasa480_mpasa480\" name=\"SMS_Ln2\" testmods=\"cam/outfrq_kessler_mpas_derecho_nooutput/\">\n    <machines>\n      <machine name=\"derecho\" compiler=\"intel\" category=\"aux_sima\"/>\n      <machine name=\"derecho\" compiler=\"gnu\" category=\"aux_sima\"/>\n    </machines>\n    <options>\n      <option name=\"wallclock\">00:10:00</option>\n      <option name=\"comment\">GNU build test for MPAS dycore (with Kessler physics)</option>\n    </options>\n  </test>\n
              "},{"location":"development/debugging/","title":"Debugging techniques","text":"

              Start with the CAM wiki.

              "},{"location":"development/debugging/#cam-sima-specific-debugging","title":"CAM-SIMA-specific debugging","text":""},{"location":"development/debugging/#build-errors","title":"Build errors","text":"

              Debugging tips if you get build errors:

              • If the output indicates that the error message or failure is coming from somewhere within $CAM-SIMA/ccpp_framework:
                • If you're getting a clear error message, it's likely that you have something wrong with your metadata
                • If you're getting an error message that indicates that something is breaking in the framework code itself (something went uncaught) - consult the AMP SEs
              • If the error happens during the atm build, you can see the full output of the atm build in the build log here: bld/atm.bldlog.*
              "},{"location":"development/debugging/#run-time-errors","title":"Run-time errors","text":"
              • Start with the atm.log* - if the issue occurred during the execution of the CAM code, it will hopefully have a clear and concise error message
              • Move to the cesm.log* - it will hopefully include a stack trace for the error in question; if the error did not occur in the CAM code (or CAM did not properly trap the error), it will help you identify the source of the issue.
              • If neither log file contains helpful information, a few first steps:
                • Resubmit the case; it could be a machine hiccup
                • Turn on DEBUG mode (if it's not on already) and rebuild/rerun
                • Look in your run directory for any log files called PETXXX - if there was an issue on the ESMF side of things, it will show up in one of these (there will be one PET file per processor)
                • Try a different compiler - maybe it'll give you a more helpful error message
                • set NTASKS=1 (./xmlchange NTASKS=1), do a clean rebuild (as instructed), and run again; maybe running in serial will identify the error
                • Look for the ***************** HISTORY FIELD LIST ****************** in the atm.log* file; if it's not there, the error occurred at init time
                  • If the error occurred during init time, try a new case with a different grid and/or dycore
                  • If the model ran for a few timesteps before dying (look for the CAM-SIMA time step advanced message in the atm.log* file), it's likely that one or more variable that you introduced or modified has gone off the rails (value has become very large or very small or zero)
                    • Update your user_nl_cam to output all possible suspected variables to a history file at some point shortly before the model dies, then inspect the output to see if any are obviously wrong
                  • If the model completed all timesteps, try running a shorter case to see if the problem persists; if so, it's an error during the model finalization
                • Run the TotalView debugger on izumi
                • Use the old standard - print statements - to narrow down where the code is stopping
                • Ask for help!
              "},{"location":"development/debugging/#unexpected-answer-changes","title":"Unexpected answer changes","text":"
              • Two paths here:
                • You're getting unexpected DIFFs from the regression testing
                  • Consult with a scientist about whether differences are expected and for which configurations (compsets, resolutions, namelists parameters, etc)
                  • If the differences are very small (look like round-off), consult with the other AMP SEs on whether we're ok with this
                  • If the differences are indeed unexpected and larger than round-off, create a case using the code from the head of development and:
                    • place print statements in both code bases (your development branch and the head of development) to identify where the numbers are going awry OR
                    • run the TotalView debugger OR
                    • use the comparison tool described below ($CAM-SIMA/tools/find_max_nonzero_index.F90)
                • You're getting unexpected answer changes compared with CAM
                  • Consult with other AMP SEs about whether the differences appear to be due to round-off error
                  • Use the comparison tool (LINK ONCE IT EXISTS): $CAM-SIMA/tools/find_max_nonzero_index.F90
                    • This tool can help you narrow down where the issue begins by printing out values at a specific index and comparing those with the \"truth\" (from CAM)
              "},{"location":"development/debugging/#totalview","title":"TotalView","text":"

              (COURTNEY - ASK CHERYL TO HELP WITH THIS)

              • Create and configure a new case (using gnu and only 1 task)
              • Build the case (./case.build)
              • Run command bash to change to bash (if not already)
              • Run the following commands:
              np=1\nnthreads=1\n\nsource .env_mach_specific.sh\n\nRUNDIR=`./xmlquery RUNDIR -value`\nEXEROOT=`./xmlquery EXEROOT -value`\nLID=`date '+%y%m%d-%H%M%S'`\n\ncd $RUNDIR\nmkdir timing\nmkdir timing/checkpoints\necho `pwd`\nexport OMP_NUM_THREADS=$nthreads\ntotalview ${EXEROOT}/cesm.exe\n
              • exit to exit the totalview window and give up the node
              "},{"location":"development/git-basics/","title":"Working with git and GitHub","text":""},{"location":"development/git-basics/#getting-started-with-git","title":"Getting started with git","text":"
              • For a quick, visual introduction to git, try a 5 Minute Overview of Git on Youtube
              • Need an introduction to version control along with git and GitHub? Try Git and GitHub for Poets, a Youtube series
              • For a glossary of common terms used in git and GitHub documentation, see the GitHub glossary.
              • GitHub's Git Handbook offers a brief overview of git and GitHub, including example workflows and links to more in-depth resources.
              • Software Carpentry offers a hands-on introduction to git.
              • A highly-regarded, comprehensive git reference is the book Pro Git (available as a book or a free pdf).
                • The first three chapters of Pro Git, along with sections 6.1 and 6.2 on GitHub, are particularly useful if you're getting started with git. [1]
                • The site also contains links to git reference pages along with introductory videos on git.
                • You can skip the last section of chapter 3, on rebasing, until you have more experience with git. It's easy to cause problems for yourself or others with rebasing so only use rebasing in CAM-SIMA following the workflow instructions below as rebasing rewrites history, which can make it impossible to reproduce exactly what you have done before.
              • For a clear introduction to git internals, try Dissecting Git Guts by Emily Xie.
              • Git from the inside out is a blog post with accommpaning video from Mary Rose Cook that looks under the covers a bit to show what happens when you execute git commands.
              • Git from the Bottom Up by Josh Wiegley provides a quick glossary of the main terms describing a git repository with links to detailed descriptions. Good site for reminders about a git concept.
              • Need to really understand the science behind how git works? Try Advanced Git: Graphs, Hashes, and Compression, Oh My! by Matthew McCullough

              Git also offers extensive built-in (i.e., command line) help, although it can be hard to understand until you are familiar with basic git concepts:

              git help

              git help COMMAND

              man gittutorial

              man giteveryday

              "},{"location":"development/git-basics/#how-to-set-up-your-git-development-environment","title":"How to set up your git development environment","text":"

              There are several stages in setting up your git and GitHub development environment. Below, they are broken into three sections which represent the different stages:

              • One time GitHub setup
              • Set up git environment on new machine
              • Working with clones
              "},{"location":"development/git-basics/#one-time-github-setup","title":"One time GitHub setup","text":"
              1. Set-up personal github account (if one does not already exist): https://github.com

              2. Create a new personal CAM-SIMA fork of the ESCOMP/CAM-SIMA repo:

              3. Set up SSH keys on GitHub (optional):

                If you will be pushing changes to GitHub (either to your own fork or to shared forks), or if you will be pulling changes from private repositories on GitHub, then it's worth the time to set up ssh keys for each machine you'll be using. By doing so, you won't have to enter your password when pushing to or pulling from GitHub.

                See here for instructions. After doing this, you can use the ssh form of GitHub URLs (e.g., git@github.com:ESCOMP/cam-sima.git) in place of the https form.

              4. Configure GitHub notifications (optional): It is important to keep track of activity on GitHub but there are ways to manage how you are notified.

                • Controlling how you receive notifications: Click on your profile picture in the upper-right of any GitHub page, then click on \"Settings\", and then on \"Notifications\". You will see several options, here are a few recommended settings:
                  • Automatically watch repositories: Yes (check box)
                  • Participating: Email
                  • Watching: Web
                • To see and manage Web notifications: Click on the bell in the upper right of any GitHub page to see Web notifications and to manage which repositories you are watching. There will be a blue dot on the bell if there are new (unread) notifications).
              "},{"location":"development/git-basics/#set-up-git-environment-on-new-machine","title":"Set up git environment on new machine","text":"

              git has global settings which apply to all repository clones on your machine. These settings reside in a file called .gitconfig in your home directory. Below are some required and some optional but recommended global git settings to apply to any new machine where you will do CAM-SIMA development (e.g., Derecho, Izumi, personal laptop). Apply the settings below or simply copy a .gitconfig file from a machine that is already configured.

              "},{"location":"development/git-basics/#required-git-global-configuration-settings","title":"Required git global configuration settings","text":"
              git config --global user.name \"Your Name\"\ngit config --global user.email <GitHub email address>\n

              We recommend that you use your UCAR email address as your GitHub email address but if you use another address, you can add your UCAR email address by clicking on your profile picture in the upper-right of any GitHub page, then clicking on \"Settings\", and then on \"Emails\".

              "},{"location":"development/git-basics/#recommended-git-global-configuration-settings","title":"Recommended git global configuration settings","text":"

              You can set which editor to use for log messages, etc., with:

              git config --global core.editor <editor of your choice: emacs, vi, vim, etc>\n

              (See http://swcarpentry.github.io/git-novice/02-setup for specific settings to use for many common editor choices.)

              The following setting generates better patches than the default:

              git config --global diff.algorithm histogram\n

              The following setting makes it easier to resolve conflicts if you're doing conflict resolution by hand as opposed to with a dedicated conflict resolution tool. Without this setting, you'll just see your version and the version you're merging in delimited by conflict markers. With this setting, you'll also see the common ancestor of the two sides of the merge, which can make it much easier to figure out how to resolve the conflict:

              git config --global merge.conflictstyle diff3\n

              Alternatively, look into using a graphical conflict-resolution tool such as kdiff3 or the Emacs built-in M-x vc-resolve-coflicts.

              We recommend that you set git to not push anything by default:

              git config --global push.default nothing\n

              This can help prevent you from accidentally pushing work to the wrong repository.

              "},{"location":"development/git-basics/#configuring-git-on-shared-machines","title":"Configuring git on shared machines","text":"

              If using git on shared resources, such as on the login nodes for CISL machines, then one may find their git commands being killed by sys admins due to git spawning too many threads and thus blocking (or at least slowing down) other users. To avoid this situation, you can limit the number of threads git spawns for various activities by setting the following git config variables:

              git config --global --add index.threads 8\ngit config --global --add grep.threads 8\ngit config --global --add pack.threads 8\n

              Please note that a limit of 8 threads was chosen specifically for CISL machines. If you are using a separate shared system you may find it beneficial to choose a different thread limit.

              "},{"location":"development/git-basics/#git-tools-for-bash","title":"git tools for Bash","text":"

              There are two helpful things you can do to streamline your use of git if you use the bash shell. These are completely optional, but improve your git experience. These are documented in the appendix of the excellent Pro Git book.

              "},{"location":"development/git-basics/#updating-your-login-to-a-github-token","title":"Updating your login to a GitHub token","text":"

              If you have been denied access to push to a GitHub repo, note that the old password system has been deprecated and no longer works. If you do not make changes, you will no longer be able to work with GitHub on personal forks and private repos.

              If you only ever git clone from public repositories like ESCOMP and ESMCI, you may ignore the rest of this wiki page.

              If you've already created and are using a GitHub token, you may ignore the rest of this wiki page.

              GitHub will soon be requiring that you use a GitHub generated token (basically a long password that they autogenerate for you).

              1. If you are on a Mac and do not need to use your password, but still receive an error message, you may have stored your password in the keychain. Remove a password on Macs
              2. Create a GitHub password token. Make sure you copy it to a safe place as they only show it to you once.

              3. Store your credentials on every machine you use GitHub on.

                • On a private / secure Linux system:

                  git config --global credential.helper store

                • Note that the store option to git's credential helper stores your GitHub token in plain text. If you are on a public machine, you should store your GitHub token in a password manager and use the credential cache mechanism to open command-line access to your GitHub repositories:

                  git config --global credential.helper cache --timeout <seconds>

                  where <seconds> is the number of seconds git 'remembers' your token. For example, to only have to enter your token once per day:

                  git config --global credential.helper cache --timeout 86400

                • On Windows:

                  git config --global credential.helper wincred - On Macs (make sure you select the Mac tab in the middle of the window): see this documentation

              4. On each machine, do a git clone of your personal repo. It will ask for your username and password. The password needs to be your saved token. You should only need to do this once.

              5. To test that your credentials are successfully cached, do a new git clone of your personal repo. If it clones without asking for your username/password you are done.
              "},{"location":"development/git-basics/#working-with-clones","title":"Working with clones","text":"

              Here are some commands for creating and working with clones:

              "},{"location":"development/git-basics/#create-a-new-clone","title":"Create a new clone","text":"
              git clone https://github.com/<GitHub userid>/CAM-SIMA\ncd CAM-SIMA\n

              or

              git clone git@github.com:<GitHub userid>/CAM-SIMA\ncd CAM-SIMA\n

              where <GitHub userid> is your GitHub account login ID. Some useful options to the clone command are:

              • --origin <origin name>: A clone knows where it came from and by default, calls that location, \"origin\". You can change that name with the --origin option. This can come in handy when dealing with multiple upstream repositories. Change the clone directory name: By default, the clone is created in a directory with the same name as the repository. You can change this by adding a directory name to the clone command. Use the appropriate example below:
              git clone https://github.com/<GitHub userid>/CAM-SIMA  <clone_dir_name>\ncd <clone_dir_name>\n

              or

              git clone git@github.com:<GitHub userid>/CAM-SIMA <clone_dir_name>\ncd <clone_dir_name>\n
              "},{"location":"development/git-basics/#checking-out-a-tag-or-switching-to-a-new-tag","title":"Checking out a tag or switching to a new tag","text":"
              • To check out a tag: 
              git checkout <tag>\n

              note that <tag> can also be the name of a branch or a commit hash. If you specify the name of a branch, you will check out the head of the branch. If you name a remote branch (e.g., origin/branch_name), you will create a detached HEAD but you can still use the code. Please note that if you plan on changing the code, first create a branch (see Working with branches

              "},{"location":"development/git-basics/#working-with-branches","title":"Working with branches","text":"

              When you create a clone, your clone will contain pointers all the branches that existed at the clone's origin (e.g., the repository at GitHub). While you can check out these branches, however, before attempting to make any changes, you should first create a local version branch (so git can keep track of the local commits).

              • To create a new local branch that starts at a certain point:
              git branch <new branch name> <tag or branch name>\n

              for example

              git branch new_feature cam6_2_024\n
              • To check out a local branch:
              git checkout <new branch name>\n
              • If you are working with a repository that uses git-fleximod (e.g., CAM-SIMA, CESM), always run that tool after checking out a new branch or tag:
              bin/git-fleximod update\n
              "},{"location":"development/git-basics/#working-with-remotes-upstream-repository-locations","title":"Working with remotes (upstream repository locations)","text":"

              While working with clones created using the methods above will be sufficient for most if not all of your development needs, there may be times when you will want to access or compare your code with code from a different repository. git has no problem storing revisions from multiple repositories in a single clone!

              To begin, your clone probably has a single remote (also known as an upstream repository). To see the current status of which upstream repositories are configured for your clone, use the git remote command:

              git remote\n

              To see the location of the remote repositories in your current directory:

              git remote -v\n

              You should see something like:

              origin  https://github.com/gituser/CAM-SIMA (fetch)\norigin  https://github.com/gituser/CAM-SIMA (push)\n

              This tells you the \"upstream\" location from where new code is downloaded (when you run git fetch origin) or where code is uploaded (when you run git push origin <branch>). Note that most git commands are purely local, using only information in the .git directory of your clone.

              You can rename an existing remote:

              git remote rename origin ESCOMP\n

              You can set the remote name as part of a clone command (the default is 'origin'):

              git clone -o ESCOMP https://github.com/ESCOMP/cam-sima\n
              "},{"location":"development/git-basics/#adding-remote-new-upstream-repository-locations","title":"Adding remote (new upstream repository locations)","text":"

              To add a new upstream repository, use the remote add command. For example:

              git remote add ESCOMP https://github.com/ESCOMP/CAM-SIMA\ngit fetch --tags ESCOMP\n

              You should see messages much like a new clone when you execute the git fetch command. Note that you can call the new remote anything, in this example we are calling it ESCOMP.

              "},{"location":"development/git-basics/#updating-your-branch-to-latest-development","title":"Updating your branch to latest development","text":"

              Note that while this section explains how to update your local branch to the ESCOMP/CAM-SIMA/development branch, the instructions can easily be generalized for any branch from any upstream remote.

              Before starting, you should have either:

              • A fresh clone of your fork with the branch you wish to update checked out (see Create a new clone and Working with branches.
              • An existing clone with the branch you wish to update checked out and in a clean state (i.e., make sure you do a git commit and that git status shows no modified files).

              Add the upstream remote, if you have not already done so (see Adding remotes).

              Merge the specific remote/branch into your branch. In this example, it is ESCOMP/development

              git fetch ESCOMP\ngit merge ESCOMP/development\n
              "},{"location":"development/git-basics/#comparing-differences-using-git-diff","title":"Comparing differences using git diff","text":"

              If you have a git clone, you can view differences between commits or tags. As far as git diff is concerned, a commit hash is the same as a tag so in the examples below will use <tag>.

              • To see the full difference between two tags (i.e., a changeset):
                git diff <tag1> <tag2>\n
              • To see the full difference between the current checkout (sandbox) and a tag:
                git diff <tag>\n
              • To see only the names of files that are different:
                git diff --name-only <tag> [ <tag2> ]\n
              • To see the difference in one or more specific files:
                git diff <tag> [ <tag2> ] -- <path_to_file1> <path_to_file2>\n
              "},{"location":"development/git-basics/#configuring-and-using-a-graphical-difference-tool","title":"Configuring and using a graphical difference tool","text":"

              git has a command, difftool, that can run a graphical tool on each file that is different between two commits.

              • To configure opendiff as the graphical difference tool:
                git config --global diff.tool opendiff\n
              • To see the available graphical difference tools:
                git difftool --tool-help\n
              • To run difftool on <file1> and <file2>
                git difftool <tag> [ <tag2> ] -- <path_to_file1> <path_to_file2>\n
              • To optionally run difftool on all files in a changeset (answer 'y' or 'n' for each file):
                git difftool <tag> [ <tag2> ]\n
              • To run difftool on all files in a changeset (i.e., same as 'y' for every file):
                yes | git difftool <tag> [ <tag2> ]\n
              "},{"location":"development/git-basics/#using-diffmerge","title":"Using diffmerge","text":"

              This section contains the complete modifications needed for using the graphical tool diffmerge for \"git difftool\" and \"git mergetool\"

              There is a tool called diffmerge which enables both side-by-side comparison of edited files in git as well as providing a three way-pane for editing merge conflicts. This tool is available for download at: https://sourcegear.com/diffmerge/. It has been installed on izumi and derecho by the system administrators in public areas, so you don't need to download it for those machines.

              To use the differencing tool type: git difftool

              If after a git merge the git command says there are conflicts, then you may type git mergetool to allow you to resolve the conflicts and complete the merge. The mergetool with diffmerge properly installed will display three panes. From left to right these panes are: 

              Code being merged in          Merged code               Your code\n

              The panes are opened in a visual editor, and any changes you make in the middle pane, may be saved by selecting the save icon at the top of the screen and then exiting the window. This will finalize the merge for that particular file.

              For a presentation which can be used as a tutorial, you may refer to: Presentation on diffmerge tool for git

              The following modifications may be copy/pasted into the user's .gitconfig file which resides in the home directory. Since the potential for making an editing mistake is possible, it is recommended that a copy be made of the .gitconfig file prior to these edits in case an error is discovered.

              If you have problems, check out the tips at the bottom of this page

              [diff]\n\u00a0 \u00a0 \u00a0 \u00a0 tool = diffmerge\n\u00a0 \u00a0 \u00a0 \u00a0 algorithm = histogram\n[difftooldiffmerge]\n\u00a0 \u00a0 \u00a0 \u00a0 cmd = diffmerge \\\"$LOCAL\\\" \\\"$REMOTE\\\"\n[difftool \"diffmerge\"]\n\u00a0 \u00a0 \u00a0 \u00a0 cmd = diffmerge \\\"$LOCAL\\\" \\\"$REMOTE\\\"\n[difftool]\n\u00a0 \u00a0 \u00a0 \u00a0 prompt = false\n[push]\n\u00a0 \u00a0 \u00a0 \u00a0 default = simple\n[merge]\n\u00a0 \u00a0 \u00a0 \u00a0 tool = diffmerge\n\u00a0 \u00a0 \u00a0 \u00a0 ff = false\n[mergetool \"diffmerge\"]\n\u00a0 \u00a0 \u00a0 \u00a0 cmd = diffmerge --merge --result=$MERGED $REMOTE $BASE $LOCAL\n[mergetool]\n\u00a0 \u00a0 \u00a0 \u00a0 keepBackup = false\n

              Useful tips

              • On derecho, if you get an error message about diffmerge not being loaded, make sure you are using ncarenv/23.09 or newer (to check say \"module list\")
              • If you do not get a response when clicking on the \"git mergetool\" window, make sure you find the \"conflict\" popup and hit \"exit\" on it
              • If your git mergetool gives blank windows and says: \"Files are identical or equivalent under the current RuleSet\", hit OK and then go to the File dropdown menu and select \"Reload\". This filled in the blank windows for the user with this error.
                • CAUTION: Check your file carefully if you do this, as a second user who did this discovered that all of the changes which were automatically merged did not reside in the final saved version.
              "},{"location":"development/git-faq/","title":"git & GitHub FAQ","text":""},{"location":"development/git-faq/#q-how-can-i-clone-someones-git-clone-to-my-directory-on-the-same-machine","title":"Q: How can I clone someone's git clone to my directory on the same machine?","text":"

              A: git clone <path_name> [<local_dir>] where <path_name> is the location of the source git clone and the optional <local_dir> is the name of the clone (default is a local directory of the same name as the original).

              "},{"location":"development/git-faq/#q-how-can-i-clone-someones-git-clone-to-my-directory-on-a-different-machine","title":"Q: How can I clone someone's git clone to my directory on a different machine?","text":"

              A: git clone ssh://<user>@<machine>:<path_name> [<local_dir>] where <user> is your user name on the remote machine, <machine> is the machine name (e.g., derecho.hpc.ucar.edu), <path_name> is the location of the source git clone on the remote machine, and the optional <local_dir> is the name of the clone (default is a local directory of the same name as the original).

              "},{"location":"development/git-faq/#q-how-can-i-look-at-someones-pr-code","title":"Q: How can I look at someone's PR code?","text":"

              A: There a a few ways to do this:

              • On GitHub (like looking at any other code on GitHub)
              • Add the PR fork to my remote (allows using tools such as git diff or git difftool with your existing branches or development)
              • As a clone (standalone clone on your machine).

              A first step is to find the link to the fork's branch. Just below the PR is a line that starts with a colored oval (e.g., \"Open\" in green) and looks something like:

              Octocat wants to merge 123 commits into ESCOMP:development from Octocat:amazing-new-feature\n

              Clicking on the last part (Octocat:amazing-new-feature) will take you to the correct branch where you can browse the code (the first method above). If you want to download that code, click the green \"Code\" button and then click the clipboard icon. Be sure to take note of the branch name.

              • To load this code into your clone, cd to your clone directory, add the PR fork as a new remote, and checkout the branch. For instance:
                  git remote add octocat https://github.com/octocat/CAM-SIMA.git\n    git fetch --no-tags octocat\n    git checkout octocat/amazing-new-feature\n

              Instead of the checkout you can also do diffs:

                  git difftool origin/development octocat/amazing-new-feature\n
              • If you want to make a new clone with the PR code, simply do:
                  git clone -b amazing-new-feature octocat https://github.com/octocat/CAM-SIMA.git octocat_cam\n    cd octocat_cam-sima\n
              "},{"location":"development/git-faq/#q-why-do-pull-request-pr-code-review-take-so-long","title":"Q: Why do Pull Request (PR) code review take so long?","text":"

              A: A code review must fulfill three purposes:

              • Code reviewers must make sure that new and/or changed code does not affect any currently-supported functionality (i.e., it cannot break anything).
                • While regression tests will catch many of these issues, reviewers must also check for usage of or reliance on deprecated code, and also for any code which is not supported on all platforms and compilers used by the CESM community.
              • Code reviewers must make sure that any new functionality or changes are implemented correctly and at least somewhat efficiently. They must also ensure that important changes are tested against future regressions.
              • The CAM SE team is almost always engaged in several projects to implement new CAM functionality along with supporting infrastructure. Each CAM SE usually looks at each PR in order to prevent new code from interfering with those plans.

              The first two steps are usually completed by a single SE although SEs engaged in a final review will often find missed errors. This is similar to peer reviewers finding problems with a paper even after reviews done by colleagues.

              "},{"location":"development/git-faq/#q-how-do-i-update-my-branch-to-the-current-cam_development","title":"Q: How do I update my branch to the current cam_development?","text":"

              A: see this section

              "},{"location":"development/git-fleximod/","title":"git-fleximod","text":""},{"location":"development/git-fleximod/#populate-andor-update-your-externals","title":"Populate and/or update your externals","text":"

              Run bin/git-fleximod update

              The process will either:

              • complete and you're good to go
              • indicate to you that modifications have been made in certain modules
                • this means you should either go commit those mods or remove them
                • you can also (if you're sure!) run bin/git-fleximod update -f to force overwrite all externals
              "},{"location":"development/git-fleximod/#update-external-repo-or-tag","title":"Update external repo or tag","text":"

              To add or update an external repo to CAM-SIMA, the following steps must be done:

              1. Modify the .gitmodules file at the head of CAM-SIMA to add or update the external. Explanations for what each entry in the .gitmodules file is can be found on Github here
              2. Once the .gitmodules file has been updated, go to the head of CAM-SIMA and run bin/git-fleximod update. This will bring in the new external code into your local repo (but will not commit them).
              3. Once you are ready to officially commit the changes, then make sure to commit both the modified .gitmodules file, and the updated submodule itself. An easy way to make sure you have commited everything is to run git status and make sure there are no files or directories that have been modified but are still un-staged.

              Once all of that is done then congrats! A new external has been successfully added/updated.

              "},{"location":"development/tool-recommendations/","title":"Tool recommendations","text":"

              This page lists recommendations for various tools one might use during CAM-SIMA development.

              Please note that these standards are currently in development, and thus any of them could change at any time.

              "},{"location":"development/tool-recommendations/#version-control","title":"Version Control","text":"

              The recommended version control tool is git.

              "},{"location":"development/tool-recommendations/#repository-hosting","title":"Repository Hosting","text":"

              We recommend Github for hosting software repositories

              "},{"location":"development/tool-recommendations/#cicd","title":"CI/CD","text":"

              When possible, we recommend running any CI/CD workflows using Github Actions.

              "},{"location":"development/tool-recommendations/#container-software","title":"Container Software","text":"

              We recommend using Docker for building and running containers.

              More to Come!

              "},{"location":"usage/creating-a-case/","title":"Creating, configuring, and running a case","text":"

              Because CAM-SIMA uses the CIME build system, creating and configuration a case is mostly the same as it was in CAM7 (and in CESM). But! We're going to assume you know nothing.

              "},{"location":"usage/creating-a-case/#1-run-create_newcase","title":"1. Run create_newcase","text":"

              Before you create a case:

              1. Choose the correct machine to run your case on (if this is a high-resolution or long run, you will likely want to choose derecho!)
              2. Consult the git basics if you need a new clone
              3. Make sure you have run bin/git-fleximod update to update and/or populate your externals
              4. Navigate to $CAM-SIMA/cime/scripts

              Assuming you have completed the above steps and have a nice checkout of a development branch of CAM-SIMA, let's go!

              Here is the (basic) structure of the command for create_newcase:

              ./create_newcase --case <CASEDIR> --compset <COMPSET_NAME> --res <RESOLUTION> --compiler <COMPILER> --project <PROJECT KEY> --run-unsupported\n

              To break that down piece by piece:

              • --case <CASEDIR> (optional): The case argument specifies the full path to the directory you would like to be created, in which your case will reside.
                • pro tip: if you choose to put your case directory in your scratch directory, your run and bld directories will be automatically placed within the case directory
                  • on derecho, this means: --case /glade/derecho/scratch/<username>/<case_name>
                  • on izumi, this means: --case /scratch/cluster/<username>/<case_name>
                • if you do not specify a case directory, it will default to your $CAM-SIMA/cime/scripts directory (and, eventually, your run and bld directories will default to a new directory in your scratch space)
              • --compset <COMPSET_NAME> (required): the compset (or \"component set\") tells CIME which base-level configurations you want to run with (including model versions and input data sets)
                • The compset supplied to the create_newcase command can be either:
                  • The long name of a compset OR
                  • The alias (if it exists) for the compset
                • Compset options and aliases can be found in here ($CAM-SIMA/cime_config/config_compsets.xml)
                • The structure of a compset long name is <initialization time>_<atmosphere model>_<land model>_<sea-ice model>_<ocean model>_<river runoff model>_<land ice model>_<wave model>
                  • Additional context: for each of the seven model components of the compset configuration, there can be three different implementations:
                    • active (prognostic, full): solves a complex set of equations to describe model's behavior (must solve all equations on a numerical grid)
                    • data: version that sends/receives same variables to/from other models, but read from files rather than computed from equations (reduces feedback within a system)
                    • stub: occupies the required place in the driver but does not send or receive data
              • --res <RESOLUTION> (required): the resolution specifies both the dycore being run and the grid.
                • The structure of the resolution argument is: <atmosphere>_<ocean>_<mask>
                • The resolution determines the dycore:
                  • A grid resolution that looks like \"ne*\" indicates that this is spectral element (SE) cube sphere grid (SE dycore)
                  • A grid resolution that looks like \"C*\" indicates that this is the FV3 (finite volume cubed sphere) dycore [not yet implemented in CAM-SIMA]
                  • A grid resolution that looks like \"mpasa*\" indicates that this is an MPAS (Model for Prediction Across Scales) grid (MPAS dycore) [work-in-progress in CAM-SIMA]
                • You can find the options for these model grid aliases in $CAM-SIMA/ccs_config/modelgrid_aliases_nuopc.xml
                • Most often, for testing CAM-SIMA, we use a coarse SE grid like ne3pg3_ne3pg3_mg37 or ne5_ne5_mg37
              • --compiler <COMPILER> (optional): the compiler you wish to use
                • Options:
                  • on derecho:
                    • intel (default)
                    • gnu
                    • nvhpc
                  • on izumi:
                    • intel (default) - NOTE: intel is not to be trusted on izumi (it's a known-buggy version)
                    • gnu
                    • nag - NOTE: until the nag version is updated beyond 7.0, CAM-SIMA won't work with nag
              • --project <PROJECT KEY>: a project key to charge to
                • Only needed on derecho
                • You can see the project keys you can use here (click on your primary group)
                  • You can also see how much of the allocation has been used/is left by navigating to Reports -> Project Search and searching for the key
              • --run-unsupported: this flag is mostly in place to indicate to scientists that the configuration they're choosing to run is not scientifically supported.

              Given all that, let's say you run the following command on izumi (assuming you're this mysterious person \"courtneyp\"):

              ./create_newcase --case /scratch/cluster/courtneyp/kessler-ne5-gnu-0722 --compset FKESSLER --res ne5_ne5_mg37 --compiler gnu --run-unsupported\n

              What will happen is that a new case directory will be created here: /scratch/cluster/courtneyp/kessler-ne5-gnu-0722 that will be configured to run the FKESSLER compset (long name: 2000_CAM%KESSLER_SLND_SICE_SOCN_SROF_ SGLC_SWAV) with the SE dycore and the ne5 grid.

              "},{"location":"usage/creating-a-case/#2-configure-the-case","title":"2. Configure the case","text":"

              Navigate into your shiny new case directory and run ./case.setup

              From here, there are A LOT of configuration options. We'll highlight a few here that you'll use often.

              "},{"location":"usage/creating-a-case/#xml-configurations","title":"XML configurations","text":"

              Many configurable settings can be found within the env_*.xml files in your case directory. In order to protect again typos, it's not advised to edit those directly. Instead, you will run ./xmlchange commands to alter those settings (and ./xmlquery can give you the current setting). You can find additional information on the configurable xml variables here. A few to highlight:

              • CAM_CONFIG_OPTS: This is where we tell CAM-SIMA what physics scheme(s) we wish to run, as well as if we wish to run the null dycore.
                • FKESSLER will default to --physics-suites kessler --analytic_ic, which means we're running the suite_kessler.xml SDF with the SE dycore (with analytic initial conditions - no need to supply an ncdata initial conditions file)
                • If you instead want to run a test with the null dycore, you'll want to change this with ./xmlchange CAM_CONFIG_OPTS = \"--physics-suites kessler --dyn none\"
              • CAM_LINKED_LIBS: Hopefully, you won't have to change this often; however, if you are getting linking errors during your build, try turning off the linked libraries (./xmlchange CAM_LINKED_LIBS='')
              • DOUT_S: This is the flag to archive log files of successful runs. During development, we advise turning this off so your log files don't disappear on you (./xmlchange DOUT_S=False)
              • STOP_OPTION & STOP_N: How long you are going to run the model. If STOP_N is 8 and STOP_OPTION is \"ndays\", you are setting the model up to run for 8 days.
                • the options for STOP_OPTION are: 'nsteps', 'nseconds', 'nminutes', 'nhours', 'ndays', 'nmonths', 'nyears'
                • STOP_N is an integer
                • NOTE: if you are running the ncdata_check with snapshot files, keep the number of timesteps you are running at or below the number of timesteps on the snapshot files
                • NOTE #2: These configurations can be updated without rebuilding the case
              • DEBUG: A flag to turn on debug mode for the run (runs without compiler optimizations) - this is very useful during development! It defaults to \"False\", so you can turn it on with ./xmlchange DEBUG=True
              • RUNDIR: this is the path to the run directory for your case. The bld directory will exist one level up.

              If you run ./xmlchange, the output will tell you if you need to re-set up and/or do a clean build of your case.

              "},{"location":"usage/creating-a-case/#namelist-configurations","title":"Namelist configurations","text":"

              There are countless namelist configuration options. These can be modified by updating the user_nl_cam file in your case directory. Namelist options can be updated without rebuilding the case.

              • How to configure the namelist for history/output
              • Information on namelist reader generation
              • You can see the full CAM6 list of namelist options here, keeping in mind that most of those options have not yet been ported to CAM-SIMA
              "},{"location":"usage/creating-a-case/#3-build-the-case","title":"3. Build the case","text":"

              Run ./case.build

              The console output will tell you the progress of the build. If you get to MODEL BUILD HAS FINISHED SUCCESSFULLY, hooray!

              Debugging tips

              "},{"location":"usage/creating-a-case/#4-run-the-case","title":"4. Run the case","text":"

              Run ./case.submit

              The job will be submitted to the system's queue. You can see the status of your job with the qstat command. Once it is finished, the log files will be in the run directory (unless it ran successfully AND your archiver is on)

              Debugging tips

              "},{"location":"usage/history/","title":"History & model output","text":""},{"location":"usage/history/#configuring-the-namelist","title":"Configuring the namelist","text":"

              The syntax for updating a the configuration for a history file is:

              <namelist_option>;<volume>: <setting>\n

              Possible namelist options for your user_nl_cam:

              • hist_add_avg_fields: add average fields to a specified history file
              • hist_add_inst_fields: add instantaneous fields to a specified history file
              • hist_add_min_fields: add minimum fields to a specified history file
              • hist_add_max_fields: add maximum fields to a specified history file
              • hist_remove_fields: remove a given field from a specified history file
              • hist_file_type: type of file (options are \"history\", \"satellite\", and \"initial_value\") - defaults to \"history\"
              • hist_max_frames: maximum number of samples written to a specified history file (after which a new one will be created)
              • hist_output_frequency: frequency with which to write samples to a specified history file (syntax is <integer>*<multiplier> like 3*nhours)
              • hist_precision: precision of the specified history file (options are \"REAL32\" and \"REAL64\") - defaults to \"REAL32\"
              • hist_write_nstep0: logical for whether or not to write the nstep = 0 sample (defaults to .false.)
              • hist_filename_template: template filename for the specified history file
                • Defaults to \"%c.cam.%u.%y-%m-%d-%s.nc\" where \"%c\" is the case name, \"%u\" is the volume, \"%y\" is the year, \"%m\" is the month, \"%d\" is the day, and \"%s\" is the number of seconds since midnight GMT, with the timestamp itself representing the model time when the file is created.

              Example Take the following sample user_nl_cam:

              hist_output_frequency;h1: 5*ndays\nhist_max_frames;h1: 3\nhist_add_inst_fields;h1: U\nhist_add_inst_fields;h1: V, Q\nhist_precision;h1: REAL64\nhist_filename_spec;h1: my-history-file%m-%d\nhist_write_nstep0;h1: .false.\n

              It will be parsed by hist_config.py and this will be the relevant section of atm_in:

              &hist_config_arrays_nl\n    hist_num_inst_fields = 3\n    hist_num_avg_fields = 2\n    hist_num_min_fields = 0\n    hist_num_max_fields = 0\n    hist_num_var_fields = 0\n/\n\n&hist_file_config_nl\n    hist_volume = 'h0'\n    hist_avg_fields = 'T', 'Q'\n    hist_max_frames = 1\n    hist_output_frequency = '1*month'\n    hist_precision = 'REAL32'\n    hist_file_type = 'history'\n    hist_filename_spec = '%c.cam.%u.%y-%m-%d-%s.nc'\n    hist_write_nstep0 = .false.\n/\n\n&hist_file_config_nl\n    hist_volume = 'h1'\n    hist_inst_fields = 'U', \u2018V\u2019, \u2018Q\u2019\n    hist_max_frames = 3\n    hist_output_frequency = '5*ndays'\n    hist_precision = 'REAL64'\n    hist_file_type = 'history'\n    hist_filename_spec = 'my-history-file%m-%d'\n    hist_write_nstep0 = .false.\n/\n

              In plain English, a one-month run with these history configuration will result in a total of three files that will look something like these:

              • my-history-file01-06.nc
                • This file will contain instantaneous output for U, V, and Q (eastward_wind, northward_wind, and water vapor)
                • It will contain three frames, one at each of the following times:
                  • 0001-01-06 (time=5)
                  • 0001-01-11 (time=10)
                  • 0001-01-16 (time=15)
              • my-history-file01-21.nc
                • This file will contain instantaneous output for U, V, and Q (eastward_wind, northward_wind, and water vapor)
                • It will contain three frames, one at each of the following times:
                  • 0001-01-21 (time=20)
                  • 0001-01-26 (time=25)
                  • 0001-01-31 (time=30)
              • .cam.h0a.0001-02-01-00000.nc
                • This file will contain averaged output for T and Q (air_temperature and water vapor)
                • It will have one frame with the time calculated at the midpoint of the month
                "},{"location":"usage/history/#adding-a-diagnostic-field-to-the-cam-sima-source-code","title":"Adding a diagnostic field to the CAM-SIMA source code","text":"

                During init time, fields can be added to the possible field list with a call to history_add_field:

                history_add_field(diagnostic_name, standard_name, vdim_name, avgflag, units, gridname, flag_xyfill, mixing_ratio)

                Field Optional? Type Description diagnostic_name No string diagnostic name for the field - will be the name in netcdf output file standard_name No string CCPP standard name for the variable vdim_name No string vertical dimension: 'horiz_only' for no vertical dimension; 'lev' for vertical_layer_dimension; 'ilev' for vertical_interface_dimension avgflag No string default average flag; options: 'avg', 'lst' (instantaneous), 'min', 'max', 'var' (standard deviation) units No string variable units gridname Yes string gridname on which the variable's data is mapped (defaults to the physics grid) flag_xyfill Yes string fill value for variable values mixing_ratio Yes string constituent mixing ratio type ('wet' or 'dry'); not set to anything if not passed in

                Example:

                call history_add_field('Q', 'water_vapor_mixing_ratio_wrt_moist_air_and_condensed_water', 'lev', 'avg', 'kg kg-1', mixing_ratio='wet')\n

                It's important to avoid adding calls to history_add_field to the CCPP-ized physics schemes (to keep them portable). Instead, create a new diagnostics scheme and place that in the diagnostics folder of the atmospheric_physics repository. The history_add_field call will be in the init phase.

                "},{"location":"usage/history/#outputting-a-diagnostic-field-to-the-cam-sima-source-code","title":"Outputting a diagnostic field to the CAM-SIMA source code","text":"

                After init time, a variable's current values can be captured for output with a call to history_out_field:

                history_out_field(diagnostic_name, field_values)

                Field Optional? Type Description diagnostic_name No string diagnostic name for the field - will cause an error if the diagnostic name was not added via history_add_field field_values No real variable values to be stored in the history buffer(s)

                Example:

                call history_out_field('Q', const_array(:,:,const_idx))\n

                It's important to avoid adding calls to history_add_field to the CCPP-ized physics schemes (to keep them portable). Instead, create a new diagnostics scheme and place that in the diagnostics folder of the atmospheric_physics repository. The history_out_field call(s) will likely be in the run phase.

                "},{"location":"usage/history/#using-the-output","title":"Using the output","text":"

                The output files can be found in your run directory. They are in netCDF format.

                See the ADF for lots that you can do with your results!

                A few useful commands for inspecting netCDF data:

                • To output the header/metadata information for the file (includes list of variables on the file):
                ncdump -h file.nc\n
                • To output the data for a specific variable on the file:
                ncdump -v <varname> file.nc\n
                • To compare two files*:
                cprnc <file1_path> <file2_path>\n
                • To get a slice of one file (subset of time samples):
                ncks -F -d time,<start time>,<end time>,<step> <snapshot_file> <output_file_name>\n

                Example to get time samples 3-5 from a file onto a new file called \"split-file.nc\":

                ncks -F -d time,3,5,1 file.nc split-file.nc\n
                • To diff a variable between two files (and output the results to a new netcdf file):
                ncdiff -v <field_name> -o <output_file> <file1> <file2>\n

                *cprnc can be found:

                • on derecho: /glade/p/cesmdata/cseg/tools/cime/tools/cprnc/cprnc
                • on izumi: /fs/cgd/csm/tools/cime/tools/cprnc/cprnc
                "}]} \ No newline at end of file diff --git a/site/sitemap.xml.gz b/site/sitemap.xml.gz index 1eed619123494fd9ce9f6a5872cac407beda5a8e..ab1d30e8fd849bf31e8d2bf25d754ee5a1fb0249 100644 GIT binary patch delta 13 Ucmb=gXP58h;7FLUU?O`303AaFXaE2J delta 13 Ucmb=gXP58h;CSFReh($ diff --git a/site/stylesheets/extra.css b/site/stylesheets/extra.css index 6bf886cf..7eacbd87 100644 --- a/site/stylesheets/extra.css +++ b/site/stylesheets/extra.css @@ -1,3 +1,3 @@ .md-grid { - max-width: initial; + max-width: 90% } diff --git a/site/usage/creating-a-case/index.html b/site/usage/creating-a-case/index.html new file mode 100644 index 00000000..77a91bd0 --- /dev/null +++ b/site/usage/creating-a-case/index.html @@ -0,0 +1,1150 @@ + + + + + + + + + + + + + + + + + + + + + + + Creating, configuring, and running a case - CAM-SIMA + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
                + +
                + + + + + + +
                + + +
                + +
                + + + + + + +
                +
                + + + +
                +
                +
                + + + + + +
                +
                +
                + + + +
                +
                +
                + + + +
                +
                +
                + + + +
                +
                + + + + +

                Creating, configuring, and running a case

                +

                Because CAM-SIMA uses the CIME build system, creating and configuration a case is mostly the same as it was in CAM7 (and in CESM). But! We're going to assume you know nothing.

                +

                1. Run create_newcase

                +

                Before you create a case:

                +
                  +
                1. Choose the correct machine to run your case on (if this is a high-resolution or long run, you will likely want to choose derecho!)
                  2. +
                2. Consult the git basics if you need a new clone
                  3. +
                3. Make sure you have run bin/git-fleximod update to update and/or populate your externals
                  4. +
                4. Navigate to $CAM-SIMA/cime/scripts
                  5. +
                +

                Assuming you have completed the above steps and have a nice checkout of a development branch of CAM-SIMA, let's go!

                +

                Here is the (basic) structure of the command for create_newcase:

                +
                ./create_newcase --case <CASEDIR> --compset <COMPSET_NAME> --res <RESOLUTION> --compiler <COMPILER> --project <PROJECT KEY> --run-unsupported
+
                +

                To break that down piece by piece:

                +
                  +
                • --case <CASEDIR> (optional): The case argument specifies the full path to the directory you would like to be created, in which your case will reside.
                    +
                  • pro tip: if you choose to put your case directory in your scratch directory, your run and bld directories will be automatically placed within the case directory
                      +
                    • on derecho, this means: --case /glade/derecho/scratch/<username>/<case_name>
                      • +
                    • on izumi, this means: --case /scratch/cluster/<username>/<case_name>
                      • +
                    +
                    • +
                  • if you do not specify a case directory, it will default to your $CAM-SIMA/cime/scripts directory (and, eventually, your run and bld directories will default to a new directory in your scratch space)
                    • +
                  +
                  • +
                • --compset <COMPSET_NAME> (required): the compset (or "component set") tells CIME which base-level configurations you want to run with (including model versions and input data sets)
                    +
                  • The compset supplied to the create_newcase command can be either:
                      +
                    • The long name of a compset OR
                      • +
                    • The alias (if it exists) for the compset
                      • +
                    +
                    • +
                  • Compset options and aliases can be found in here ($CAM-SIMA/cime_config/config_compsets.xml)
                    • +
                  • The structure of a compset long name is <initialization time>_<atmosphere model>_<land model>_<sea-ice model>_<ocean model>_<river runoff model>_<land ice model>_<wave model>
                      +
                    • Additional context: for each of the seven model components of the compset configuration, there can be three different implementations:
                        +
                      • active (prognostic, full): solves a complex set of equations to describe model's behavior (must solve all equations on a numerical grid)
                        • +
                      • data: version that sends/receives same variables to/from other models, but read from files rather than computed from equations (reduces feedback within a system)
                        • +
                      • stub: occupies the required place in the driver but does not send or receive data
                        • +
                      +
                      • +
                    +
                    • +
                  +
                  • +
                • --res <RESOLUTION> (required): the resolution specifies both the dycore being run and the grid.
                    +
                  • The structure of the resolution argument is: <atmosphere>_<ocean>_<mask>
                    • +
                  • The resolution determines the dycore:
                      +
                    • A grid resolution that looks like "ne*" indicates that this is spectral element (SE) cube sphere grid (SE dycore)
                      • +
                    • A grid resolution that looks like "C*" indicates that this is the FV3 (finite volume cubed sphere) dycore [not yet implemented in CAM-SIMA]
                      • +
                    • A grid resolution that looks like "mpasa*" indicates that this is an MPAS (Model for Prediction Across Scales) grid (MPAS dycore) [work-in-progress in CAM-SIMA]
                      • +
                    +
                    • +
                  • You can find the options for these model grid aliases in $CAM-SIMA/ccs_config/modelgrid_aliases_nuopc.xml
                    • +
                  • Most often, for testing CAM-SIMA, we use a coarse SE grid like ne3pg3_ne3pg3_mg37 or ne5_ne5_mg37
                    • +
                  +
                  • +
                • --compiler <COMPILER> (optional): the compiler you wish to use
                    +
                  • Options:
                      +
                    • on derecho:
                        +
                      • intel (default)
                        • +
                      • gnu
                        • +
                      • nvhpc
                        • +
                      +
                      • +
                    • on izumi:
                        +
                      • intel (default) - NOTE: intel is not to be trusted on izumi (it's a known-buggy version)
                        • +
                      • gnu
                        • +
                      • nag - NOTE: until the nag version is updated beyond 7.0, CAM-SIMA won't work with nag
                        • +
                      +
                      • +
                    +
                    • +
                  +
                  • +
                • --project <PROJECT KEY>: a project key to charge to
                    +
                  • Only needed on derecho
                    • +
                  • You can see the project keys you can use here (click on your primary group)
                      +
                    • You can also see how much of the allocation has been used/is left by navigating to Reports -> Project Search and searching for the key
                      • +
                    +
                    • +
                  +
                  • +
                • --run-unsupported: this flag is mostly in place to indicate to scientists that the configuration they're choosing to run is not scientifically supported.
                  • +
                +

                Given all that, let's say you run the following command on izumi (assuming you're this mysterious person "courtneyp"):

                +
                ./create_newcase --case /scratch/cluster/courtneyp/kessler-ne5-gnu-0722 --compset FKESSLER --res ne5_ne5_mg37 --compiler gnu --run-unsupported
+
                +

                What will happen is that a new case directory will be created here: /scratch/cluster/courtneyp/kessler-ne5-gnu-0722 that will be configured to run the FKESSLER compset (long name: 2000_CAM%KESSLER_SLND_SICE_SOCN_SROF_ SGLC_SWAV) with the SE dycore and the ne5 grid.

                +

                2. Configure the case

                +

                Navigate into your shiny new case directory and run ./case.setup

                +

                From here, there are A LOT of configuration options. We'll highlight a few here that you'll use often.

                +

                XML configurations

                +

                Many configurable settings can be found within the env_*.xml files in your case directory. In order to protect again typos, it's not advised to edit those directly. Instead, you will run ./xmlchange commands to alter those settings (and ./xmlquery can give you the current setting). You can find additional information on the configurable xml variables here. A few to highlight:

                +
                  +
                • CAM_CONFIG_OPTS: This is where we tell CAM-SIMA what physics scheme(s) we wish to run, as well as if we wish to run the null dycore.
                    +
                  • FKESSLER will default to --physics-suites kessler --analytic_ic, which means we're running the suite_kessler.xml SDF with the SE dycore (with analytic initial conditions - no need to supply an ncdata initial conditions file)
                    • +
                  • If you instead want to run a test with the null dycore, you'll want to change this with ./xmlchange CAM_CONFIG_OPTS = "--physics-suites kessler --dyn none"
                    • +
                  +
                  • +
                • CAM_LINKED_LIBS: Hopefully, you won't have to change this often; however, if you are getting linking errors during your build, try turning off the linked libraries (./xmlchange CAM_LINKED_LIBS='')
                  • +
                • DOUT_S: This is the flag to archive log files of successful runs. During development, we advise turning this off so your log files don't disappear on you (./xmlchange DOUT_S=False)
                  • +
                • STOP_OPTION & STOP_N: How long you are going to run the model. If STOP_N is 8 and STOP_OPTION is "ndays", you are setting the model up to run for 8 days.
                    +
                  • the options for STOP_OPTION are: 'nsteps', 'nseconds', 'nminutes', 'nhours', 'ndays', 'nmonths', 'nyears'
                    • +
                  • STOP_N is an integer
                    • +
                  • NOTE: if you are running the ncdata_check with snapshot files, keep the number of timesteps you are running at or below the number of timesteps on the snapshot files
                    • +
                  • NOTE #2: These configurations can be updated without rebuilding the case
                    • +
                  +
                  • +
                • DEBUG: A flag to turn on debug mode for the run (runs without compiler optimizations) - this is very useful during development! It defaults to "False", so you can turn it on with ./xmlchange DEBUG=True
                  • +
                • RUNDIR: this is the path to the run directory for your case. The bld directory will exist one level up.
                  • +
                +

                If you run ./xmlchange, the output will tell you if you need to re-set up and/or do a clean build of your case.

                +

                Namelist configurations

                +

                There are countless namelist configuration options. These can be modified by updating the user_nl_cam file in your case directory. +Namelist options can be updated without rebuilding the case.

                + +

                3. Build the case

                +

                Run ./case.build

                +

                The console output will tell you the progress of the build. If you get to MODEL BUILD HAS FINISHED SUCCESSFULLY, hooray!

                +

                Debugging tips

                +

                4. Run the case

                +

                Run ./case.submit

                +

                The job will be submitted to the system's queue. You can see the status of your job with the qstat command. Once it is finished, the log files will be in the run directory (unless it ran successfully AND your archiver is on)

                +

                Debugging tips

                + + + + + + + + + + + + + +
                +
                + + + +
                + +
                + + + +
                +
                +
                +
                + + + + + + + + + + \ No newline at end of file diff --git a/site/usage/history/index.html b/site/usage/history/index.html new file mode 100644 index 00000000..220c1f9c --- /dev/null +++ b/site/usage/history/index.html @@ -0,0 +1,1199 @@ + + + + + + + + + + + + + + + + + + + + + History & model output - CAM-SIMA + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
                + +
                + + + + + + +
                + + +
                + +
                + + + + + + +
                +
                + + + +
                +
                +
                + + + + + +
                +
                +
                + + + +
                +
                +
                + + + +
                +
                +
                + + + +
                +
                + + + + +

                History & model output

                +

                Configuring the namelist

                +

                The syntax for updating a the configuration for a history file is:

                +
                <namelist_option>;<volume>: <setting>
+
                +

                Possible namelist options for your user_nl_cam:

                +
                  +
                • hist_add_avg_fields: add average fields to a specified history file
                  • +
                • hist_add_inst_fields: add instantaneous fields to a specified history file
                  • +
                • hist_add_min_fields: add minimum fields to a specified history file
                  • +
                • hist_add_max_fields: add maximum fields to a specified history file
                  • +
                • hist_remove_fields: remove a given field from a specified history file
                  • +
                • hist_file_type: type of file (options are "history", "satellite", and "initial_value") - defaults to "history"
                  • +
                • hist_max_frames: maximum number of samples written to a specified history file (after which a new one will be created)
                  • +
                • hist_output_frequency: frequency with which to write samples to a specified history file (syntax is <integer>*<multiplier> like 3*nhours)
                  • +
                • hist_precision: precision of the specified history file (options are "REAL32" and "REAL64") - defaults to "REAL32"
                  • +
                • hist_write_nstep0: logical for whether or not to write the nstep = 0 sample (defaults to .false.)
                  • +
                • hist_filename_template: template filename for the specified history file
                    +
                  • Defaults to "%c.cam.%u.%y-%m-%d-%s.nc" where "%c" is the case name, "%u" is the volume, "%y" is the year, "%m" is the month, "%d" is the day, and "%s" is the number of seconds since midnight GMT, with the timestamp itself representing the model time when the file is created.
                    • +
                  +
                  • +
                +

                Example +Take the following sample user_nl_cam:

                +
                hist_output_frequency;h1: 5*ndays
+hist_max_frames;h1: 3
+hist_add_inst_fields;h1: U
+hist_add_inst_fields;h1: V, Q
+hist_precision;h1: REAL64
+hist_filename_spec;h1: my-history-file%m-%d
+hist_write_nstep0;h1: .false.
+
                +

                It will be parsed by hist_config.py and this will be the relevant section of atm_in:

                +
                &hist_config_arrays_nl
+    hist_num_inst_fields = 3
+    hist_num_avg_fields = 2
+    hist_num_min_fields = 0
+    hist_num_max_fields = 0
+    hist_num_var_fields = 0
+/
+
+&hist_file_config_nl
+    hist_volume = 'h0'
+    hist_avg_fields = 'T', 'Q'
+    hist_max_frames = 1
+    hist_output_frequency = '1*month'
+    hist_precision = 'REAL32'
+    hist_file_type = 'history'
+    hist_filename_spec = '%c.cam.%u.%y-%m-%d-%s.nc'
+    hist_write_nstep0 = .false.
+/
+
+&hist_file_config_nl
+    hist_volume = 'h1'
+    hist_inst_fields = 'U', ‘V’, ‘Q’
+    hist_max_frames = 3
+    hist_output_frequency = '5*ndays'
+    hist_precision = 'REAL64'
+    hist_file_type = 'history'
+    hist_filename_spec = 'my-history-file%m-%d'
+    hist_write_nstep0 = .false.
+/
+
                +

                In plain English, a one-month run with these history configuration will result in a total of three files that will look something like these:

                +
                  +
                • my-history-file01-06.nc
                    +
                  • This file will contain instantaneous output for U, V, and Q (eastward_wind, northward_wind, and water vapor)
                    • +
                  • It will contain three frames, one at each of the following times:
                      +
                    • 0001-01-06 (time=5)
                      • +
                    • 0001-01-11 (time=10)
                      • +
                    • 0001-01-16 (time=15)
                      • +
                    +
                    • +
                  +
                  • +
                • my-history-file01-21.nc
                    +
                  • This file will contain instantaneous output for U, V, and Q (eastward_wind, northward_wind, and water vapor)
                    • +
                  • It will contain three frames, one at each of the following times:
                      +
                    • 0001-01-21 (time=20)
                      • +
                    • 0001-01-26 (time=25)
                      • +
                    • 0001-01-31 (time=30)
                      • +
                    +
                    • +
                  +
                  • +
                • .cam.h0a.0001-02-01-00000.nc
                    +
                  • This file will contain averaged output for T and Q (air_temperature and water vapor)
                    • +
                  • It will have one frame with the time calculated at the midpoint of the month
                    • +
                  +
                  • +
                +

                Adding a diagnostic field to the CAM-SIMA source code

                +

                During init time, fields can be added to the possible field list with a call to history_add_field:

                +

                history_add_field(diagnostic_name, standard_name, vdim_name, avgflag, units, gridname, flag_xyfill, mixing_ratio)

                + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
                FieldOptional?TypeDescription
                diagnostic_nameNostringdiagnostic name for the field - will be the name in netcdf output file
                standard_nameNostringCCPP standard name for the variable
                vdim_nameNostringvertical dimension: 'horiz_only' for no vertical dimension; 'lev' for vertical_layer_dimension; 'ilev' for vertical_interface_dimension
                avgflagNostringdefault average flag; options: 'avg', 'lst' (instantaneous), 'min', 'max', 'var' (standard deviation)
                unitsNostringvariable units
                gridnameYesstringgridname on which the variable's data is mapped (defaults to the physics grid)
                flag_xyfillYesstringfill value for variable values
                mixing_ratioYesstringconstituent mixing ratio type ('wet' or 'dry'); not set to anything if not passed in
                +

                Example:

                +
                call history_add_field('Q', 'water_vapor_mixing_ratio_wrt_moist_air_and_condensed_water', 'lev', 'avg', 'kg kg-1', mixing_ratio='wet')
+
                +

                It's important to avoid adding calls to history_add_field to the CCPP-ized physics schemes (to keep them portable). Instead, create a new diagnostics scheme and place that in the diagnostics folder of the atmospheric_physics repository. The history_add_field call will be in the init phase.

                +

                Outputting a diagnostic field to the CAM-SIMA source code

                +

                After init time, a variable's current values can be captured for output with a call to history_out_field:

                +

                history_out_field(diagnostic_name, field_values)

                + + + + + + + + + + + + + + + + + + + + + + + +
                FieldOptional?TypeDescription
                diagnostic_nameNostringdiagnostic name for the field - will cause an error if the diagnostic name was not added via history_add_field
                field_valuesNorealvariable values to be stored in the history buffer(s)
                +

                Example:

                +
                call history_out_field('Q', const_array(:,:,const_idx))
+
                +

                It's important to avoid adding calls to history_add_field to the CCPP-ized physics schemes (to keep them portable). Instead, create a new diagnostics scheme and place that in the diagnostics folder of the atmospheric_physics repository. The history_out_field call(s) will likely be in the run phase.

                +

                Using the output

                +

                The output files can be found in your run directory. They are in netCDF format.

                +

                See the ADF for lots that you can do with your results!

                +

                A few useful commands for inspecting netCDF data:

                +
                  +
                • To output the header/metadata information for the file (includes list of variables on the file):
                  • +
                +
                ncdump -h file.nc
+
                +
                  +
                • To output the data for a specific variable on the file:
                  • +
                +
                ncdump -v <varname> file.nc
+
                +
                  +
                • To compare two files*:
                  • +
                +
                cprnc <file1_path> <file2_path>
+
                +
                  +
                • To get a slice of one file (subset of time samples):
                  • +
                +
                ncks -F -d time,<start time>,<end time>,<step> <snapshot_file> <output_file_name>
+
                +

                Example to get time samples 3-5 from a file onto a new file called "split-file.nc":

                +
                ncks -F -d time,3,5,1 file.nc split-file.nc
+
                +
                  +
                • To diff a variable between two files (and output the results to a new netcdf file):
                  • +
                +
                ncdiff -v <field_name> -o <output_file> <file1> <file2>
+
                +

                *cprnc can be found:

                +
                  +
                • on derecho: /glade/p/cesmdata/cseg/tools/cime/tools/cprnc/cprnc
                  • +
                • on izumi: /fs/cgd/csm/tools/cime/tools/cprnc/cprnc
                  • +
                + + + + + + + + + + + + + +
                +
                + + + +
                + +
                + + + +
                +
                +
                +
                + + + + + + + + + + \ No newline at end of file