Skip to content

Commit

Permalink
add usage and development sections
Browse files Browse the repository at this point in the history
  • Loading branch information
peverwhee committed Jul 24, 2024
1 parent e8be14d commit b089600
Show file tree
Hide file tree
Showing 32 changed files with 1,240 additions and 95 deletions.
4 changes: 2 additions & 2 deletions docs/design/cam-build-process.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# CAM-SIMA Build Process
# Build process

In order to describe the build process, we need to define several source and build directories:

Expand Down Expand Up @@ -38,7 +38,7 @@ Given the context above, the following is the CAM-SIMA build sequence:
- Call the [CCPP Framework](https://ccpp-techdoc.readthedocs.io/en/v6.0.0/) to generate glue code (CAPS), (if required).

## CAM-SIMA source and namelist generation (buildnml) workflow
![text](buildnml_workflow.jpg "CAM-SIMA buildnml workflow")
![text](figures/buildnml_workflow.jpg "CAM-SIMA buildnml workflow")

The diagram above displays everything that occurs when CAM-SIMA's `buildnml` is called by CIME, which occurs after the user calls `preview_namelists`, `case.build`, or `case.submit`.

Expand Down
24 changes: 18 additions & 6 deletions docs/design/cam-run-process.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,11 @@
# CAM-SIMA Run Process
# Run process

![text](run-sequence.gif "CAM-SIMA run sequence")
<figure markdown="span">
![text](figures/run-sequence.gif "CAM-SIMA run sequence"){width="80%"}
<figcaption>CAM-SIMA run sequence*</figcaption>
</figure>

*Static images can be found at the [bottom](#static-run-sequence-images) of this page

## CAM-SIMA API
Upon running `./case.submit` the core CAM-SIMA driver code* is in `$CAM-SIMA/src/control/cam_comp.F90`. This section lays out each of the subroutines within `cam_comp.F90`.
Expand Down Expand Up @@ -51,10 +56,10 @@ The subroutines in `cam_comp.F90` are set up to mirror the phases of the [Common
1. `cam_ctrl_init` (`src/control/cam_control_mod.F90`): Sets the module-level run configuration variables; logs configurations to the atm log
1. `cam_ctrl_set_orbit` (`src/control/cam_control_mod.F90`): Sets the module-level orbital variables
1. `timemgr_init (`src/utils/time_manager.F90`): Initializes the time manager; logs configurations to the atm log
1. `read_namelist` (`src/control/runtime_opts.F90`): Reads all namelists for the run, including auto-generated scheme namelists (see [build process](/design/cam-build-process/#cam-sima-source-and-namelist-generation-buildnml-workflow))
1. `read_namelist` (`src/control/runtime_opts.F90`): Reads all namelists for the run, including auto-generated scheme namelists (see [build process](cam-build-process.md/#cam-sima-source-and-namelist-generation-buildnml-workflow))
1. `cam_ctrl_set_physics_type` (`src/control/cam_control_mod.F90`): sets module-level configuration for variables for simple physics and moist physics schemes; logs configurations to atm log
1. `cam_initfiles_open` (`src/control/cam_initfiles.F90`): Opens initial or restart file, and topography file if specified
1. `cam_register_constituents` (`src/control/cam_comp.F90`): Sets the total number and advected number of [constituents](/design/constituents); currently ALWAYS adds water vapor as constituent (expected by the SE dycore)
1. `cam_register_constituents` (`src/control/cam_comp.F90`): Sets the total number and advected number of [constituents](constituents.md); currently ALWAYS adds water vapor as constituent (expected by the SE dycore)
1. `air_composition_init` (`src/data/air_composition.F90`): Initializes air-composition-dependent model constants
1. `model_grid_init` (`src/dynamics/<dycore>/dyn_grid.F90`): Initializes model grids and decompositions
1. `cam_ccpp_initialize_constituents` (`$CASE/bld/atm/obj/ccpp/cam_ccpp_cap.F90`): initializes the constituent data array; after this point, we cannot add new constituents
Expand Down Expand Up @@ -120,7 +125,7 @@ The routine calls the following subroutines (locations) in this order:

`cam_timestep_final` calls the following subroutines (locations):

1. [History](/design/history) routines. If it's not the last (half) timestep,
1. [History](history.md) routines. If it's not the last (half) timestep,
1. `history_write_files` (`src/history/cam_history.F90`): Writes fields to user-configured history files (if applicable)
1. `history_wrap_up` (`src/history/cam_history.F90`): Closes files and zeros buffers as necessary
1. `phys_timestep_final` (`src/physics/utils/phys_comp.F90`):
Expand All @@ -140,4 +145,11 @@ The routine calls the following subroutines (locations) in this order:

1. `phys_final` (`src/physics/utils/phys_comp.F90`): calls "final" phase of all schemes in the suite definition file (SDF)
1. `stepon_final` (`src/dynamics/<dycore>/stepon.F90`): finalize dycore (doesn't currently do anything)
1. `atm2hub_deallocate` and `hub2atm_deallocate` (`src/control/camsrfexch.F90`): deallocate cam_in/cam_out objects
1. `atm2hub_deallocate` and `hub2atm_deallocate` (`src/control/camsrfexch.F90`): deallocate cam_in/cam_out objects

## *Static run sequence images*

![text](figures/run-sequence-api.png "CAM-SIMA run sequence from CAM API perspective")
![text](figures/run-sequence-physics.png "CAM-SIMA run sequence from physics perspective")
![text](figures/run-sequence-dynamics.png "CAM-SIMA run sequence from dynamics perspective")
![text](figures/run-sequence-history.png "CAM-SIMA run sequence from history perspective")
158 changes: 158 additions & 0 deletions docs/design/ccpp-in-cam-sima.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,158 @@
# CCPP in CAM-SIMA
## Overview
The core Common Community Physics Package (CCPP) documentation can be found [here](https://dtcenter.org/community-code/common-community-physics-package-ccpp/documentation). This section details the code structure and implementation of the CCPP Framework within CAM-SIMA. That said here's a quick overview of the CCPP:

- A CCPP-enabled model consists of the following components (depicted in the diagram below):
- Host model and dynamical core
- CCPP-compliant physics schemes (organized into Suite Definition Files [SDFs])
- Caps generated by the CCPP framework to connect the host model to the physics schemes

![text](figures/ccpp-framework.PNG "CCPP design")

CCPP-compliant physics schemes must adhere to the following criteria:

- Must be broken up into only the following phases:
- *register* (anything in the scheme that must be done before the grid is initialized) - run once at startup
- *init* - run once at startup
- *timestep_init* - run at beginning of every physics timestep
- *run* - run on every timestep
- *timestep_final* - run at the end of every timestep
- *final* - run once at the end of model execution
- Must contain metadata for all input/output/inout variables passed into each phase (see metadata example below)
- Must not have use statements outside of the following allowed CCPP use statements (with the exception of `dependencies`, but we're not getting into that now):
- ccpp_kinds
- ccpp_constituent_prop_mod

*Metadata example*
(snippet taken from `kessler.meta`)

```
[ precl ]
standard_name = total_precipitation_rate_at_surface
long_name = Total precipitation rate at surface
units = m s-1
dimensions = (horizontal_loop_extent)
type = real | kind = kind_phys
intent = out
[ relhum ]
standard_name = relative_humidity
long_name = Relative humidity
units = percent
dimensions = (horizontal_loop_extent, vertical_layer_dimension)
type = real | kind = kind_phys
intent = out
[ scheme_name ]
standard_name = scheme_name
units = none
type = character | kind = len=64
dimensions = ()
intent = out
```

CCPP-compliant physics schemes are organized into suite definition files ([SDFs](https://ccpp-techdoc.readthedocs.io/en/v6.0.0/ConstructingSuite.html?highlight=sdf#constructing-suites)). An SDF tells the framework which schemes will be run in what order. Separating schemes into "groups" also allows the run phases of those groups to be called separately by the host model. Here's an example SDF (from suite_kessler.xml):
```
<?xml version="1.0" encoding="UTF-8"?>
<suite name="kessler" version="1.0">
<group name="physics_before_coupler">
<scheme>calc_exner</scheme>
<scheme>temp_to_potential_temp</scheme>
<scheme>calc_dry_air_ideal_gas_density</scheme>
<scheme>wet_to_dry_water_vapor</scheme>
<scheme>wet_to_dry_cloud_liquid_water</scheme>
<scheme>wet_to_dry_rain</scheme>
<scheme>kessler</scheme>
<scheme>potential_temp_to_temp</scheme>
<scheme>dry_to_wet_water_vapor</scheme>
<scheme>dry_to_wet_cloud_liquid_water</scheme>
<scheme>dry_to_wet_rain</scheme>
<scheme>kessler_update</scheme>
<scheme>qneg</scheme>
<scheme>geopotential_temp</scheme>
<scheme>cam_state_diagnostics</scheme>
<scheme>kessler_diagnostics</scheme>
</group>
<group name="physics_after_coupler">
<scheme>cam_tend_diagnostics</scheme>
</group>
</suite>
```

The framework code is primarily python code that generates Fortran caps. The class structure looks like:

<figure markdown="span">
![text](figures/capgen-design.PNG "CCPP Framework capgen class diagram"){width="80%"}
</figure>

Given CCPP-compliant physics schemes and one or more SDF, the framework generates caps for the host model to call at the appropriate time. The core files generated by the framework are:

- `<host>_ccpp_cap.F90`: contains the interface layer between the host and the suite(s)
- `ccpp_<suite>_cap.F90`: contains one subroutine per phase (including one run phase per group) in which the phases of the schemes within the suite are called in order
- There is one suite cap generated for each suite being run
- `ccpp_datatable.xml`: consolidates metadata into an XML file to be used as desired by the host model

## Code Structure
Given the context above, this section describes how the CCPP is integrated into CAM-SIMA.

How CAM-SIMA and the CCPP come together:
![text](figures/ccpp-framework-in-cam-sima.PNG "CCPP design in CAM-SIMA")

### Host Model
The core host model code is what is held in the [CAM-SIMA](https://github.com/ESCOMP/CAM-SIMA/tree/development) github repository, plus code that is generated at build-time (or preview_namelists-time) based on the registry (`src/data/registry.xml`).

### Physics
The CCPP physics scheme code exists in the [atmospheric_physics](https://github.com/ESCOMP/atmospheric_physics) repository, which exists as a submodule of CAM-SIMA in the following location:
```$CAM-SIMA/src/physics/ncar_ccpp```

SDFs are located in the root directory of the repository and scheme source code is in the relevant subdirectories.

The `diagnostics` directory contains all diagnostic schemes (the global ones used for state and tendency output, as well as the scheme-specific diagnostic schemes).

The `to_be_ccppized` directory contains physics schemes and utilties that have not been [CCPP-ized](../conversion/ccpp-conversion-guide.md), but were needed by an CCPP-ized scheme.

The `utilities` directory contains schemes that are used regularly, such as tendency applicators and state converters. See [below](#state-and-tendency-variables) for more.

### Generated caps
The caps generated by the CCPP Framework at model build time (or preview_namelists-time) can be found in the following location:
`$CASE/bld/atm/obj/ccpp/`

## Implementation
All CCPP phases are called from the physics driver (`src/physics/utils/phys_comp.F90`). You can see the order of these calls more thoroughly in the documented [run sequence](cam-run-process.md).

### Host-side variables and metadata
For a CCPP-ized physics scheme to work, the framework needs to be able to find a matching variable on the host side for each input variable for the suite(s) in question. This means that CAM-SIMA needs to allocate, initialize, and provide metadata for these variables. We do this in two ways:

- Adding metadata to existing CAM modules (such as `src/data/physconst.F90`) so that a scheme can access an existing host model variable
- All static host-side metadata schemes are included at the top of the registry
- Adding a variable to the registry (`src/data/registry.xml`)
- You can learn more about how the registry is used to generate a Fortran module and corresponding metadata [here](cam-build-process.md#cam-sima-source-and-namelist-generation-buildnml-workflow)

The registry-based code generator is run before the CCPP framework does its magic, so, when it's time, the framework can connect the dots between the host model and the physics.

### State and tendency variables

Two of the most commonly used and referred-to objects in CAM-SIMA are:

- **physics_state**
- **physics_tend**

The Fortran for both objects is auto-generated by CAM-SIMA based on the registry (`$CAM-SIMA/src/data/registry.xml`). The generated code can be found here: `$CASE/bld/atm/obj/cam_registry/physics_types.F90`. The objects are used by the host model (CAM-SIMA), while the CCPP physics take the individual component variables as inputs.

#### Physics state
The *physics_state* object in CAM-SIMA contains the current values for a select set of variables that describe the atmosphere, at the resolution specified by the input grid.

Some examples of these core "state" variables include temperature (`T`), eastward wind (`U`), and northward wind (`V`)

As a rule, CAM-SIMA physics schemes do not update the state directly and instead return tendencies (see below) which are then applied to the state later in the run phase. This is called *time splitting*, which means that all physics since the last state update get the same input state. The alternative, *process splitting*, means that the output state of one scheme serves as the input state of the next.

<figure markdown="span">
![text](figures/process-vs-time-split.PNG "Process vs Time Split"){width="70%"}
</figure>

NOTE: Although [constituents](constituents.md) are handled independently of the `physics_state` object (they are handled by the CCPP framework), they ARE considered state variables.

#### Physics tendencies
The physics tendencies represent how a given scheme (or schemes) changes the state in a single model timestep. The tendencies are accumulated until it is time to apply them to the state. There is one tendency for each state variable being "updated" by the scheme. Some of these tendency variables are held within the *physics_tend* object, but others are internal to the physics.

The module `$CAM-SIMA/src/physics/ncar_ccpp/utilities/physics_tendency_updaters.F90` includes the schemes to apply the tendencies to the relevant state variables. These schemes are added to the SDF whenever the state should be updated. Each calculation looks like: `state_var = state_var + tend*dt` where `dt` is the timestep size.
1 change: 0 additions & 1 deletion docs/design/ccpp-physics.md

This file was deleted.

2 changes: 1 addition & 1 deletion docs/design/constituents.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ This section can be removed when constituents are documented in the CCPP Framewo
### Constituent object (Fortran)
The constituent object (found in `$CAM-SIMA/ccpp_framework/src/ccpp_constituent_prop_ mod.F90`) is a flexible and extendable means of containing necessary constituent data for the framework. The primary object is `ccpp_model_constituents_t`.

![text](constituents-classes.PNG "CCPP Framework constituent object(s)")
![text](figures/constituents-classes.PNG "CCPP Framework constituent object(s)")

This object, importantly, contains the following properties (for which there is metadata; CCPP standard name in parenthesis):

Expand Down
File renamed without changes
File renamed without changes.
Binary file added docs/design/figures/capgen-design.PNG
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/design/figures/ccpp-framework.PNG
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
File renamed without changes
File renamed without changes
Binary file added docs/design/figures/process-vs-time-split.PNG
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/design/figures/run-sequence-api.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/design/figures/run-sequence-dynamics.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/design/figures/run-sequence-history.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/design/figures/run-sequence-physics.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
File renamed without changes
Loading

0 comments on commit b089600

Please sign in to comment.