diff --git a/docs/design/cam-build-process.md b/docs/design/cam-build-process.md
index cc0d312e..4aa95b4e 100644
--- a/docs/design/cam-build-process.md
+++ b/docs/design/cam-build-process.md
@@ -1,4 +1,4 @@
-# CAM-SIMA Build Process
+# Build process
In order to describe the build process, we need to define several source and build directories:
@@ -38,7 +38,7 @@ Given the context above, the following is the CAM-SIMA build sequence:
- Call the [CCPP Framework](https://ccpp-techdoc.readthedocs.io/en/v6.0.0/) to generate glue code (CAPS), (if required).
## CAM-SIMA source and namelist generation (buildnml) workflow
-
+
The diagram above displays everything that occurs when CAM-SIMA's `buildnml` is called by CIME, which occurs after the user calls `preview_namelists`, `case.build`, or `case.submit`.
diff --git a/docs/design/cam-run-process.md b/docs/design/cam-run-process.md
index 544d36d8..8a278ff4 100644
--- a/docs/design/cam-run-process.md
+++ b/docs/design/cam-run-process.md
@@ -1,6 +1,11 @@
-# CAM-SIMA Run Process
+# Run process
-
+
+ {width="80%"}
+ CAM-SIMA run sequence*
+
+
+*Static images can be found at the [bottom](#static-run-sequence-images) of this page
## CAM-SIMA API
Upon running `./case.submit` the core CAM-SIMA driver code* is in `$CAM-SIMA/src/control/cam_comp.F90`. This section lays out each of the subroutines within `cam_comp.F90`.
@@ -51,10 +56,10 @@ The subroutines in `cam_comp.F90` are set up to mirror the phases of the [Common
1. `cam_ctrl_init` (`src/control/cam_control_mod.F90`): Sets the module-level run configuration variables; logs configurations to the atm log
1. `cam_ctrl_set_orbit` (`src/control/cam_control_mod.F90`): Sets the module-level orbital variables
1. `timemgr_init (`src/utils/time_manager.F90`): Initializes the time manager; logs configurations to the atm log
-1. `read_namelist` (`src/control/runtime_opts.F90`): Reads all namelists for the run, including auto-generated scheme namelists (see [build process](/design/cam-build-process/#cam-sima-source-and-namelist-generation-buildnml-workflow))
+1. `read_namelist` (`src/control/runtime_opts.F90`): Reads all namelists for the run, including auto-generated scheme namelists (see [build process](cam-build-process.md/#cam-sima-source-and-namelist-generation-buildnml-workflow))
1. `cam_ctrl_set_physics_type` (`src/control/cam_control_mod.F90`): sets module-level configuration for variables for simple physics and moist physics schemes; logs configurations to atm log
1. `cam_initfiles_open` (`src/control/cam_initfiles.F90`): Opens initial or restart file, and topography file if specified
-1. `cam_register_constituents` (`src/control/cam_comp.F90`): Sets the total number and advected number of [constituents](/design/constituents); currently ALWAYS adds water vapor as constituent (expected by the SE dycore)
+1. `cam_register_constituents` (`src/control/cam_comp.F90`): Sets the total number and advected number of [constituents](constituents.md); currently ALWAYS adds water vapor as constituent (expected by the SE dycore)
1. `air_composition_init` (`src/data/air_composition.F90`): Initializes air-composition-dependent model constants
1. `model_grid_init` (`src/dynamics//dyn_grid.F90`): Initializes model grids and decompositions
1. `cam_ccpp_initialize_constituents` (`$CASE/bld/atm/obj/ccpp/cam_ccpp_cap.F90`): initializes the constituent data array; after this point, we cannot add new constituents
@@ -120,7 +125,7 @@ The routine calls the following subroutines (locations) in this order:
`cam_timestep_final` calls the following subroutines (locations):
-1. [History](/design/history) routines. If it's not the last (half) timestep,
+1. [History](history.md) routines. If it's not the last (half) timestep,
1. `history_write_files` (`src/history/cam_history.F90`): Writes fields to user-configured history files (if applicable)
1. `history_wrap_up` (`src/history/cam_history.F90`): Closes files and zeros buffers as necessary
1. `phys_timestep_final` (`src/physics/utils/phys_comp.F90`):
@@ -140,4 +145,11 @@ The routine calls the following subroutines (locations) in this order:
1. `phys_final` (`src/physics/utils/phys_comp.F90`): calls "final" phase of all schemes in the suite definition file (SDF)
1. `stepon_final` (`src/dynamics//stepon.F90`): finalize dycore (doesn't currently do anything)
-1. `atm2hub_deallocate` and `hub2atm_deallocate` (`src/control/camsrfexch.F90`): deallocate cam_in/cam_out objects
\ No newline at end of file
+1. `atm2hub_deallocate` and `hub2atm_deallocate` (`src/control/camsrfexch.F90`): deallocate cam_in/cam_out objects
+
+## *Static run sequence images*
+
+
+
+
+
\ No newline at end of file
diff --git a/docs/design/ccpp-in-cam-sima.md b/docs/design/ccpp-in-cam-sima.md
new file mode 100644
index 00000000..cb0bc61f
--- /dev/null
+++ b/docs/design/ccpp-in-cam-sima.md
@@ -0,0 +1,158 @@
+# CCPP in CAM-SIMA
+## Overview
+The core Common Community Physics Package (CCPP) documentation can be found [here](https://dtcenter.org/community-code/common-community-physics-package-ccpp/documentation). This section details the code structure and implementation of the CCPP Framework within CAM-SIMA. That said here's a quick overview of the CCPP:
+
+- A CCPP-enabled model consists of the following components (depicted in the diagram below):
+ - Host model and dynamical core
+ - CCPP-compliant physics schemes (organized into Suite Definition Files [SDFs])
+ - Caps generated by the CCPP framework to connect the host model to the physics schemes
+
+
+
+CCPP-compliant physics schemes must adhere to the following criteria:
+
+- Must be broken up into only the following phases:
+ - *register* (anything in the scheme that must be done before the grid is initialized) - run once at startup
+ - *init* - run once at startup
+ - *timestep_init* - run at beginning of every physics timestep
+ - *run* - run on every timestep
+ - *timestep_final* - run at the end of every timestep
+ - *final* - run once at the end of model execution
+- Must contain metadata for all input/output/inout variables passed into each phase (see metadata example below)
+- Must not have use statements outside of the following allowed CCPP use statements (with the exception of `dependencies`, but we're not getting into that now):
+ - ccpp_kinds
+ - ccpp_constituent_prop_mod
+
+*Metadata example*
+(snippet taken from `kessler.meta`)
+
+```
+[ precl ]
+ standard_name = total_precipitation_rate_at_surface
+ long_name = Total precipitation rate at surface
+ units = m s-1
+ dimensions = (horizontal_loop_extent)
+ type = real | kind = kind_phys
+ intent = out
+[ relhum ]
+ standard_name = relative_humidity
+ long_name = Relative humidity
+ units = percent
+ dimensions = (horizontal_loop_extent, vertical_layer_dimension)
+ type = real | kind = kind_phys
+ intent = out
+[ scheme_name ]
+ standard_name = scheme_name
+ units = none
+ type = character | kind = len=64
+ dimensions = ()
+ intent = out
+
+```
+
+CCPP-compliant physics schemes are organized into suite definition files ([SDFs](https://ccpp-techdoc.readthedocs.io/en/v6.0.0/ConstructingSuite.html?highlight=sdf#constructing-suites)). An SDF tells the framework which schemes will be run in what order. Separating schemes into "groups" also allows the run phases of those groups to be called separately by the host model. Here's an example SDF (from suite_kessler.xml):
+```
+
+
+
+
+ calc_exner
+ temp_to_potential_temp
+ calc_dry_air_ideal_gas_density
+ wet_to_dry_water_vapor
+ wet_to_dry_cloud_liquid_water
+ wet_to_dry_rain
+ kessler
+ potential_temp_to_temp
+ dry_to_wet_water_vapor
+ dry_to_wet_cloud_liquid_water
+ dry_to_wet_rain
+ kessler_update
+ qneg
+ geopotential_temp
+ cam_state_diagnostics
+ kessler_diagnostics
+
+
+ cam_tend_diagnostics
+
+
+```
+
+The framework code is primarily python code that generates Fortran caps. The class structure looks like:
+
+
+{width="80%"}
+
+
+Given CCPP-compliant physics schemes and one or more SDF, the framework generates caps for the host model to call at the appropriate time. The core files generated by the framework are:
+
+- `_ccpp_cap.F90`: contains the interface layer between the host and the suite(s)
+- `ccpp__cap.F90`: contains one subroutine per phase (including one run phase per group) in which the phases of the schemes within the suite are called in order
+ - There is one suite cap generated for each suite being run
+- `ccpp_datatable.xml`: consolidates metadata into an XML file to be used as desired by the host model
+
+## Code Structure
+Given the context above, this section describes how the CCPP is integrated into CAM-SIMA.
+
+How CAM-SIMA and the CCPP come together:
+
+
+### Host Model
+The core host model code is what is held in the [CAM-SIMA](https://github.com/ESCOMP/CAM-SIMA/tree/development) github repository, plus code that is generated at build-time (or preview_namelists-time) based on the registry (`src/data/registry.xml`).
+
+### Physics
+The CCPP physics scheme code exists in the [atmospheric_physics](https://github.com/ESCOMP/atmospheric_physics) repository, which exists as a submodule of CAM-SIMA in the following location:
+```$CAM-SIMA/src/physics/ncar_ccpp```
+
+SDFs are located in the root directory of the repository and scheme source code is in the relevant subdirectories.
+
+The `diagnostics` directory contains all diagnostic schemes (the global ones used for state and tendency output, as well as the scheme-specific diagnostic schemes).
+
+The `to_be_ccppized` directory contains physics schemes and utilties that have not been [CCPP-ized](../conversion/ccpp-conversion-guide.md), but were needed by an CCPP-ized scheme.
+
+The `utilities` directory contains schemes that are used regularly, such as tendency applicators and state converters. See [below](#state-and-tendency-variables) for more.
+
+### Generated caps
+The caps generated by the CCPP Framework at model build time (or preview_namelists-time) can be found in the following location:
+`$CASE/bld/atm/obj/ccpp/`
+
+## Implementation
+All CCPP phases are called from the physics driver (`src/physics/utils/phys_comp.F90`). You can see the order of these calls more thoroughly in the documented [run sequence](cam-run-process.md).
+
+### Host-side variables and metadata
+For a CCPP-ized physics scheme to work, the framework needs to be able to find a matching variable on the host side for each input variable for the suite(s) in question. This means that CAM-SIMA needs to allocate, initialize, and provide metadata for these variables. We do this in two ways:
+
+- Adding metadata to existing CAM modules (such as `src/data/physconst.F90`) so that a scheme can access an existing host model variable
+ - All static host-side metadata schemes are included at the top of the registry
+- Adding a variable to the registry (`src/data/registry.xml`)
+ - You can learn more about how the registry is used to generate a Fortran module and corresponding metadata [here](cam-build-process.md#cam-sima-source-and-namelist-generation-buildnml-workflow)
+
+The registry-based code generator is run before the CCPP framework does its magic, so, when it's time, the framework can connect the dots between the host model and the physics.
+
+### State and tendency variables
+
+Two of the most commonly used and referred-to objects in CAM-SIMA are:
+
+- **physics_state**
+- **physics_tend**
+
+The Fortran for both objects is auto-generated by CAM-SIMA based on the registry (`$CAM-SIMA/src/data/registry.xml`). The generated code can be found here: `$CASE/bld/atm/obj/cam_registry/physics_types.F90`. The objects are used by the host model (CAM-SIMA), while the CCPP physics take the individual component variables as inputs.
+
+#### Physics state
+The *physics_state* object in CAM-SIMA contains the current values for a select set of variables that describe the atmosphere, at the resolution specified by the input grid.
+
+Some examples of these core "state" variables include temperature (`T`), eastward wind (`U`), and northward wind (`V`)
+
+As a rule, CAM-SIMA physics schemes do not update the state directly and instead return tendencies (see below) which are then applied to the state later in the run phase. This is called *time splitting*, which means that all physics since the last state update get the same input state. The alternative, *process splitting*, means that the output state of one scheme serves as the input state of the next.
+
+
+{width="70%"}
+
+
+NOTE: Although [constituents](constituents.md) are handled independently of the `physics_state` object (they are handled by the CCPP framework), they ARE considered state variables.
+
+#### Physics tendencies
+The physics tendencies represent how a given scheme (or schemes) changes the state in a single model timestep. The tendencies are accumulated until it is time to apply them to the state. There is one tendency for each state variable being "updated" by the scheme. Some of these tendency variables are held within the *physics_tend* object, but others are internal to the physics.
+
+The module `$CAM-SIMA/src/physics/ncar_ccpp/utilities/physics_tendency_updaters.F90` includes the schemes to apply the tendencies to the relevant state variables. These schemes are added to the SDF whenever the state should be updated. Each calculation looks like: `state_var = state_var + tend*dt` where `dt` is the timestep size.
\ No newline at end of file
diff --git a/docs/design/ccpp-physics.md b/docs/design/ccpp-physics.md
deleted file mode 100644
index 03a8c11a..00000000
--- a/docs/design/ccpp-physics.md
+++ /dev/null
@@ -1 +0,0 @@
-# CCPP Physics
\ No newline at end of file
diff --git a/docs/design/constituents.md b/docs/design/constituents.md
index 343d8b31..8a8e7bbf 100644
--- a/docs/design/constituents.md
+++ b/docs/design/constituents.md
@@ -66,7 +66,7 @@ This section can be removed when constituents are documented in the CCPP Framewo
### Constituent object (Fortran)
The constituent object (found in `$CAM-SIMA/ccpp_framework/src/ccpp_constituent_prop_ mod.F90`) is a flexible and extendable means of containing necessary constituent data for the framework. The primary object is `ccpp_model_constituents_t`.
-
+
This object, importantly, contains the following properties (for which there is metadata; CCPP standard name in parenthesis):
diff --git a/docs/design/buildnml_workflow.jpg b/docs/design/figures/buildnml_workflow.jpg
similarity index 100%
rename from docs/design/buildnml_workflow.jpg
rename to docs/design/figures/buildnml_workflow.jpg
diff --git a/docs/design/cam_history_classes b/docs/design/figures/cam_history_classes
similarity index 100%
rename from docs/design/cam_history_classes
rename to docs/design/figures/cam_history_classes
diff --git a/docs/design/figures/capgen-design.PNG b/docs/design/figures/capgen-design.PNG
new file mode 100644
index 00000000..53b5e622
Binary files /dev/null and b/docs/design/figures/capgen-design.PNG differ
diff --git a/docs/design/figures/ccpp-framework-in-cam-sima.PNG b/docs/design/figures/ccpp-framework-in-cam-sima.PNG
new file mode 100644
index 00000000..5fe1d5d6
Binary files /dev/null and b/docs/design/figures/ccpp-framework-in-cam-sima.PNG differ
diff --git a/docs/design/figures/ccpp-framework.PNG b/docs/design/figures/ccpp-framework.PNG
new file mode 100644
index 00000000..6d22d6d2
Binary files /dev/null and b/docs/design/figures/ccpp-framework.PNG differ
diff --git a/docs/design/constituents-classes.PNG b/docs/design/figures/constituents-classes.PNG
similarity index 100%
rename from docs/design/constituents-classes.PNG
rename to docs/design/figures/constituents-classes.PNG
diff --git a/docs/design/hist_config_classes.PNG b/docs/design/figures/hist_config_classes.PNG
similarity index 100%
rename from docs/design/hist_config_classes.PNG
rename to docs/design/figures/hist_config_classes.PNG
diff --git a/docs/design/figures/process-vs-time-split.PNG b/docs/design/figures/process-vs-time-split.PNG
new file mode 100644
index 00000000..f9ff86f5
Binary files /dev/null and b/docs/design/figures/process-vs-time-split.PNG differ
diff --git a/docs/design/figures/run-sequence-api.png b/docs/design/figures/run-sequence-api.png
new file mode 100644
index 00000000..2e70d1e4
Binary files /dev/null and b/docs/design/figures/run-sequence-api.png differ
diff --git a/docs/design/figures/run-sequence-dynamics.png b/docs/design/figures/run-sequence-dynamics.png
new file mode 100644
index 00000000..28f82902
Binary files /dev/null and b/docs/design/figures/run-sequence-dynamics.png differ
diff --git a/docs/design/figures/run-sequence-history.png b/docs/design/figures/run-sequence-history.png
new file mode 100644
index 00000000..b5f6aa3b
Binary files /dev/null and b/docs/design/figures/run-sequence-history.png differ
diff --git a/docs/design/figures/run-sequence-physics.png b/docs/design/figures/run-sequence-physics.png
new file mode 100644
index 00000000..87036616
Binary files /dev/null and b/docs/design/figures/run-sequence-physics.png differ
diff --git a/docs/design/run-sequence.gif b/docs/design/figures/run-sequence.gif
similarity index 100%
rename from docs/design/run-sequence.gif
rename to docs/design/figures/run-sequence.gif
diff --git a/docs/design/history.md b/docs/design/history.md
index a866e74b..7e4cde80 100644
--- a/docs/design/history.md
+++ b/docs/design/history.md
@@ -1,4 +1,4 @@
-# History & Model Output
+# History & model output
CAM-SIMA history is the mechanism for configuring and generating diagnostic output from a model run. It is also used to generate initial-data files and aids in the model-restart process by saving the state of diagnostic fields whose processing window (e.g., averaging, standard deviation) crosses a restart-write cycle. This page describes the implementation of CAM-SIMA history in CAM-SIMA.
@@ -15,77 +15,16 @@ CAM-SIMA history is the mechanism for configuring and generating diagnostic outp
| hist_add_avg_fields *" where “time_period” can be: steps, seconds, minutes, hours, days, months, years. The closest CAM7 equivalent is “nhtfrq”. | &hist_file_config_nl.cam.h0a.0001-02-01-00000.nc
- - This file will contain averaged output for T and Q (air_temperature and water vapor)
- - It will have one frame with the time calculated at the midpoint of the month
+
### Setting up the history data structures
@@ -95,7 +34,7 @@ In plain English, a one-month run with these history configuration will result i
- The `hist_file_t` object contains information about the configuration options for a given history volume. This includes the maximum number of frames that can be written to the file, the current number of frames on the file, the name of the file, and all of the history fields to be written to the file. It also contains methods to populate the field lists (`config_set_up_fields`), set up the metadata of the file (`config_define_file`), and write history fields to the file (`config_write_time_dependent_variables`).
- Each `hist_file_t` object contains both a hash table and allocatable field list to keep track of the fields written to the file. The core class for each of these is the `hist_field_info_t` (in `src/history/buffers/src/hist_field.F90`), which contains information about a history field. This includes the field names, the accumulate flag (average, instantaneous, minimum, etc), units, type, dimensions, and fill value. It also includes the buffer(s) (type is/are `hist_buffer_t`) that will and do hold the actual data.
-![text](cam_history_classes"CAM-SIMA history Fortran class diagram")
+![text](figures/cam_history_classes"CAM-SIMA history Fortran class diagram")
## Populating the possible field list
The possible fields to be output by the history infrastructure are tracked in `cam_history.F90` via the `possible_field_list` hash table. It is populated during init time by calls to the subroutine `history_add_field` (found in `src/history/cam_history.F90`). “Init time,” means that all calls to `history_add_field` must occur during the execution of `cam_init` (found in `src/control/cam_comp.F90`).
@@ -168,7 +107,7 @@ The bolded step #3 above occurs any time the criteria for #1 is satisfied. At th
```
call hist_configs(file_idx)%write_time_dependent_variables(file_idx, restart)
```
-It is during this call that we increment the number of samples written for this volume and actual write the data held within the buffer(s) to the netcdf file (as well as the time-dependent metadata for the fields).
+It is during this call that we increment the number of samples written for this volume and actually write the data held within the buffer(s) to the netcdf file (as well as the time-dependent metadata for the fields).
## Defining history restart files
*Restarts not yet implemented in CAM-SIMA*
diff --git a/docs/design/sima-design-goals.md b/docs/design/sima-design-goals.md
new file mode 100644
index 00000000..b37b80e9
--- /dev/null
+++ b/docs/design/sima-design-goals.md
@@ -0,0 +1,56 @@
+# Design goals & features
+
+Motivated by the Singletrack project (a precursor to [SIMA](https://sima.ucar.edu/about)), the CAM-SIMA project was created to build a new CAM infrastructure to meet the SIMA science and computational needs. This page documents those needs and some of the features that implement them.
+
+## CAM needs to be more run-time configurable
+
+- To make CAM and CESM more available, e.g., for usage in containers and the cloud, a build of CAM should be more configurable than it is at present.
+- One feature that makes CAM more run-time configurable is moving physics suites to the [CCPP](https://dtcenter.org/community-code/common-community-physics-package-ccpp). By allowing CAM to compile in more than one physics suite, the physics suite can be selected at run time (e.g., as a namelist variable).
+- Another feature needed to make CAM more run-time configurable is making dycores themselves more run-time configurable. For instance, the SE dycore will no longer require the number of advected constituents to be specified at compile time.
+
+## Remove obstacles to use of specialized high-performance processing units (e.g., GPUs, FPGAs)
+
+The chunk structure in CAM physics served its purpose when threading was the only way to accelerate code. However, to make the best use of both threading and modern accelerators, a flexible chunking mechanism is required. The new infrastructure enables this by using flat arrays for all fields.
+
+- Moving to flexible precision of data is important for being able to test both performance improvements and the affect on model quality. **The CCPP is explicitly designed to allow for compile-time selection of precision at the physics suite level as well as for individual fields.** In addition, the new infrastructure is explicitly designed to handle the case where the dycore is running at a different precision than the physcs (i.e., by using proper field promotion and demotion primitives).
+- Pointers in Fortran are less efficient because they prevent some optimization techniques. The new infrastructure avoids pointers as much as possible by making use of the automatic data management capability of the CCPP (which does not create pointers).
+- The new infrastructure provides greater flexibility in that the model can be built with multiple physics suites to increase run-time flexibility. There is a tradeoff in that building more physics suites will often increase build time. Builds with a single suite should be faster than now since only the schemes that are required for the suite are compiled (currently most schemes are compiled all the time even if they will not be used).
+
+## Modularity
+In order to continue to allow CAM to grow without an ever increasing cost of bringing in new features, CAM must be more modular. A classic example is chemistry which ACOM would like to make modular but which is currently entwined with CAM in many areas (e.g., code in CAM repository, extensive configuration code dedicated to chemistry, extensive namelist building code and data dedicated to chemistry, large number of use cases containing chemistry configuration data). The new CAM infrastructure contains features to increase modularity.
+
+- Support for multiple namelists. This allows modular components to contain their own namelist (run-time configuration options). The active namelists are combined to produce atm_in.
+- Flexible handling of constituent information. Modular components can provide constituent information via metadata (if component is a CCPP scheme) or at run time.
+
+Modularity will allow CAM to move components to external repositories. This process cuts development costs for both CAM and for the component (e.g., as has happened with PUMAS). Some ways this is accomplished are listed here:
+
+- Code reviews are more efficient since CAM SEs do not have to review every routine in the external module so they can just focus on the interfaces. The external developers do not have to be involved in CAM modifications.
+- Externals can develop and maintain they own namelist definition files, they do not have to coordinate with the larger CAM namelist (which itself has been broken into several smaller namelists).
+- Namelists associated with physics schemes do not have to have separate namelist-reading code. The new infrastructure automatically creates an appropriate Fortran module to read in the runtime data from atm_in. The system then also ensures that all active namelists are called at run time. This process ensures that namelists are always read correctly while not requiring coding or reviews to keep up to date with namelist changes.
+
+Use of the CCPP to build physics suites also makes CAM more modular because the CCPP treats physics schemes as modular which allows flexibility in building physics suites. The CCPP takes care of making sure variables are available before they are used and also builds the code currently handled via hand-written code in the various versions of physpkg.F90.
+
+## Run-time data checking
+CAM needs data to run but the data needs vary with the simulation. The new infrastructure facilitates this.
+
+- Before running physics, the new infrastructures queries the physics suite as to what input fields are required (using a CCPP interface). Then it makes sure that all of these fields have been initialized or reads the values from the initial data file. Any uninitialized fields that are not found on the initial data file will trigger a run-time error.
+
+## Efficient offline testing and simulation
+
+CAM currently has a few ways to run offline testing (e.g., SCAM, PORT). The new infrastructure builds these capabilities in for more efficient and flexible use.
+
+- The new infrastructure has the ability to run without a dycore.
+- Offline mode (NULL dycore) can be run with any number of columns.
+- Offline mode does not required gridded input.
+
+## Software quality control
+To enable efficient quality control, the new infrastructure implements a number of continuous integration (CI) techniques.
+
+- To implement all the flexibility mentioned above, the new infrastructure makes extensive use of python scripts.
+ - Python scripts make extensive use of python doctests (for simpler tests).
+ - There are python unit tests to cover more complicated situations.
+ - The GitHub CI also runs a static analysis tool (pylint) to provide feedback on potential coding issues.
+ - Python tests can easily be run by hand but are also automatically run on GitHub
+- The new infrastructure will use offline mode (see above) to run many physics configurations quickly without requiring large machines.
+ - This will enable quick testing during development on a laptop.
+ - We hope many of these tests can also be run automatically on GitHub.
\ No newline at end of file
diff --git a/docs/development/cam-coding-standards.md b/docs/development/cam-coding-standards.md
new file mode 100644
index 00000000..2c65c936
--- /dev/null
+++ b/docs/development/cam-coding-standards.md
@@ -0,0 +1,101 @@
+# Coding standards
+
+The standards in this document are largely prescriptive, i.e., they should be followed.
+
+- *MUST*: Exceptions must be discussed and agreed to by the CAM SEs and noted in this document.
+- *SHOULD*: Exceptions must be approved by the CAM SEs and documented in the ChangeLog.
+
+While some legacy code will not follow these rules, efforts SHOULD be made to improve the code whenever you are working on it (e.g., bug fixes, enhancements).
+
+## General coding standards
+### MUST
+- Always use spaces instead of tabs
+- No trailing spaces (i.e., no spaces at the end of a line)
+
+See [tips for configuring editors](#tips-for-configuring-editors)
+
+### SHOULD
+
+- Use comments to explain the purpose of the following code and/or include any important but non-obvious information. When working with code, always check the comments to make sure they are still correct and useful.
+- Do not use comments to 'save code for later in case it might be useful'.
+- Do not include a comment that merely restates the following code logic (e.g., 'Loop over variables')
+
+## Python coding standards
+We expect all python code to pass with a perfect score (10) using `pylint` with [this version](https://github.com/ESCOMP/CAM-SIMA/blob/development/test/.pylintrc) of `pylintrc`. However, external repos may have their own `pylintrc` version depending on their needs.
+
+We also expect all python files to follow the [black code style and format](https://black.readthedocs.io/en/stable/the_black_code_style/index.html).
+
+Finally, one can also follow the [Google Python](https://google.github.io/styleguide/pyguide.html) Style Guide.
+
+## Fortran coding standards
+The standards described in this section represent the CAM Fortran standards. Other Fortran standards:
+
+- [CTSM Fortran Standards](https://wiki.ucar.edu/display/ccsm/Comprehensive+list+of+standards)
+- [MOM Fortran Standards](https://github.com/mom-ocean/MOM6/wiki/Code-style-guide)
+
+### MUST
+* No naked `use` statements
+* No continued single-line `if` statements (i.e., all `if` statements should have a `then` if the statement is on more than one line)
+* Every namelist variable in each active namelist group is present in the namelist file. An active namelist group is one which may be read during the current run.
+* All namelist variables except for logical quantities are initialized to invalid values (integer: `-HUGE(1)`, real: `NaN`, character: `'UNSET'`).
+* Functions may _not_ have side effects, and should include the `pure` keyword.
+* Do not combine statements on a single line (i.e., avoid use of the semi-colon to combine statements).
+* Use `intent` for dummy arguments except for pointers.
+* All variables of type real must have a specified kind, including literals. For example, use `1.5_r8`, not `1.5` or `1.5D0`. Literals must also include the decimal point.
+* All character declarations must use Fortran 90+ syntax (e.g., `character(len=*)` or `character(len=CL)`).
+* All variable declarations must use Fortran 90+ syntax (i.e., must include the double colon between the attributes and the variable name).
+* All type and procedure declarations must use Fortran 90+ syntax (i.e., must include the double colon before the type or procedure name).
+* All modules should include an `implicit none` statement in the preamble (after the `use` statements). Module routines then do not need this statement.
+* All optional arguments must be passed via keyword (e.g. use `call subroutine(x, optional_y=y)` instead of `call subroutine(x, y)` for the optional variable `optional_y`).
+* Initialize local (non-parameter) variables in subroutines and functions at the top of the executable code, NOT on a variable declaration lines.
+ - Initializing a local variable on a declaration line invokes the `SAVE` attribute and is not thread safe.
+ - Local pointer variables MUST be initialized before other (non-initialization) statements. By default, use the `nullify` statement.
+* All variables that are on the physics grid must have their horizontal dimension declared with `pcols`, even if only a subset of the variable is used in the subroutine or function.
+### SHOULD
+* Avoid use of preprocessor directives (e.g., `#if`, `#ifdef`). Always try for runtime variable logic instead.
+* Keep formula statements relatively short. Use temporary variables to break long formulas into easier-to-read sections.
+* Use subroutines to avoid repeated (cut and paste) code logic.
+* Avoid side effects in subroutines. Pass variables to routines instead of 'using' them from elsewhere.
+* Use the `pure` keyword if a subroutine has no side effects.
+* List dummy arguments one per line, however, related items may be grouped.
+* Dummy argument order should match the order in the argument list.
+* Use symbolic numerical comparison operators (e.g., `==`, `/=`, `<`, `>=`) not old character versions (e.g., `.eq.`).
+* Avoid the use of pointers as dummy arguments (exceptions must be discussed in design or code review)
+* Modules should be default `private`. Public interfaces are declared after the `private` declaration.
+* `private` module interfaces (i.e., subroutines and functions) should be declared private in the module header.
+* Module names should conform to their filename (i.e., the module name should be the filename without the `.F90`).
+* Functions _should_ use the `pure` attribute. If they cannot, the reason should be included as a comment in the function's preamble.
+* All functions and subroutines should avoid un-necessary statements (e.g. a blank `return` at the end of a subroutine).
+* `use` statements should be brought in at the smallest scope possible (e.g. inside individual subroutines instead of at the module level).
+
+### Indentation and style
+
+* *Scoping*: Indentation should follow scope. That is, whenever entering a new scope (e.g., `module`, `subroutine`, `if`, `do`), indent that scope relative to the scoping statement (recommended 3 spaces but each module should at least be self consistent).
+* A single line should be less than 133 characters long.
+* *Continue lines*: Indent continue lines 5 spaces or align with similar lines in statement.
+* Use spaces to ease reading statements (e.g., before and after operators, after commas except in a dimensions list)
+* Include a space after `if`, `else`, `end`, `do`, and `while`.
+* Include a space before and after `::`
+* No space after `only`, i.e., `only:`, not `only :`.
+* When aligning code for readability, commas go immediately after a symbol (no space).
+
+## Tips for configuring editors
+### emacs (add to your `.emacs` file)
+- To automatically remove trailing spaces whenever you save a file:
+```
+(add-hook 'before-save-hook 'delete-trailing-whitespace)
+```
+- To automatically indent with spaces instead of tabs:
+```
+(setq-default indent-tabs-mode nil)
+```
+- To use 4 spaces for each indent:
+```
+(setq tab-width 4)
+```
+
+### vi (add to your `.vimrc` file)
+- To automatically remove trailing spaces whenever you save a file:
+```
+ autocmd BufWritePre * :%s/\s\+$//e
+```
\ No newline at end of file
diff --git a/docs/development/cam-testing.md b/docs/development/cam-testing.md
new file mode 100644
index 00000000..bcb28404
--- /dev/null
+++ b/docs/development/cam-testing.md
@@ -0,0 +1,107 @@
+# Testing
+
+This page describes the various automated and manual tests that are run for CAM-SIMA whenever the code is modified, as well as instructions for how to add new tests.
+
+## Python unit testing
+CAM-SIMA supports two kinds of python unit tests, `doctest` and `unittest` tests, both of which are part of the standard python library.
+
+All `unittest` tests should be in:
+
+`CAM-SIMA/test/unit`
+
+while all files used by the tests should be in:
+
+`CAM-SIMA/test/unit/sample_files`
+
+All `unittest` tests are automatically run via Github Actions whenever a Pull Request (PR) is opened, modified, or merged.
+
+All `doctest` tests are also run automatically as long as the scripts they are located in are under `CAM-SIMA/cime_config` or `CAM-SIMA/src/data`.
+
+To manually run all of the unit tests at any time, simply run the following shell script:
+
+`CAM-SIMA/test/run_tests.sh`
+
+Finally, when adding new tests, determine if the test can be done in only a few lines with minimal interaction with external files or variables. If so, then it would likely be best as a `doctest`. Otherwise it should be a `unittest` test. Failure to follow this rule of thumb could result in test failures in the Github Actions workflow. Also remember to add your new tests to the `run_tests.sh` script so future users can easily run the tests manually.
+
+## Static Source Code Analysis
+
+### Python
+
+Any python script which is added or modified via a PR will automatically be analyzed using `pylint`, and must have a score of 9.5 or greater in order to not be marked as a test failure. The hidden `pylintrc` file used for the analysis can be found here:
+
+`CAM-SIMA/test/.pylintrc`
+
+Users can also manually run `pylint` against the core python build scripts by running the following shell script:
+
+`CAM-SIMA/test/pylint_test.sh`
+
+Please note that `pylint` is not part of the standard python library, and so it may need to be installed before being able to run the shell script.
+
+## Regression Testing
+### Running the regression tests (manual)
+
+**NOTE: Regression testing on Derecho should be done for every PR before merging!**
+
+Users can manually run regression tests on Derecho to ensure that the model builds correctly in various configurations. The tests can be run with a local copy of CAM-SIMA by using the `test_driver.sh` script under `$CAM-SIMA/test/system`. To run the tests associated with a particular compiler option one can do the following commands:
+
+For running GNU tests*:
+```
+env CAM_FC=gnu ./test_driver.sh -f
+```
+
+For running Intel tests*:
+```
+env CAM_FC=intel ./test_driver.sh -f
+```
+
+Running the script will produce a directory in your scratch space labeled `aux_sima__`, where `` is the compiler you chose, and `` is the timestamp (starting with the date) of when the tests were started, along with a job submitted to the local batch system.
+
+Inside the directory you should see an executable labeled `cs.status.*`. Running that command after the submitted job has finished will display the test results. Currently for all tests everything should be labeled `PASS` except the `SUBMIT` step, which should be labeled `PEND`. Any other label indicates that a test may have failed, and should be investigated.
+
+Finally, the tests themselves are listed in `/cime_config/testdefs/testlist_cam.xml`. Any files that need to be included in order for the tests to run properly are located in `` XML entry
+- If you are adding a fully new test, add a new `` XML entry with the following structure:
+```
+
+
+
+
+ COMMENT HERE
+ WALLCLOCK_TIME
+
+
+```
+
+- ``: component set alias (or long name) - you can see more about compsets [here](../usage/creating-a-case.md)
+- ``: model grid/resolution you'd like to run on - you can see more about grids [here](../usage/creating-a-case.md)
+- ``: type of test to be run. You can find the testing types [here](https://esmci.github.io/cime/versions/master/html/users_guide/testing.html#testtype).
+- ``: test modifier that changes the default behavior of the test type. More [here](https://esmci.github.io/cime/versions/master/html/users_guide/testing.html#modifiers)
+- ``: relative path to the testmods directory for this run; usually looks something like `"cam/some_directory_name/"`
+ - The testmods directory will contain any namelist mods and XML configuration variable changes for the test (`user_nl_cam` and/or `shell_commands`)
+ - testmods directories can be found in `$CAM-SIMA/cime_config/testdefs/testmods_dirs/cam/`
+- ``: machine name - will almost definitely be either `derecho` or `izumi`
+- ``: compiler to be used (options: `gnu`, `nag`, `intel`, `nvhpc`)
+- ``: group of tests that this test belongs to - the default run by `test_driver.sh` is `aux_sima` (which is run for each PR to CAM-SIMA)
+- `WALLCLOCK_TIME`: maximum amount of time that the job will be allowed to run
+
+Here is an example test entry for a 2-timestep smoke test of kessler physics on the MPAS grid, run with both intel and gnu
+```
+
+
+
+
+ 00:10:00
+ GNU build test for MPAS dycore (with Kessler physics)
+
+
+```
\ No newline at end of file
diff --git a/docs/development/debugging.md b/docs/development/debugging.md
new file mode 100644
index 00000000..d3099970
--- /dev/null
+++ b/docs/development/debugging.md
@@ -0,0 +1,73 @@
+# Debugging techniques
+Start with the [CAM wiki](https://github.com/ESCOMP/CAM/wiki/CAM-debugging-techniques).
+
+## CAM-SIMA-specific debugging
+### Build errors
+Debugging tips if you get build errors:
+
+- If the output indicates that the error message or failure is coming from somewhere within $CAM-SIMA/ccpp_framework:
+ - If you're getting a clear error message, it's likely that you have something wrong with your metadata
+ - If you're getting an error message that indicates that something is breaking in the framework code itself (something went uncaught) - consult the AMP SEs
+- If the error happens during the atm build, you can see the full output of the atm build in the build log here: `bld/atm.bldlog.*`
+
+### Run-time errors
+
+- Start with the atm.log* - if the issue occurred during the execution of the CAM code, it will hopefully have a clear and concise error message
+- Move to the cesm.log* - it will hopefully include a stack trace for the error in question; if the error did not occur in the CAM code (or CAM did not properly trap the error), it will help you identify the source of the issue.
+- If neither log file contains helpful information, a few first steps:
+ - Resubmit the case; it could be a machine hiccup
+ - Turn on DEBUG mode (if it's not on already) and rebuild/rerun
+ - Look in your run directory for any log files called `PETXXX` - if there was an issue on the [ESMF](https://earthsystemmodeling.org/docs/release/latest/ESMF_usrdoc/) side of things, it will show up in one of these (there will be one PET file per processor)
+ - Try a different compiler - maybe it'll give you a more helpful error message
+ - set NTASKS=1 (`./xmlchange NTASKS=1`), do a clean rebuild (as instructed), and run again; maybe running in serial will identify the error
+ - Look for the `***************** HISTORY FIELD LIST ******************` in the atm.log* file; if it's not there, the error occurred at init time
+ - If the error occurred during init time, try a new case with a different grid and/or dycore
+ - If the model ran for a few timesteps before dying (look for the `CAM-SIMA time step advanced` message in the atm.log* file), it's likely that one or more variable that you introduced or modified has gone off the rails (value has become very large or very small or zero)
+ - Update your user_nl_cam to output all possible suspected variables to a history file at some point shortly before the model dies, then inspect the output to see if any are obviously wrong
+ - If the model completed all timesteps, try running a shorter case to see if the problem persists; if so, it's an error during the model finalization
+ - Run the [TotalView](#totalview) debugger on izumi
+ - Use the old standard - print statements - to narrow down where the code is stopping
+ - Ask for help!
+
+### Unexpected answer changes
+
+- Two paths here:
+ - You're getting unexpected DIFFs from the regression testing
+ - Consult with a scientist about whether differences are expected and for which configurations (compsets, resolutions, namelists parameters, etc)
+ - If the differences are very small (look like round-off), consult with the other AMP SEs on whether we're ok with this
+ - If the differences are indeed unexpected and larger than round-off, create a case using the code from the head of `development` and:
+ - place print statements in both code bases (your development branch and the head of `development`) to identify where the numbers are going awry OR
+ - run the [TotalView](#totalview) debugger OR
+ - use the comparison tool described below (`$CAM-SIMA/tools/find_max_nonzero_index.F90`)
+ - You're getting unexpected answer changes compared with CAM
+ - Consult with other AMP SEs about whether the differences appear to be due to round-off error
+ - Use the comparison tool (LINK ONCE IT EXISTS): `$CAM-SIMA/tools/find_max_nonzero_index.F90`
+ - This tool can help you narrow down where the issue begins by printing out values at a specific index and comparing those with the "truth" (from CAM)
+
+### TotalView
+(COURTNEY - ASK CHERYL TO HELP WITH THIS)
+
+- Create and configure a new case (using gnu and only 1 task)
+- Build the case (`./case.build`)
+- Run command `bash` to change to bash (if not already)
+- Run the following commands:
+
+```
+np=1
+nthreads=1
+
+source .env_mach_specific.sh
+
+RUNDIR=`./xmlquery RUNDIR -value`
+EXEROOT=`./xmlquery EXEROOT -value`
+LID=`date '+%y%m%d-%H%M%S'`
+
+cd $RUNDIR
+mkdir timing
+mkdir timing/checkpoints
+echo `pwd`
+export OMP_NUM_THREADS=$nthreads
+totalview ${EXEROOT}/cesm.exe
+```
+
+- `exit` to exit the totalview window and give up the node
\ No newline at end of file
diff --git a/docs/development/figures/fork_button.png b/docs/development/figures/fork_button.png
new file mode 100644
index 00000000..061b2cc8
Binary files /dev/null and b/docs/development/figures/fork_button.png differ
diff --git a/docs/development/git-basics.md b/docs/development/git-basics.md
new file mode 100644
index 00000000..f4e478c5
--- /dev/null
+++ b/docs/development/git-basics.md
@@ -0,0 +1,343 @@
+# Working with git and GitHub
+
+## Getting started with git
+
+- For a quick, visual introduction to git, try a [5 Minute Overview of Git](https://www.youtube.com/watch?v=xvwBtODV0ms) on Youtube
+- Need an introduction to version control along with git and GitHub? Try [Git and GitHub for Poets](https://www.youtube.com/watch?v=BCQHnlnPusY), a Youtube series
+- For a glossary of common terms used in git and GitHub documentation, see the [GitHub glossary](https://docs.github.com/en/get-started/learning-about-github/github-glossary).
+- [GitHub's Git Handbook](https://docs.github.com/en/get-started/using-git/about-git) offers a brief overview of git and GitHub, including example workflows and links to more in-depth resources.
+- Software Carpentry offers a [hands-on introduction to git](https://swcarpentry.github.io/git-novice/).
+- A highly-regarded, comprehensive git reference is the book Pro Git (available as a [book](https://www.boulderbookstore.net/book/9781484200773) or a [free pdf](https://git-scm.com/book/en/v2)).
+ - The first three chapters of Pro Git, along with sections 6.1 and 6.2 on GitHub, are particularly useful if you're getting started with git. [1]
+ - The site also contains [links](https://git-scm.com/doc) to git reference pages along with introductory videos on git.
+ - You can skip the last section of chapter 3, on rebasing, until you have more experience with git. It's easy to cause problems for yourself or others with rebasing so only use rebasing in CAM-SIMA following the workflow instructions below as rebasing rewrites history, which can make it impossible to reproduce exactly what you have done before.
+- For a clear introduction to git internals, try [Dissecting Git Guts](https://www.youtube.com/watch?v=Y2Msq90ZknI) by Emily Xie.
+- [Git from the inside out](https://maryrosecook.com/blog/post/git-from-the-inside-out) is a blog post with accommpaning video from Mary Rose Cook that looks under the covers a bit to show what happens when you execute git commands.
+- [Git from the Bottom Up](https://jwiegley.github.io/git-from-the-bottom-up/) by Josh Wiegley provides a quick glossary of the main terms describing a git repository with links to detailed descriptions. Good site for reminders about a git concept.
+- Need to really understand the science behind how git works? Try [Advanced Git: Graphs, Hashes, and Compression, Oh My!](https://www.youtube.com/watch?v=ig5E8CcdM9g) by Matthew McCullough
+
+Git also offers extensive built-in (i.e., command line) help, although it can be hard to understand until you are familiar with basic git concepts:
+
+`git help`
+
+`git help COMMAND`
+
+[man gittutorial](https://mirrors.edge.kernel.org/pub/software/scm/git/docs/gittutorial.html)
+
+[man giteveryday](https://mirrors.edge.kernel.org/pub/software/scm/git/docs/giteveryday.html)
+
+## How to set up your git development environment
+There are several stages in setting up your git and GitHub development environment. Below, they are broken into three sections which represent the different stages:
+
+- [One time GitHub setup](#one-time-github-setup)
+- [Set up git environment on new machine](#set-up-git-environment-on-new-machine)
+- [Working with clones](#working-with-clones)
+
+### One time GitHub setup
+
+1. Set-up personal github account (if one does not already exist): [https://github.com](https://github.com)
+1. Create a new personal CAM-SIMA fork of the [ESCOMP/CAM-SIMA](https://github.com/ESCOMP/CAM-SIMA) repo:
+
+ 
+
+1. Set up SSH keys on GitHub (optional):
+
+ If you will be pushing changes to GitHub (either to your own fork or to shared forks), or if you will be pulling changes from private repositories on GitHub, then it's worth the time to set up ssh keys for each machine you'll be using. By doing so, you won't have to enter your password when pushing to or pulling from GitHub.
+
+ See [here](https://help.github.com/articles/connecting-to-github-with-ssh) for instructions. After doing this, you can use the ssh form of GitHub URLs (e.g., `git@github.com:ESCOMP/cam-sima.git`) in place of the `https` form.
+
+1. Configure GitHub notifications (optional): It is important to keep track of activity on GitHub but there are ways to manage how you are notified.
+ - **Controlling how you receive notifications**: Click on your profile picture in the upper-right of any GitHub page, then click on "Settings", and then on "Notifications". You will see several options, here are a few recommended settings:
+ - *Automatically watch repositories*: **Yes** (check box)
+ - *Participating*: **Email**
+ - *Watching*: **Web**
+ - **To see and manage Web notifications**: Click on the bell in the upper right of any GitHub page to see Web notifications and to manage which repositories you are watching. There will be a blue dot on the bell if there are new (unread) notifications).
+
+### Set up git environment on new machine
+git has global settings which apply to all repository clones on your machine. These settings reside in a file called `.gitconfig` in your home directory. Below are some required and some optional but recommended global git settings to apply to any new machine where you will do CAM-SIMA development (e.g., Derecho, Izumi, personal laptop). Apply the settings below or simply copy a `.gitconfig` file from a machine that is already configured.
+
+#### Required git global configuration settings
+```
+git config --global user.name "Your Name"
+git config --global user.email
+```
+
+We recommend that you use your UCAR email address as your GitHub email address but if you use another address, you can add your UCAR email address by clicking on your profile picture in the upper-right of any GitHub page, then clicking on "Settings", and then on "Emails".
+
+#### Recommended git global configuration settings
+You can set which editor to use for log messages, etc., with:
+```
+git config --global core.editor
+```
+(See [http://swcarpentry.github.io/git-novice/02-setup](http://swcarpentry.github.io/git-novice/02-setup) for specific settings to use for many common editor choices.)
+
+The following setting generates better patches than the default:
+```
+git config --global diff.algorithm histogram
+```
+The following setting makes it easier to resolve conflicts if you're doing conflict resolution by hand as opposed to with a dedicated conflict resolution tool. Without this setting, you'll just see your version and the version you're merging in delimited by conflict markers. With this setting, you'll also see the common ancestor of the two sides of the merge, which can make it much easier to figure out how to resolve the conflict:
+```
+git config --global merge.conflictstyle diff3
+```
+Alternatively, look into using a graphical conflict-resolution tool such as kdiff3 or the Emacs built-in `M-x vc-resolve-coflicts`.
+
+We recommend that you set git to not push anything by default:
+```
+git config --global push.default nothing
+```
+This can help prevent you from accidentally pushing work to the wrong repository.
+
+#### Configuring git on shared machines
+If using git on shared resources, such as on the login nodes for CISL machines, then one may find their git commands being killed by sys admins due to git spawning too many threads and thus blocking (or at least slowing down) other users. To avoid this situation, you can limit the number of threads git spawns for various activities by setting the following git config variables:
+```
+git config --global --add index.threads 8
+git config --global --add grep.threads 8
+git config --global --add pack.threads 8
+```
+Please note that a limit of 8 threads was chosen specifically for CISL machines. If you are using a separate shared system you may find it beneficial to choose a different thread limit.
+
+#### git tools for Bash
+There are two helpful things you can do to streamline your use of git if you use the bash shell. These are completely optional, but improve your git experience. These are documented in the appendix of the excellent Pro Git book.
+
+### Updating your login to a GitHub token
+If you have been denied access to push to a GitHub repo, note that the old password system has been deprecated and no longer works. If you do not make changes, you will no longer be able to work with GitHub on personal forks and private repos.
+
+If you only ever git clone from public repositories like ESCOMP and ESMCI, you may ignore the rest of this wiki page.
+
+If you've already created and are using a GitHub token, you may ignore the rest of this wiki page.
+
+GitHub will soon be requiring that you use a GitHub generated token (basically a long password that they autogenerate for you).
+
+1. If you are on a Mac and do not need to use your password, but still receive an error message, you may have stored your password in the keychain. [Remove a password on Macs](https://docs.github.com/en/github/getting-started-with-github/getting-started-with-git/updating-credentials-from-the-macos-keychain)
+1. [Create a GitHub password token](https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token). Make sure you copy it to a safe place as they only show it to you once.
+1. Store your credentials on every machine you use GitHub on.
+ - On a private / secure Linux system:
+
+ ```
+ git config --global credential.helper store
+ ```
+
+ - Note that the `store` option to git's credential helper stores your GitHub token in plain text. If you are on a public machine, you should store your GitHub token in a password manager and use the credential cache mechanism to open command-line access to your GitHub repositories:
+
+ ```
+ git config --global credential.helper cache --timeout
+ ```
+
+ where `` is the number of seconds git 'remembers' your token. For example, to only have to enter your token once per day:
+
+ ```
+ git config --global credential.helper cache --timeout 86400
+ ```
+
+ - On Windows:
+
+ ```
+ git config --global credential.helper wincred
+ ```
+ - On Macs (make sure you select the Mac tab in the middle of the window): see [this documentation](https://docs.github.com/en/github/getting-started-with-github/caching-your-github-credentials-in-git)
+
+1. On each machine, do a `git clone` of your personal repo. It will ask for your username and password. The password needs to be your saved token. You should only need to do this once.
+1. To test that your credentials are successfully cached, do a new `git clone` of your personal repo. If it clones without asking for your username/password you are done.
+
+## Working with clones
+Here are some commands for creating and working with clones:
+
+### Create a new clone
+```
+git clone https://github.com//CAM-SIMA
+cd CAM-SIMA
+```
+or
+```
+git clone git@github.com:/CAM-SIMA
+cd CAM-SIMA
+```
+where `` is your GitHub account login ID. Some useful options to the clone command are:
+
+- `--origin `: A clone knows where it came from and by default, calls that location, "origin". You can change that name with the `--origin` option. This can come in handy when dealing with multiple upstream repositories.
+Change the clone directory name: By default, the clone is created in a directory with the same name as the repository. You can change this by adding a directory name to the clone command. Use the appropriate example below:
+
+```
+git clone https://github.com//CAM-SIMA
+cd
+```
+or
+```
+git clone git@github.com:/CAM-SIMA
+cd
+```
+
+### Checking out a tag or switching to a new tag
+
+- To check out a tag:
+```
+git checkout
+```
+
+note that `` can also be the name of a branch or a commit hash. If you specify the name of a branch, you will check out the head of the branch. If you name a remote branch (e.g., `origin/branch_name`), you will create a detached HEAD but you can still use the code. Please note that if you plan on changing the code, first create a branch (see [Working with branches](#working-with-branches)
+
+### Working with branches
+When you create a clone, your clone will contain pointers all the branches that existed at the clone's origin (e.g., the repository at GitHub). While you can check out these branches, however, before attempting to make any changes, you should first create a local version branch (so git can keep track of the local commits).
+
+- To create a new local branch that starts at a certain point:
+
+```
+git branch
+```
+for example
+```
+git branch new_feature cam6_2_024
+```
+
+- To check out a local branch:
+```
+git checkout
+```
+- If you are working with a repository that uses [`git-fleximod`](git-fleximod.md) (e.g., CAM-SIMA, CESM), always run that tool after checking out a new branch or tag:
+```
+bin/git-fleximod update
+```
+
+### Working with remotes (upstream repository locations)
+While working with clones created using the methods above will be sufficient for most if not all of your development needs, there may be times when you will want to access or compare your code with code from a different repository. git has no problem storing revisions from multiple repositories in a single clone!
+
+To begin, your clone probably has a single remote (also known as an upstream repository). To see the current status of which upstream repositories are configured for your clone, use the `git remote` command:
+```
+git remote
+```
+
+To see the location of the remote repositories in your current directory:
+```
+git remote -v
+```
+
+You should see something like:
+```
+origin https://github.com/gituser/CAM-SIMA (fetch)
+origin https://github.com/gituser/CAM-SIMA (push)
+```
+This tells you the "upstream" location from where new code is downloaded (when you run `git fetch origin`) or where code is uploaded (when you run `git push origin `). Note that most `git` commands are purely local, using only information in the .git directory of your clone.
+
+You can rename an existing remote:
+```
+git remote rename origin ESCOMP
+```
+
+You can set the remote name as part of a clone command (the default is 'origin'):
+```
+git clone -o ESCOMP https://github.com/ESCOMP/cam-sima
+```
+
+#### Adding remote (new upstream repository locations)
+To add a new upstream repository, use the `remote add` command. For example:
+```
+git remote add ESCOMP https://github.com/ESCOMP/CAM-SIMA
+git fetch --tags ESCOMP
+```
+You should see messages much like a new clone when you execute the `git fetch` command. Note that you can call the new remote anything, in this example we are calling it ESCOMP.
+
+## Updating your branch to latest development
+Note that while this section explains how to update your local branch to the `ESCOMP/CAM-SIMA/development` branch, the instructions can easily be generalized for any branch from any upstream remote.
+
+Before starting, you should have either:
+
+- A fresh clone of your fork with the branch you wish to update checked out (see [Create a new clone](#create-a-new-clone) and [Working with branches](#working-with-branches).
+- An existing clone with the branch you wish to update checked out and in a clean state (i.e., make sure you do a `git commit` and that `git status` shows no modified files).
+
+Add the upstream remote, if you have not already done so (see [Adding remotes](#adding-remote-new-upstream-repository-locations)).
+
+Merge the specific remote/branch into your branch. In this example, it is ESCOMP/development
+```
+git fetch ESCOMP
+git merge ESCOMP/development
+```
+
+## Comparing differences using git diff
+If you have a git clone, you can view differences between commits or tags. As far as `git diff` is concerned, a commit hash is the same as a tag so in the examples below will use ``.
+
+- To see the full difference between two tags (i.e., a changeset):
+```
+ git diff
+```
+- To see the full difference between the current checkout (sandbox) and a tag:
+```
+ git diff
+```
+- To see only the names of files that are different:
+```
+ git diff --name-only [ ]
+```
+- To see the difference in one or more specific files:
+```
+ git diff [ ] --
+```
+
+### Configuring and using a graphical difference tool
+git has a command, `difftool`, that can run a graphical tool on each file that is different between two commits.
+
+- To configure `opendiff` as the graphical difference tool:
+```
+ git config --global diff.tool opendiff
+```
+- To see the available graphical difference tools:
+```
+ git difftool --tool-help
+```
+- To run `difftool` on `` and ``
+```
+ git difftool [ ] --
+```
+- To optionally run `difftool` on all files in a changeset (answer 'y' or 'n' for each file):
+```
+ git difftool [ ]
+```
+- To run `difftool` on all files in a changeset (i.e., same as 'y' for every file):
+```
+ yes | git difftool [ ]
+```
+
+### Using `diffmerge`
+This section contains the complete modifications needed for using the graphical tool diffmerge for "git difftool" and "git mergetool"
+
+There is a tool called `diffmerge` which enables both side-by-side comparison of edited files in git as well as providing a three way-pane for editing merge conflicts. This tool is available for download at: [https://sourcegear.com/diffmerge/](https://sourcegear.com/diffmerge/). It has been installed on izumi and derecho by the system administrators in public areas, so you don't need to download it for those machines.
+
+To use the differencing tool type: `git difftool`
+
+If after a `git merge` the git command says there are conflicts, then you may type `git mergetool` to allow you to resolve the conflicts and complete the merge. The mergetool with diffmerge properly installed will display three panes. From left to right these panes are:
+
+ Code being merged in Merged code Your code
+
+The panes are opened in a visual editor, and any changes you make in the middle pane, may be saved by selecting the save icon at the top of the screen and then exiting the window. This will finalize the merge for that particular file.
+
+For a presentation which can be used as a tutorial, you may refer to: [Presentation on diffmerge tool for git](https://docs.google.com/presentation/d/1BqrbnpZjjaODVjAtLUCfj-OeBC_4jTPa/edit#slide=id.g2c48a8cae8d_0_75)
+
+The following modifications may be copy/pasted into the user's `.gitconfig` file which resides in the home directory. Since the potential for making an editing mistake is possible, it is recommended that a copy be made of the `.gitconfig` file prior to these edits in case an error is discovered.
+
+If you have problems, check out the tips at the bottom of this page
+```
+[diff]
+ tool = diffmerge
+ algorithm = histogram
+[difftooldiffmerge]
+ cmd = diffmerge \"$LOCAL\" \"$REMOTE\"
+[difftool "diffmerge"]
+ cmd = diffmerge \"$LOCAL\" \"$REMOTE\"
+[difftool]
+ prompt = false
+[push]
+ default = simple
+[merge]
+ tool = diffmerge
+ ff = false
+[mergetool "diffmerge"]
+ cmd = diffmerge --merge --result=$MERGED $REMOTE $BASE $LOCAL
+[mergetool]
+ keepBackup = false
+```
+
+**Useful tips**
+
+- On derecho, if you get an error message about diffmerge not being loaded, make sure you are using `ncarenv/23.09` or newer (to check say "module list")
+- If you do not get a response when clicking on the "git mergetool" window, make sure you find the "conflict" popup and hit "exit" on it
+- If your git mergetool gives blank windows and says: "Files are identical or equivalent under the current RuleSet", hit OK and then go to the File dropdown menu and select "Reload". This filled in the blank windows for the user with this error.
+ - **CAUTION**: Check your file carefully if you do this, as a second user who did this discovered that all of the changes which were automatically merged did not reside in the final saved version.
diff --git a/docs/development/git-faq.md b/docs/development/git-faq.md
new file mode 100644
index 00000000..11fdf790
--- /dev/null
+++ b/docs/development/git-faq.md
@@ -0,0 +1,52 @@
+# git & GitHub FAQ
+
+## Q: How can I clone someone's git clone to my directory on the same machine?
+*A*: `git clone []` where `` is the location of the source git clone and the optional `` is the name of the clone (default is a local directory of the same name as the original).
+
+## Q: How can I clone someone's git clone to my directory on a different machine?
+*A*: `git clone ssh://@: []` where `` is your user name on the remote machine, `` is the machine name (e.g., derecho.hpc.ucar.edu), `` is the location of the source git clone on the remote machine, and the optional `` is the name of the clone (default is a local directory of the same name as the original).
+
+## Q: How can I look at someone's PR code?
+*A*: There a a few ways to do this:
+
+- On GitHub (like looking at any other code on GitHub)
+- Add the PR fork to my remote (allows using tools such as [`git diff` or `git difftool`](git-basics.md#comparing-differences-using-git-diff) with your existing branches or `development`)
+- As a clone (standalone clone on your machine).
+
+A first step is to find the link to the fork's branch. Just below the PR is a line that starts with a colored oval (e.g., "Open" in green) and looks something like:
+```
+Octocat wants to merge 123 commits into ESCOMP:development from Octocat:amazing-new-feature
+```
+
+Clicking on the last part (`Octocat:amazing-new-feature`) will take you to the correct branch where you can browse the code (the first method above). If you want to download that code, click the green "Code" button and then click the clipboard icon. Be sure to take note of the branch name.
+
+- To load this code into your clone, cd to your clone directory, add the PR fork as a new remote, and checkout the branch. For instance:
+```
+ git remote add octocat https://github.com/octocat/CAM-SIMA.git
+ git fetch --no-tags octocat
+ git checkout octocat/amazing-new-feature
+```
+
+Instead of the `checkout` you can also do diffs:
+```
+ git difftool origin/development octocat/amazing-new-feature
+```
+
+- If you want to make a new clone with the PR code, simply do:
+```
+ git clone -b amazing-new-feature octocat https://github.com/octocat/CAM-SIMA.git octocat_cam
+ cd octocat_cam-sima
+```
+
+## Q: Why do Pull Request (PR) code review take so long?
+*A*: A code review must fulfill three purposes:
+
+- Code reviewers must make sure that new and/or changed code does not affect any currently-supported functionality (i.e., it cannot break anything).
+ - While regression tests will catch many of these issues, reviewers must also check for usage of or reliance on deprecated code, and also for any code which is not supported on all platforms and compilers used by the CESM community.
+- Code reviewers must make sure that any new functionality or changes are implemented correctly and at least somewhat efficiently. They must also ensure that important changes are tested against future regressions.
+- The CAM SE team is almost always engaged in several projects to implement new CAM functionality along with supporting infrastructure. Each CAM SE usually looks at each PR in order to prevent new code from interfering with those plans.
+
+The first two steps are usually completed by a single SE although SEs engaged in a final review will often find missed errors. This is similar to peer reviewers finding problems with a paper even after reviews done by colleagues.
+
+## Q: How do I update my branch to the current cam_development?
+*A*: [see this section](git-basics.md#updating-your-branch-to-latest-development)
\ No newline at end of file
diff --git a/docs/development/git-fleximod.md b/docs/development/git-fleximod.md
new file mode 100644
index 00000000..17177cd0
--- /dev/null
+++ b/docs/development/git-fleximod.md
@@ -0,0 +1,19 @@
+# git-fleximod
+## Populate and/or update your externals
+Run `bin/git-fleximod update`
+
+The process will either:
+
+- complete and you're good to go
+- indicate to you that modifications have been made in certain modules
+ - this means you should either go commit those mods or remove them
+ - you can also (if you're sure!) run `bin/git-fleximod update -f` to force overwrite all externals
+
+## Update external repo or tag
+To add or update an external repo to CAM-SIMA, the following steps must be done:
+
+1. Modify the `.gitmodules` file at the head of CAM-SIMA to add or update the external. Explanations for what each entry in the `.gitmodules` file is can be found on Github [here](https://github.com/ESMCI/git-fleximod?tab=readme-ov-file#supported-gitmodules-variables)
+1. Once the `.gitmodules` file has been updated, go to the head of CAM-SIMA and run `bin/git-fleximod update`. This will bring in the new external code into your local repo (but will not commit them).
+1. Once you are ready to officially commit the changes, then make sure to commit both the modified `.gitmodules` file, and the updated submodule itself. An easy way to make sure you have commited everything is to run `git status` and make sure there are no files or directories that have been modified but are still un-staged.
+
+Once all of that is done then congrats! A new external has been successfully added/updated.
diff --git a/docs/development/tool-recommendations.md b/docs/development/tool-recommendations.md
new file mode 100644
index 00000000..a1e8a322
--- /dev/null
+++ b/docs/development/tool-recommendations.md
@@ -0,0 +1,23 @@
+# Tool recommendations
+
+This page lists recommendations for various tools one might use during CAM-SIMA development.
+
+**Please note that these standards are currently in development, and thus any of them could change at any time.**
+
+## Version Control
+
+The recommended version control tool is [git](https://git-scm.com/).
+
+## Repository Hosting
+
+We recommend [Github](https://github.com/) for hosting software repositories
+
+## CI/CD
+
+When possible, we recommend running any CI/CD workflows using [Github Actions](https://docs.github.com/en/actions).
+
+## Container Software
+
+We recommend using [Docker](https://www.docker.com/) for building and running containers.
+
+**More to Come!**
\ No newline at end of file
diff --git a/docs/index.md b/docs/index.md
index 000ea345..4bc4ade5 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -1,17 +1,2 @@
-# Welcome to MkDocs
+# CAM-SIMA developer documentation
-For full documentation visit [mkdocs.org](https://www.mkdocs.org).
-
-## Commands
-
-* `mkdocs new [dir-name]` - Create a new project.
-* `mkdocs serve` - Start the live-reloading docs server.
-* `mkdocs build` - Build the documentation site.
-* `mkdocs -h` - Print help message and exit.
-
-## Project layout
-
- mkdocs.yml # The configuration file.
- docs/
- index.md # The documentation homepage.
- ... # Other markdown pages, images and other files.
diff --git a/docs/stylesheets/extra.css b/docs/stylesheets/extra.css
index 6bf886cf..7eacbd87 100644
--- a/docs/stylesheets/extra.css
+++ b/docs/stylesheets/extra.css
@@ -1,3 +1,3 @@
.md-grid {
- max-width: initial;
+ max-width: 90%
}
diff --git a/docs/usage/creating-a-case.md b/docs/usage/creating-a-case.md
new file mode 100644
index 00000000..edc1591d
--- /dev/null
+++ b/docs/usage/creating-a-case.md
@@ -0,0 +1,112 @@
+# Creating, configuring, and running a case
+Because CAM-SIMA uses the [CIME](https://esmci.github.io/cime/versions/master/html/index.html) build system, creating and configuration a case is mostly the same as it was in CAM7 (and in CESM). But! We're going to assume you know *nothing*.
+
+## 1. Run `create_newcase`
+Before you create a case:
+
+1. Choose the correct machine to run your case on (if this is a high-resolution or long run, you will likely want to choose derecho!)
+1. Consult the [git basics](../development/git-basics.md) if you need a new clone
+1. Make sure you have run `bin/git-fleximod update` to update and/or populate your externals
+1. Navigate to `$CAM-SIMA/cime/scripts`
+
+Assuming you have completed the above steps and have a nice checkout of a development branch of CAM-SIMA, let's go!
+
+Here is the (basic) structure of the command for create_newcase:
+
+```
+./create_newcase --case --compset --res --compiler --project --run-unsupported
+```
+
+To break that down piece by piece:
+
+- `--case ` (optional): The `case` argument specifies the full path to the directory you would like to be created, in which your case will reside.
+ - pro tip: if you choose to put your case directory in your scratch directory, your `run` and `bld` directories will be automatically placed within the case directory
+ - on derecho, this means: `--case /glade/derecho/scratch//`
+ - on izumi, this means: `--case /scratch/cluster//`
+ - if you do not specify a case directory, it will default to your `$CAM-SIMA/cime/scripts` directory (and, eventually, your `run` and `bld` directories will default to a new directory in your scratch space)
+- `--compset ` (required): the compset (or "component set") tells CIME which base-level configurations you want to run with (including model versions and input data sets)
+ - The compset supplied to the `create_newcase` command can be either:
+ - The long name of a compset OR
+ - The alias (if it exists) for the compset
+ - Compset options and aliases can be found in [here](https://github.com/ESCOMP/CAM-SIMA/blob/development/cime_config/config_compsets.xml) (`$CAM-SIMA/cime_config/config_compsets.xml`)
+ - The structure of a compset long name is `_______`
+ - Additional context: for each of the seven model components of the compset configuration, there can be three different implementations:
+ - **active** (prognostic, full): solves a complex set of equations to describe model's behavior (must solve all equations on a numerical grid)
+ - **data**: version that sends/receives same variables to/from other models, but read from files rather than computed from equations (reduces feedback within a system)
+ - **stub**: occupies the required place in the driver but does not send or receive data
+- `--res ` (required): the resolution specifies both the dycore being run and the grid.
+ - The structure of the resolution argument is: `__`
+ - The resolution determines the dycore:
+ - A grid resolution that looks like "ne*" indicates that this is spectral element (SE) cube sphere grid (SE dycore)
+ - A grid resolution that looks like "C*" indicates that this is the FV3 (finite volume cubed sphere) dycore [not yet implemented in CAM-SIMA]
+ - A grid resolution that looks like "mpasa*" indicates that this is an MPAS (Model for Prediction Across Scales) grid (MPAS dycore) [work-in-progress in CAM-SIMA]
+ - You can find the options for these model grid aliases in `$CAM-SIMA/ccs_config/modelgrid_aliases_nuopc.xml`
+ - Most often, for testing CAM-SIMA, we use a coarse SE grid like `ne3pg3_ne3pg3_mg37` or `ne5_ne5_mg37`
+- `--compiler ` (optional): the compiler you wish to use
+ - Options:
+ - on derecho:
+ - intel (default)
+ - gnu
+ - nvhpc
+ - on izumi:
+ - intel (default) - NOTE: intel is not to be trusted on izumi (it's a known-buggy version)
+ - gnu
+ - nag - NOTE: until the nag version is updated beyond 7.0, CAM-SIMA won't work with nag
+- `--project `: a project key to charge to
+ - Only needed on derecho
+ - You can see the project keys you can use [here](https://sam.ucar.edu) (click on your primary group)
+ - You can also see how much of the allocation has been used/is left by navigating to Reports -> Project Search and searching for the key
+- `--run-unsupported`: this flag is mostly in place to indicate to scientists that the configuration they're choosing to run is not scientifically supported.
+
+Given all that, let's say you run the following command on izumi (assuming you're this mysterious person "courtneyp"):
+```
+./create_newcase --case /scratch/cluster/courtneyp/kessler-ne5-gnu-0722 --compset FKESSLER --res ne5_ne5_mg37 --compiler gnu --run-unsupported
+```
+What will happen is that a new case directory will be created here: `/scratch/cluster/courtneyp/kessler-ne5-gnu-0722` that will be configured to run the FKESSLER compset (long name: `2000_CAM%KESSLER_SLND_SICE_SOCN_SROF_ SGLC_SWAV`) with the SE dycore and the ne5 grid.
+
+
+## 2. Configure the case
+Navigate into your shiny new case directory and run `./case.setup`
+
+From here, there are A LOT of configuration options. We'll highlight a few here that you'll use often.
+
+### XML configurations
+Many configurable settings can be found within the env_*.xml files in your case directory. In order to protect again typos, it's not advised to edit those directly. Instead, you will run `./xmlchange` commands to alter those settings (and `./xmlquery` can give you the current setting). You can find additional information on the configurable xml variables [here](https://ncar.github.io/CESM-Tutorial/notebooks/modifications/xml.html). A few to highlight:
+
+- `CAM_CONFIG_OPTS`: This is where we tell CAM-SIMA what physics scheme(s) we wish to run, as well as if we wish to run the `null` dycore.
+ - FKESSLER will default to `--physics-suites kessler --analytic_ic`, which means we're running the `suite_kessler.xml` SDF with the SE dycore (with analytic initial conditions - no need to supply an ncdata initial conditions file)
+ - If you instead want to run a test with the null dycore, you'll want to change this with `./xmlchange CAM_CONFIG_OPTS = "--physics-suites kessler --dyn none"`
+- `CAM_LINKED_LIBS`: Hopefully, you won't have to change this often; however, if you are getting linking errors during your build, try turning off the linked libraries (`./xmlchange CAM_LINKED_LIBS=''`)
+- `DOUT_S`: This is the flag to archive log files of successful runs. During development, we advise turning this off so your log files don't disappear on you (`./xmlchange DOUT_S=False`)
+- `STOP_OPTION` & `STOP_N`: How long you are going to run the model. If `STOP_N` is 8 and `STOP_OPTION` is "ndays", you are setting the model up to run for 8 days.
+ - the options for `STOP_OPTION` are: 'nsteps', 'nseconds', 'nminutes', 'nhours', 'ndays', 'nmonths', 'nyears'
+ - `STOP_N` is an integer
+ - NOTE: if you are running the ncdata_check with snapshot files, keep the number of timesteps you are running at or below the number of timesteps on the snapshot files
+ - NOTE #2: These configurations can be updated without rebuilding the case
+- `DEBUG`: A flag to turn on debug mode for the run (runs without compiler optimizations) - this is very useful during development! It defaults to "False", so you can turn it on with `./xmlchange DEBUG=True`
+- `RUNDIR`: this is the path to the `run` directory for your case. The `bld` directory will exist one level up.
+
+If you run `./xmlchange`, the output will tell you if you need to re-set up and/or do a clean build of your case.
+
+### Namelist configurations
+There are countless namelist configuration options. These can be modified by updating the `user_nl_cam` file in your case directory.
+Namelist options can be updated without rebuilding the case.
+
+- [How to configure the namelist for history/output](history.md)
+- [Information on namelist reader generation](../design/cam-build-process.md/#cam-sima-source-and-namelist-generation-buildnml-workflow)
+- You can see the full CAM6 list of namelist options [here](https://docs.cesm.ucar.edu/models/cesm2/settings/current/cam_nml.html), keeping in mind that most of those options have not yet been ported to CAM-SIMA
+
+## 3. Build the case
+Run `./case.build`
+
+The console output will tell you the progress of the build. If you get to `MODEL BUILD HAS FINISHED SUCCESSFULLY`, hooray!
+
+[Debugging tips](../development/debugging.md#build-errors)
+
+## 4. Run the case
+Run `./case.submit`
+
+The job will be submitted to the system's queue. You can see the status of your job with the `qstat` command. Once it is finished, the log files will be in the `run` directory (unless it ran successfully AND your archiver is on)
+
+[Debugging tips](../development/debugging.md#run-time-errors)
+
diff --git a/docs/usage/history.md b/docs/usage/history.md
new file mode 100644
index 00000000..e3312573
--- /dev/null
+++ b/docs/usage/history.md
@@ -0,0 +1,166 @@
+# History & model output
+
+## Configuring the namelist
+The syntax for updating a the configuration for a history file is:
+```
+;:
+```
+
+Possible namelist options for your `user_nl_cam`:
+
+- `hist_add_avg_fields`: add average fields to a specified history file
+- `hist_add_inst_fields`: add instantaneous fields to a specified history file
+- `hist_add_min_fields`: add minimum fields to a specified history file
+- `hist_add_max_fields`: add maximum fields to a specified history file
+- `hist_remove_fields`: remove a given field from a specified history file
+- `hist_file_type`: type of file (options are "history", "satellite", and "initial_value") - defaults to "history"
+- `hist_max_frames`: maximum number of samples written to a specified history file (after which a new one will be created)
+- `hist_output_frequency`: frequency with which to write samples to a specified history file (syntax is `*` like `3*nhours`)
+- `hist_precision`: precision of the specified history file (options are "REAL32" and "REAL64") - defaults to "REAL32"
+- `hist_write_nstep0`: logical for whether or not to write the nstep = 0 sample (defaults to .false.)
+- `hist_filename_template`: template filename for the specified history file
+ - Defaults to "%c.cam.%u.%y-%m-%d-%s.nc" where "%c" is the case name, "%u" is the volume, "%y" is the year, "%m" is the month, "%d" is the day, and "%s" is the number of seconds since midnight GMT, with the timestamp itself representing the model time when the file is created.
+
+**Example**
+Take the following sample `user_nl_cam`:
+```
+hist_output_frequency;h1: 5*ndays
+hist_max_frames;h1: 3
+hist_add_inst_fields;h1: U
+hist_add_inst_fields;h1: V, Q
+hist_precision;h1: REAL64
+hist_filename_spec;h1: my-history-file%m-%d
+hist_write_nstep0;h1: .false.
+```
+
+It will be parsed by `hist_config.py` and this will be the relevant section of atm_in:
+```
+&hist_config_arrays_nl
+ hist_num_inst_fields = 3
+ hist_num_avg_fields = 2
+ hist_num_min_fields = 0
+ hist_num_max_fields = 0
+ hist_num_var_fields = 0
+/
+
+&hist_file_config_nl
+ hist_volume = 'h0'
+ hist_avg_fields = 'T', 'Q'
+ hist_max_frames = 1
+ hist_output_frequency = '1*month'
+ hist_precision = 'REAL32'
+ hist_file_type = 'history'
+ hist_filename_spec = '%c.cam.%u.%y-%m-%d-%s.nc'
+ hist_write_nstep0 = .false.
+/
+
+&hist_file_config_nl
+ hist_volume = 'h1'
+ hist_inst_fields = 'U', ‘V’, ‘Q’
+ hist_max_frames = 3
+ hist_output_frequency = '5*ndays'
+ hist_precision = 'REAL64'
+ hist_file_type = 'history'
+ hist_filename_spec = 'my-history-file%m-%d'
+ hist_write_nstep0 = .false.
+/
+```
+
+In plain English, a one-month run with these history configuration will result in a total of three files that will look something like these:
+
+- my-history-file01-06.nc
+ - This file will contain instantaneous output for U, V, and Q (eastward_wind, northward_wind, and water vapor)
+ - It will contain three frames, one at each of the following times:
+ - 0001-01-06 (time=5)
+ - 0001-01-11 (time=10)
+ - 0001-01-16 (time=15)
+- my-history-file01-21.nc
+ - This file will contain instantaneous output for U, V, and Q (eastward_wind, northward_wind, and water vapor)
+ - It will contain three frames, one at each of the following times:
+ - 0001-01-21 (time=20)
+ - 0001-01-26 (time=25)
+ - 0001-01-31 (time=30)
+- .cam.h0a.0001-02-01-00000.nc
+ - This file will contain averaged output for T and Q (air_temperature and water vapor)
+ - It will have one frame with the time calculated at the midpoint of the month
+
+## Adding a diagnostic field to the CAM-SIMA source code
+During **init** time, fields can be added to the possible field list with a call to `history_add_field`:
+
+*history_add_field(diagnostic_name, standard_name, vdim_name, avgflag, units, gridname, flag_xyfill, mixing_ratio)*
+
+| Field | Optional? | Type | Description |
+|:---------------------|-----------|-----------|------------------------------------------------|
+| diagnostic_name | No | string | diagnostic name for the field - will be the name in netcdf output file |
+| standard_name | No | string | CCPP standard name for the variable |
+| vdim_name | No | string | vertical dimension: 'horiz_only' for no vertical dimension; 'lev' for vertical_layer_dimension; 'ilev' for vertical_interface_dimension |
+| avgflag | No | string | default average flag; options: 'avg', 'lst' (instantaneous), 'min', 'max', 'var' (standard deviation) |
+| units | No | string | variable units |
+| gridname | Yes | string | gridname on which the variable's data is mapped (defaults to the physics grid) |
+| flag_xyfill | Yes | string | fill value for variable values |
+| mixing_ratio | Yes | string | constituent mixing ratio type ('wet' or 'dry'); not set to anything if not passed in |
+
+Example:
+```
+call history_add_field('Q', 'water_vapor_mixing_ratio_wrt_moist_air_and_condensed_water', 'lev', 'avg', 'kg kg-1', mixing_ratio='wet')
+```
+
+It's important to avoid adding calls to `history_add_field` to the CCPP-ized physics schemes (to keep them portable). Instead, create a new diagnostics scheme and place that in the `diagnostics` folder of the atmospheric_physics repo. The `history_add_field` call will be in the `init` phase.
+
+## Outputting a diagnostic field to the CAM-SIMA source code
+*After init time*, a variable's current values can be captured for output with a call to `history_out_field`:
+
+*history_out_field(diagnostic_name, field_values)*
+
+| Field | Optional? | Type | Description |
+|:---------------------|-----------|------------|-----------------------------------------------|
+| diagnostic_name | No | string | diagnostic name for the field - will cause an error if the diagnostic name was not added via history_add_field |
+| field_values | No | real | variable values to be stored in the history buffer(s) |
+
+Example:
+```
+call history_out_field('Q', const_array(:,:,const_idx))
+```
+
+It's important to avoid adding calls to `history_add_field` to the CCPP-ized physics schemes (to keep them portable). Instead, create a new diagnostics scheme and place that in the `diagnostics` folder of the atmospheric_physics repo. The `history_out_field` call(s) will likely be in the `run` phase.
+
+## Using the output
+The output files can be found in your `run` directory. They are in NetCDF format.
+
+See the [ADF](https://github.com/NCAR/ADF) for lots that you can do with your results!
+
+A few useful commands for inspecting NetCDF data:
+
+- To output the header/metadata information for the file (includes list of variables on the file):
+```
+ncdump -h file.nc
+```
+
+- To output the data for a specific variable on the file:
+```
+ncdump -v file.nc
+```
+
+- To compare two files*:
+
+```
+cprnc
+```
+- To get a slice of one file (subset of time samples):
+```
+ncks -F -d time,,,
+```
+Example to get time samples 3-5 from a file onto a new file called "split-file.nc":
+```
+ncks -F -d time,3,5,1 file.nc split-file.nc
+```
+
+- To diff a variable between two files (and output the results to a new netcdf file):
+```
+ncdiff -v -o
+```
+
+*`cprnc` can be found:
+
+- on derecho: `/glade/p/cesmdata/cseg/tools/cime/tools/cprnc/cprnc`
+- on izumi: `/fs/cgd/csm/tools/cime/tools/cprnc/cprnc`
\ No newline at end of file