-
Notifications
You must be signed in to change notification settings - Fork 4
Summary of JEDI Directory Structure
JEDI applications are organized into high-level bundles that conveniently gather together all the git repositories necessary for the JEDI applications to run. Often a bundle is associated with a particular model, in our case, FV3, but there are also others such as MPAS or WRF. Other times, such as the ufo-bundle, the bundle may be just for development or demonstrative purposes. If one looks at the repository/the fv3-bundle directory before running ecbuild, the directory will only contain a few files in it. The only important file in this repository is the CMakeLists.txt file, described on another page, that tells ecbuild which repositories to clone and which branches/tags to checkout.
After running ecbuild, the bundle source directory will contain a series of subdirectories, each cloned from their respective GitHub repositories. For the fv3-bundle this includes:
- crtm
- femps
- fms
- fv3
- fv3-jedi
- fv3-jedi-lm
- ioda
- jedicmake
- oops
- saber
- tutorials* (note this is in the fv3-bundle repository, not its own)
- ufo
The basic structure of most of these repositories/directories is the same and will be described in greater detail in the subsequent sections. The non-JEDI repositories will not be as standard (e.g. FMS or FV3), and will not be described in detail here. This document assumes the reader will either 1) not have to modify model source code or 2) already has a working knowledge of these other repositories.For each section below, replace “component” with any option in the following list.
- ufo
- saber
- oops
- ioda
- fv3-jedi
This is the top level of each repository, which mostly contains a number of subdirectories, and like the bundle directory, a CMakeLists.txt file. This CMakeLists.txt file is not nearly as important (to most people) as others, as it really only sets up the dependencies for this component, defines each subdirectory (ex. src/ or test/), and other configuration options to build this component’s libraries/executables.
Generally speaking, each component/ directory may have these subdirectories:
- CI - information for the continuous integration software that runs for GitHub
- cmake - files needed for the cmake build
- docs - doxygen configuration files
- ewok - files to connect to JEDI EWOK workflow system
- src - component source code files
- test - files needed for tests
- tools - miscellaneous tools/scripts A few of the more important subdirectories will be described in the next sections.
The component’s src directory is generally broken down into two subdirectories, mains and another directory with the same name as the component (ex. ufo/src/ufo).
The src/mains/ directory contains source code for all of the stand-alone executables to be built (not unit tests) by this component and the CMakeLists.txt file tells ecbuild/cmake about each of these executables.
For UFO, for example, there is only one (as of the time of writing) .cc file here (and it’s corresponding .h file), an executable to run the CRTM. But for FV3-JEDI, there are over a dozen .cc files (their .h files are in OOPS). These executables will be briefly described in the build section later, but you can see by filenames here that there are executables to convert the model state between grids, to add increments, to difference model states, to run variational data assimilation, to run ensemble data assimilation, to compute H(x), and more.
The component/src/component directory (ex. ufo/src/ufo) is the primary source directory for each JEDI component. Within this directory, you will find the vast majority of the Fortran and C++ source files.
In UFO, for example, there are a number of subdirectories, one for each type of generic observation operator (ex. crtm, atmvertinterp, identity) that house the C++, Fortran, and interface source code for the respective operator. Some of the operators are down one additional layer, for example the marine/ directory contains marine specific observation operators. There is also a filters directory which contains the code for the generic quality control filters.
In FV3-JEDI, the source code is broken up into subdirectories based on function/type. There are directories for the model IO, the model geometry, the increment, and others. Each of these subdirectories then contains the relevant source code as the directory name indicates. Sometimes these are split between gfs and geos, as GFS/UFS may handle some things differently than GEOS does (the model restart files, for example). even though it is the same model dynamical core.
JEDI’s main components are written in C++, but the models (and most observation operators) are written in modern Fortran. Because of this, there are often .interface files in the source code subdirectories. These interface files, one each in .h and .F90, provide for “translation” between the high level C++ routines and the Fortran routines that are actually doing the heavy lifting (calculation, IO, etc.). These interface files will mostly be formulaic, so if you need to add a new observation operator, look at an existing set (.interface.h and .interface.F90) as an example.
This directory contains all of the files (source code, YAML, even possibly some input data depending on component) needed for the JEDI ctests for this component.
The CMakeLists.txt file in this directory is used to set up the environment at runtime for all of the ctests (including linking required input files such as YAML, or reference datasets). It also contains the commands to tell cmake/ecbuild which tests to create.
Within this test directory, there are subdirectories similar to the source code directory, mains, and possibly, “component”. These are the top level C++ source code files for the executables that are created just for testing purposes. For components without a component/test/component directory (ex: fv3-jedi/test), the header files are in OOPS.
The testinput/ subdirectory contains YAML files for each of the tests for this component. For FV3-JEDI, the test name should/will match the YAML filename, but this is not necessarily the case for UFO, particularly since multiple tests will use the same YAML file (for example the non-linear operator and the TL/AD tests). A few of these YAML files from different components will be described in the Introduction to YAML section later.
There may also be a testoutput/ subdirectory for some of the components, for example, FV3-JEDI. In this directory, there are reference files that are compared against the output of the ctest unit tests, and verify that the values are the same to within some tolerance. For many components, these are just simply text files of stdout, but in other cases, like ioda-converters, the testout files are sample netCDF IODA files.
Another optional subdirectory is Data/. In FV3-JEDI, for example, this contains a number of subdirectories, some with YAML files, others with netCDF files, and others with different types of fixed input files (Fortran namelists for example). All of these various input data files will not be described here, but when the YAML files are dissected later, if a path points to this directory, it will briefly mention what that file is for. Depending on if you have run the ctest that grabs the additional test data yet, there may be even more files/directories in Data/, such as sample model backgrounds.
As the name suggests, this is where all of the JEDI executables are placed after they are built. There will be something like 100 different executables in this folder, but we only care about a few of them (at this time).
All of the executables that start with “test_”, are test executables. Many (but not all) of the ctests use these specific test executables. Notably, all the UFO obs operator and linear operator tests use test_ObsOperator.x and test_ObsOperatorTLAD.x, respectively, as they use input test model data that is not representative of an actual 3D model state, but rather pre-interpolated to the observation location. There are other examples here of test executables that do not function (nor do their input YAML files look) the same as a real JEDI application does.
Speaking of “real” JEDI applications, let’s look at the executables that begin with “fv3jedi_”. A list of some of the ones that will be used in this tutorial include:
- fv3jedi_diffstates.x - Use this application to difference two states and write an increment file
- fv3jedi_hofx_nomodel.x - Takes input IODA observations, and a model background and writes H(x) to output IODA file(s)
- fv3jedi_parameters.x - Use this to generate parameters for BUMP to use as input for other applications
- fv3jedi_var.x - Variational data assimilation solver application
Some other useful ones may include:
- fv3jedi_addincrement.x - Basically the opposite of fv3jedi_diffstates.x, takes an input state (background) and increment and produces an analysis
- fv3jedi_convertstate.x - Comparable to chgres_cube, will regrid FV3 tiles to different resolutions or even write out a lat/lon grid file for simplified plotting/post processing.
- fv3jedi_letkf.x - LETKF data assimilation solver application
Note that again, the ioda-converters are separate, but they also have a build/bin directory containing useful Python scripts and other executables to convert observations to IODA netCDF files.
Each component has, just like in the source code bundle, a corresponding directory in the build directory. If you run ‘ctest -N’ in these directories, you only see the tests for this specific component. As the final executables are in build/bin, and there are a bunch of cmake/ecbuild files/directories added here, we only care about looking in the component/test directory from here on out.
The component/test/ directory is the actual runtime directory for all of the ctests. When you look at the YAML for each test, the relative paths to data are assumed to be relative to this directory. There is a list of files, some empty, some executables, that are what cmake uses to determine what the available ctests are for each component.
Within component/test/Data, the list of files will look different depending on component. We will focus on just UFO and FV3-JEDI here. First in UFO, this directory contains input test observations (ioda subdirectory) and corresponding test GeoVaLs interpolated to the observation locations (ufo subdirectory). These are now either linked by ecbuild from a standard location on a supported machine, or downloaded as part of a ctest to grab test data. Additionally, there are symbolic links of numerous CRTM fixed files (coefficients for each satellite instrument) that again are either linked from a standard location or downloaded in a ctest.
In fv3-jedi/test/Data, there are a number of subdirectories, each with a descriptive name. Some, like fv3files and fieldsets, contain files that are symbolically linked from the source code corresponding fv3-jedi/test/Data directories. Others, such as inputs and obs, are linked to the build/test_data (to be described below) directory. Finally, there are other directories such as hofx, analysis, and increment, where the FV3-JEDI ctests write output to. If you’ve run the correct tests, you can look at the model analysis files in analysis, the increments in increment, and O-F, O-A, and H(x) values in hofx.
This directory contains symbolically linked YAML input files for the various ctests for this component. They are linked to the corresponding testinput directory in the source code, so any changes made here will also be made to the git repository.
Not all components will have this subdirectory. UFO, for example, does not. But for FV3-JEDI, this contains first a series of symbolically linked reference text files to compare to the output of some of the ctests. Once the tests have been executed, there will be output files that match the filename of the links, except ending in .run instead of .ref. These are compared within some tolerance to make sure the tests pass. There may also be files that end in a series of numbers, these represent each PE for tests that are run with multiple processors.
The test_data/ directory has subdirectories for some of the components (mainly CRTM, IODA, and UFO). Within each of these subdirectories is another subdirectory, named for either the git branch or tag that is being built by ecbuild. These then contain the coefficient files (CRTM), IODA observation files, or GeoVaLs for UFO tests which are symbolically linked to their respective build/component/test directories. The test_data subdirectories are populated either through running certain ctests designed to download the data, or by copying from a fixed directory on supported platforms (such as Hera).
The Testing/ directory (which also exists in component/Testing) contains a couple of useful files. Testing/Temporary/LastTest.log will save the output of the last ctest that you ran, in case you want to parse the output. Additionally, there is Testing/Temporary/LastTestFailed.log, which will show the output of the last failed ctest.