Skip to content

Latest commit

 

History

History
405 lines (284 loc) · 16 KB

README.md

File metadata and controls

405 lines (284 loc) · 16 KB

Organization

The instructions for generation of MOM6 documentation is divided into three sections.

Terminology and abbreviations

To assist with writing of this documentation, we use several abbreviations.

DOX: Doxygen refers to the markdown language used by doxygen or a doxygen file.

RST: reStructuredText; a markdown language leveraged by sphinx via docutils.

RTD: Read the Docs

Documentation process

An API and user manual are the main forms of the MOM6 documentation produced.

API Manual

The API (reference) manual contains descriptions of all the modules, subroutines, functions, arguments and members that form data structures for variables. This version of the html documentation is hosted here: APIs

This version of the manual utilizes the html output of doxygen.

User Manual

A user manual format of the manual contains more descriptive materials. This version of the manual may also contain the APIs. RTD is a popular hosting service for documentation. A copy of the may be found here: MOM6 User Manual.

Please see the software operation section for details on how to produce various portions of this documentation locally and on your own RTD site as desired.

Overview

Source code with embedded documentation is processed by doxygen.

Doxygen in run in two modes. For the API manual, it is instructed to generate only the html version of the documentation. For the user manual, it is instructed to generate intermediate files for use with sphinx and RTD. For purposes of this documentation the API manual is referred to as generated using html(doxygen). The html generated by sphinx is referred to as html(sphinx). The same references may be applied to the pdf versions: pdf(doxygen) and pdf(sphinx).

If the software is installed, local copies of html and pdf instead of those found at RTD. The html and pdf versions produced by doxygen will not have any of the additional documentation from the sphinx portion of the pipeline.

RTD by default runs html(sphinx) pipeline and adds a bit of styling. The RTD can be instructed to also produce a PDF. However, the current pipeline exceeds 900 seconds execution time and fails if RTD is instructed to generate the html and PDF in one session. By default, the PDF generation is currently turned off.

Starting structure

We define SRC as the root of the source directory assuming the $(SRC) directory was created from downloading the tree from github using git clone.

The starting state of directories and files:

SRC/
  config_src/
    drivers/
      FMS_cap
      ice_solo_driver
      mct_cap
      nuopc_cap
      solo_driver
      unit_drivers
    external
    infra/
      FMS1
    memory/
      dynamic
      dynamic_symmetric
  pkg/
    CVMix-src
    GSW-Fortran
  src/
    ALE
    core
    diagnostics
    equation_of_state
    framework
    ice_shelf
    initialization
    ocean_data_assim
    parameterizations
    tracer
    user
  docs/
    _static/
    _templates/
    *.rst
    api/*.rst
    images/*

Doxygen

The Doxygen package is used to scan the source code for embedded documentation. Doxygen also automatically produces API documentation from arguments to subroutines, functions and member of types. The guide for using doxygen in MOM6 is hosted on the MOM6 developer's wiki.

The majority of documentation is embedded in the comments of the source code. Additional documents have been added in the form of doxygen files (*.dox). The additional documents can be identified as those prefixed with an underscore (_) and have the dox extension.

The content of the additional documents and source code should conform to usage as defined by the doxygen user and reference manuals. Further guidance is provided on the MOM6 developer's wiki. Troubleshooting guides are provided with most commonly reported problems with MOM6 documentation.

NOTE: Not all doxygen commands are supported through the sphinx documentation processor. Support can be added by adding an issue to the github repository.

For the API documentation, the resultant tree will look like this:

SRC/
  docs/
    _build/
      doxygen_warn_rtd_log.txt
      [html|latex]
        _static
    MOM6.tags

The main driver for doxygen is a configuration file. The content on MOM6 developer's wiki uses the Doxyfile_nortd configuration file.

By default, the html directory is only available after processing the documentation. Please see software operation on how to generate the pdf companion of the documentation.

For the user manual documentation run, this is the first phase of the process. This process produces the following directories and files:

SRC/
  docs/
    _build/
      doxygen_warn_rtd_log.txt
    xml/
    MOM6.tags

The documentation generated by Sphinx and RTD uses the Doxyfile_rtd. These options can be overridden at the command line. See software operation for more details.

The _build/doxygen_warn_rtd_log.txt should be reviewed for warnings and errors.

Sphinx

The Sphinx package renders final versions of html and pdf documents. The provided RST files (*.rst) provide the organization structure for the html and pdf documents. It is recommended that any significant content be pushed into the source code or additional dox files.

During a complete documentation run, this is the second phase of the process. Files and directories created:

SRC/
  docs/
    _build/
      html_log.txt
      doctrees
      [html|latex]

The directories, html and or latex, will be present depending on how Sphinx was called. The doctrees directory contains processed documents saved as python pickles (binary data structures).

The html_log.txt should be reviewed for warnings and errors.

Read the Docs

The RTD site can be configured to watch for updates on a github repository. A documentation update may be triggered when an update is pushed to the repository. The entire documentation process is run twice. The first run produces html. The second run produces a pdf (if so instructed).

NOTE: There is a rough execution time limit of about 900 seconds. Trying to do more than that will cause a "timeout" error.

NOTE: To post process equations within RTD, see UPDATEHTMLEQS below for more information.

Special Cases

Please see the details for special cases to the documentation syntax not found in any of the user manuals for Doxygen or Sphinx.

Software operation

The entire documentation processing pipeline can be operated using the provided Makefile. Running make or make help will provide some helpful information.

Otherwise, savvy users can run individual pieces on the command line using provided configuration files.

Certain environment variables may be set prior or during the make command to influence the documentation build process. Explanations on how to run various parts of the pipeline are given below.

Doxygen

For generation of the API portion of the documentation, see the MOM6 developer's wiki.

The configuration file used for doxygen is Doxyfile_nortd.

From the Makefile, the API portion of the pipeline may be run using: make nortd

NOTE: make nortd will attempt to install doxygen if a binary is not found.

Makefile

See below, the following Makefile options are honored:

  • BUILDDIR
  • DOXYGEN_BIN
  • DOXYGEN_CONF
  • DOXYGEN_RELEASE
  • UPDATEHTMLEQS
  • UPDATEHTMLEQSVERBOSE
  • PAPER

Sphinx

The full documentation can be generated locally with

# NOTE: make sure _build directory exists
make html >& _build/html_log.txt

which will generate html in docs/_build/html/. Start at docs/_build/html/index.html. This will also place log information into the _build tree that is viewable. Warning messages from doxygen is written to _build/doxygen_warn_rtd_log.txt.

To fix equation targets and references the full update to the RTD/Sphinx documentation requires:

make html UPDATEHTMLEQS=Y

Options

Many options can be passed to sphinx with make on the command line after the last argument. e.g. make html SPHINXOPTS=-v

Multiple arguments may be passed using a space as separation between arguments.

Makefile

Useful combinations:

  • make clean (Cleans the working directory of most intermediate and final documentation)
  • make Clean (Removes the _build and xml directory and MOM6.tags file)
  • make html (Builds html based documentation)
  • make latex (Builds latex based documentation; does not compile a pdf)
  • make latexpdf (Attempts to build a final pdf of the documentation)

Run make or make help for other options.

The doxygen generated HTML can be obtained locally (and slightly more quickly) with

make nortd

which will generate html in docs/APIs/. Start at docs/APIs/index.html. If doxygen is not already available this will install a local copy of doxygen.

SPHINXOPTS

This option is passed to sphinx-build. See sphinx-build --help for more information.

BUILDDIR

The default output directory is _build. The default doxygen configuration files are expecting an output directory of _build.

UPDATEHTMLEQS UPDATEHTMLEQSVERBOSE

If UPDATEHTMLEQS is set to Y, postProcessEquations.py will be run to attempt to update equation numbers in the generated html. This environment variable has to be added to the RTD administrative interface if you want to trigger post processing of equations.

If UPDATEHTMLEQSVERBOSE is set to -v this will turn on verbose printing for the post processor.

NOTE: These options affect solo doxygen html processing only for the make nortd option. For sphinx, they are utilized in the sphinx python module to handle post processing and are not part of the Makefile. This was done as RTD runs sphinx processing directly using sphinx-build and not the Makefile.

PAPER

The default value is empty (PAPER=). There are two options currently available in the Makefile (a4 or letter). The PAPER argument has no impact on html rendered content.

conf.py

Other options are read by conf.py and can be passed on the command line when running make. These options may also be set in the shell environment prior to running make. These can be set on RTD too.

DOXYGEN_BIN

Using this option, you can specify a different doxygen binary for processing the documentation. Otherwise, a doxygen executable needs to be available in the system path.

DOXYGEN_CONF

Use this option to specify a custom configuration file to doxygen.

DOXYGEN_RELEASE

This option allows selection of a doxygen version from https://github.com/doxygen. The default is Release_1_8_13. NOTE: This command does not have any impact if an existing binary is found.

NCAR_FORK

This option activates the NCAR configuration file: ncar/Doxyfile_ncar_rtd

Software installation

On a relatively bare system, you can install a fairly stable documentation pipeline.

Dependencies

If you do not have doxygen, to build a local version of the doxygen processor you will need the following packages:

  • cmake
  • g++ (or a c++ compiler)
  • flex
  • bison
  • graphviz
  • pdf2svg

(e.g. apt-get install cmake g++ flex bison graphviz)

If you are building the full generated sphinx documentation you will need the following packages in addition to those for doxygen above:

  • libxml2-dev
  • libxslt-dev (may also be called libxslt1-dev)

(e.g. apt-get install libxml2-dev libxslt-dev)

We strongly recommend using python3 with its virtual environment and pip3.

(e.g. apt-get install python3 python3-venv python3-pip)

Before running sphinx (make html) you will need to issue:

pip install -r requirements.txt

You may need to use pip3 to install requirements for python3.

Requirements:

  • sphinx
  • sphinx-rtd-theme
  • sphinx-bibtex
  • sphinx-fortran
  • sphinxcontrib_autodox-doxygen
  • flint
  • lxml
  • numpy
  • future

For machines that need to build future, numpy or lxml, these packages are required:

  • Cython
  • wheel

Latex/PDF generation

PDF generation requires the following packages

  • texlive-latex-base
  • texlive-latex-recommended
  • texlive-latex-extra
  • latexmk

doxygen

You may choose to download the source.

Latest is doxygen-1.8.20.src.tar.gz.

tar xzf doxygen-1.8.20.src.tar.gz
cd doxygen-1.8.20
mkdir build
cd build
cmake -G "Unix Makefiles" ..
make
sudo make install

The makefile for doxygen attempts to install the compiled version into /usr/local/bin. You can link to a specific executable within the virtual environment. At this point we also recommend renaming doxygen to doxygen-1.8.20 within /usr/local/bin.

NOTE: The makefile for the documentation framework will attempt to compile a local doxygen binary of version 1.8.13 if a binary cannot be found in the $PATH.

Testing

A lot of manual testing has been completed using the following versions:

  • 1.8.13
  • 1.8.14
  • 1.8.19
  • 1.8.20

Read the Docs

The Read the Docs (RTD) site uses a virtual machine (VM) for processing documentation. The VM architecture is type x86_64. The default version for doxygen is 1.8.13 on the RTD VM.

NOTE: Using modified python modules on RTD is possible through careful crafting of the requirements.txt file. It is impossible to replace system binaries or compile code on RTD. It is possible to ship replacement binaries that can be run from the repo. For security reasons, a binary cannot be included in the MOM6 repository.

Logfiles

See Sphinx run options below. We capture up to three logfiles for RTD. The main default logfile is LOGS/doxygen_warn_rtd_log.txt.

Logfiles were renamed to *.txt to allow easier access and viewing from most websites. Most websites force download of *.log files.

Credits

2020

The documentation pipeline was upgraded by Rob Cermak and Marshall Ward. Four modified python modules are required to process the MOM6 documentation. The versions are tagged and placed into the production version of requirements.txt. Development versions may be found in the respective dev branches.

Source Modified Version Development
sphinx sphinx-3.2.1mom6.4 B:3.2.1mom6.4 B:dev
sphinxcontrib-autodoc-doxygen sphinxcontrib-autodoc-doxygen T:0.7.13 B:dev
sphinx-fortran sphinx-fortran T:1.2.2 B:dev
flint flint T:0.0.1 B:dev
MOM6 esmg-docs esmg-docs B:esmg-test

T: tag B: branch

2017

The sphinx documentation of MOM6 is made possible by modifications by Angus Gibson to two packages, sphinx-fortran and autodoc_doxygen.