User Guide: convert to sphinx

[hjoliver]

LaTeX produces book-quality PDF and hardcopy, but the generated HTML is awful.

The new sphinx Rose docs are very nice: https://metomi.github.io/rose/doc/html/index.html

Email from @oliver-sanders on this:

A good chunk of the work for the new Rose documentation was in developing the infrastructure to build the docs e.g:

• Animated cylc graph diagrams.
• System for auto-generating the CLI docs.
• System for creating / auto-generating the rose API docs.
• Pygments lexers for the Rose and Cylc configuration formats.

This code can be easily copied and pasted into cylc.

As regards decanting the current LaTeX documentation into a sphinx project:

The documentation would need to be broken down into a file hierarchy. It's possible that it might be auto-convertible to some extend but this might not work with some of the LaTeX modules we are using. Some of it would definitely need to be processed by hand e.g. some references.

Sphinx supports multiple documentation builders including PDF so you don't have to loose the hard copy. Some highlighted builders:

• Single page HTML
• Directory HTML [as used by rose]
• Slides (via HTML) [as used by rose]
• PDF (via LaTeX)
• PDF (via RST2LaTeX)
• Epub3 (via HTML)

http://www.sphinx-doc.org/en/stable/builders.html

Assuming no major changes to the documentation itself it's probably not that big a job. Sphinx has a tool which can be used to link documentation between sphinx projects which might form a useful bridge between the rose and cylc documentation:

http://www.sphinx-doc.org/en/master/ext/intersphinx.html

[oliver-sanders]

Sphinx Infrastructure For Cylc Documentation

Rather than opening loads of issues for different aspects of the documentation I've listed them here. This comment relates to the "infrastructure" side of creating a sphinx project into which the LaTeX documentation can be decanted.

Auto Documentation

Sphinx makes auto-documentation easy providing a good way to reduce the maintenance and the risk of documentation diverging from the code base.

The Suite Configuration API

As cylc has a rigid API we could auto-document this to prevent the need to document it separately in LaTeX. See #2660 (comment).

Ideally we would also bring in information from the upgrader for:

• Deprecation warnings
• Version added markers
• Timeline of suite API changes
  • It would be useful to have a document we can point users at where they could see find a list of syntax changes between any two cylc versions.

This would require:

• Documentation to be moved into the cfgspec modules.
  • Due to the volume of text this may have an impact on memory usage, weather this is likely to be significant or not I don't know.
  • One option to avoid this may be to place the documentation in comments, sphinx supports parsing comments prefixed #: (http://www.sphinx-doc.org/en/1.4.8/ext/autodoc.html#directive-autoattribute).
• A sphinx domain for the cylc suite API
  • This could more or less be copied from the rose domain (https://github.com/metomi/rose/blob/2018.05.0/sphinx/ext/rose_domain.py

Cylc Suite API (HTTP)

See API On The Fly - #2123

This would require:

• Sphinx domain for HTTP(S) interfaces
  • https://sphinxcontrib-httpdomain.readthedocs.io/en/stable/
• The documentation component of API On The Fly

Cylc API (CLI)

With Rose we wrote a parser which converts the standard CLI help into rst (http://metomi.github.io/rose/doc/html/api/command-reference.html).

We could use this code or adopt an alternative markup e.g. markdown.

This would require:

• Small changes to the syntax of cylc CLI help (e.g. using backquotes for literals).
• A system for building this into the documentation.
  • https://github.com/metomi/rose/blob/2018.05.0/sphinx/ext/auto_cli_doc.py

Cylc Python Interfaces

If we wish to expose any python interfaces, sphinx can auto-document from the source code.

Rose example: http://metomi.github.io/rose/doc/html/api/configuration/api.html#python

For the moment the only modules I can think we might want to auto-document from the source code are the Jinja2 filters.

In the future the main driver for this would be the Python API For Suite Definition - #1962

This would require:

• Standardising on a particular docstring format, options:
  • Plain rst
  • Google style docstrings (as used by rose)
  • Numpy style docstrings
  • See http://www.sphinx-doc.org/en/master/ext/napoleon.html#google-vs-numpy
• Improving the docstrings of chosen modules

Software Requirements

Rose now auto-documents its software requirements from the rose check-software command (http://metomi.github.io/rose/doc/html/installation.html#system-requirements). It should be fairly straightforward to do the same with cylc check-software.

This would require:

• The cylc check-software to have an option to generate rst output
  • https://github.com/metomi/rose/blob/2018.05.0/bin/rose-check-software#L351
• A sphinx extension to include this into the built documentation
  • https://github.com/metomi/rose/blob/2018.05.0/sphinx/ext/script_include.py

The Suite Database

See https://github.com/cylc/cylc/issues/2483, I don't know of any Sphinx extensions which can be of help here.

This would require:

• A database domain for Sphinx
• A method of documenting the database in the source code (inserting comments into the SQL create table statement?)

Cylc Specific Sphinx Extensions

Desired / required Sphinx extensions:

• Pygments lexer for cylc suite.rc files and the cylc graph language
  • Done - https://github.com/metomi/rose/blob/2018.05.0/sphinx/ext/cylc_lang.py
• Validation of cylc code examples?
• Graphviz extension for drawing cylc graphs
  • Done - http://www.sphinx-doc.org/en/master/ext/graphviz.html)
• Extension for drawing simple graphs from cylc graph strings and animating simple cylc suites in simulation mode
  • Simple implementation - https://github.com/metomi/rose/blob/2018.05.0/sphinx/ext/minicylc.py)
  • Further work (required/desired):
    • Conditional expressions
    • Cycling
    • Suicide Triggers
    • Parametrisation

Production Of Documentation For Multiple Versions In Multiple Formats

See the version selector in the bottom left-hand corner of the rose documentation http://metomi.github.io/rose/doc/html/index.html.

We can serve multiple versions of the documentation in formats including:

• HTML
• HTML Slideshow
• PDF
• ePub

This would require:

• A system for running sphinx builds in a structured way
  • In rose we build in the directory doc/<version>/<format> with a symlink doc pointing at the default version.
  • Rose user guide: versioning metomi/rose#2190
• For PDF output we may want to write a style file

Production Of Multiple Documents Under One System

We currently serve two PDF documents from the cylc website:

• Cylc User Guide
• Suite Design Guide

Sphinx builds PDFs from a file hierarchy meaning that we can build multiple documents from a single source tree. I.E. we could move the Suite Design Guide into Sphinx as well.

Documents we may want to serve:

• Cylc User Guide
• Suite Design Guide
• Cylc Tutorial
• Cylc API Reference

This would require:

• Nothing, Sphinx does this out of the box :)

[hjoliver]

@oliver-sanders - wow, excellent!

[oliver-sanders]

metomi/rose#2216 means that objects documented in the Rose documentation (e.g. Rose configurations, environment variables, etc) can now be cross-referenced in other projects.

This opens up cross-referencing between the Rose and Cylc documentation at an object level. If the projects become out of sync and references get broken Sphinx will raise warnings accordingly.

It would be fairly streightforeward to extend this functionality to raise warnings if referencing deprecated configurations etc.

[wxtim]

I've just had a look at a ticket #2747 which raises a few more points:

1. Can we use cylc graph to generate diagrams of example suites?
2. Can we ensure that graph examples are rendered a style similar to the one the user will see.

[sadielbartholomew]

Can we use cylc graph to generate diagrams of example suites?
Can we ensure that graph examples are rendered a style similar to the one the user will see.

For the new Rose docs we use the digraph directive from the sphinx.ext.graphviz extension, which maps a graph string to an embedded graph diagram, in the same styling as cylc graph given both are displayed using Graphviz (via PyGraphviz in the latter case).

The difference in specification for a graph string is minimal, with digraphs being specified via the DOT language, e.g.:

one -> two -> three

which would clearly take only a simple sed replacement on = in a dependency 'arrow' to convert from a suite.rc graph string, if it is not quicker to do that manually.

The only cases I can think of where it would be time-consuming &/or tricky to get equivalent graphs converted into digraphs for the docs would be when a suite has many different graph strings which combine in a non-trivial way e.g. due to multiple recurrence sections & inter-cycle dependencies, or has many conditional triggers or parameters.

But since the CUG has mainly simple example suites, by nature, I think it would not be worth the effort to do anything fancier e.g. with cylc graph. The few complex scheduling sections from the CUG suites could be manually converted. The in-built graphs from the sphinx.ext.graphviz digraph (e.g. those here from this directive) are certainly better than cylc graph screenshots, as the task text can be highlighted & they are sized appropriately, etc. Though that is just IMO!

[hjoliver]

I definitely like the new Rose sphinx approach.

However, I guess only cylc can generate cycling graphs, for those examples - unless you manually code the all the cycle point instances shown, as if they were different tasks.

are certainly better than cylc graph screenshots,..

I'll just fight back here a tiny smidgeon 😁 - you don't have to take screenshots. cylc graph -O foo.png suite.rc or even cylc graph -O foo.svg suite.rc (i.e. SVG format).

[sadielbartholomew]

you don't have to take screenshots

@hjoliver good point! I should have said images as well as, or instead of, screenshots. Either way, they would not be built into the documentation to be displayed nicely & to make change diffs explicit etc. I personally prefer the embedded graphviz extension directives.

There would be some drudgery required to either manually convert some aspects of non-trivial docs example suites or get a working cylc graph graph-generator in the docs, so it depends what route one thinks will produce a better result for the least effort.

[oliver-sanders]

However, I guess only cylc can generate cycling graphs

I created the minicylc directive which can read basic graph strings and turn then into an animated SVG using Javascript. Optionally it can also put the graph string in a nicely formatted code box above the graph to ensure the graph is kept in sync with the code (#2874, metomi/rose#2248 et al).

This was just a quick bit of JS, nothing serious, I would have taken it further to make the documentation more illustrative (e.g. explaining the concept of a stalled suite is much easier with an animation). In the end I moved onto more pressing matters and left this one by the wayside, but it's perfectly possible to expand this functionality and build a [primitive] Cylc scheduler in Javascript. It would only have to emulate true/false (also maybe sleep X) type commands so is less complex than it sounds.

[oliver-sanders]

Quick note to add to the feature wish-list, a quick test to ensure that the syntax highlighting doesn't encounter any erroneous syntax. This is a cheep way to spot invalid code examples and to ensure the lexer stays up-to-date with language specification.

A quick grep of the HTML provides a hacky solution:

$ grep -l --color=none -r '<span class="err"' | sed 's/\.html/\.rst/' | sed 's/^/doc\/src\//'

[sadielbartholomew]

The core origin of this Issue, namely "convert [the User Guide] to sphinx" as per the title, has been fully addressed by #2910 which has now been merged. Follow-on work which was registered in comments here, & as noted during work towards #2910, has been moved into #2933 which supersedes (though links into the detail) contained here.

Therefore this can be closed.