docs: Generate manual from markdown using mkdocs

[landam]

Goal

Build user manual from Markdown using mkdocs

How to test

1. Install requirements, see man/mkdocs/requirements.txt
2. Create file /tmp/code.lua:

function CodeBlock (cb)
    return pandoc.RawBlock('markdown', '```bash\n' .. cb.text .. '\n```\n')
end

3. Convert HTML files to Markdown:

for f in `find . -name *.html`; do echo $f; cat $f | sed 's#<div class="code"><pre>#<pre><code>#g' | sed 's#</pre></div>#</code></pre>#g' | pandoc --from=html --to=markdown -t gfm --lua-filter /tmp/code.lua > ${f%%.html}.md;done

4. Uncomment parts commented in 6743f1a
5. Recompile GRASS (make distclean; ./configure ...; make)
6. Build documentation from markdown files: (cd man; make build-mkdocs)
7. Open dist.x86_64-pc-linux-gnu/docs/mkdocs/site/index.html in web browser

Tasks

Critical:

• fix CI
• clean-up mkmarkdown.py
• man : integrate changes from main
• separate makefile rules (Markdown.make, MarkdownRules.make) ?
• make install
• fix gui/script/ compilation
• fix compilation issue in raster
• fix compilation issue in man
• fix segfault in --md-description
• fix code blocks
• fix commit link

Important:

• implement build_md.py and indices
• solve warnings generated by mkdocs
• add missing meta page comments to MD files (as YAML metadata block)
• consider to keep supporting HTML in g.extension for the addons
• parser_standard_options: choose formatting than table
• index: choose better formatting than table
• replace html with md in links
• remove toc-related code from mkmarkdown.py and change mkdocs style to show page-related TOC
• fix "## DESCRIPTION" appearance from keyword list
• logo points to index.html
• site_name to "GRASS GIS 8.5.0dev Reference Manual"
• fix menu items
• implement parser_standard_options
• implement manual_gallery
• use theme footer

Nice to have:

• implement class_graphical (https://grass.osgeo.org/grass84/manuals/class_graphical.html is empty ???)
• manual_gallery: different number of images (md vs html), fix layout (run in man dir: GISBASE="/home/martin/src/grass/dist.x86_64-pc-linux-gnu" ARCH="x86_64-pc-linux-gnu" ARCH_DISTDIR="/home/martin/src/grass/dist.x86_64-pc-linux-gnu" VERSION_NUMBER=8.5.0dev VERSION_DATE=2024 python3 ./build_manual_gallery.py) and see dist.x86_64-pc-linux-gnu/docs/markdown/site/manual_gallery.html and https://grass.osgeo.org/grass84/manuals/manual_gallery.html

Tasks before merge

• rebase main and regenerate MD files from HTML once again

Tasks after merge(?)

• remove rest-related scripts
• generate man from markdown
• remove HTML-based documentation and clean-up scripts

[neteler]

(using prettier on the HTML files generated from MD ( npx prettier --print-width 80 -w *.html) I have found some remove leftover HTML code in the MD part: fixed in ac5e963 and a1fc15e)

[neteler]

In total, 693 HTML files are generated in dist.x86_64-pc-linux-gnu/docs/mkdocs/site/.

Only 3 files now still contain errors (excellent result!):

i.zc.html
[error] i.zc.html: SyntaxError: Opening tag "meta" not terminated. (9:9)
[error]    7 |       <meta name="viewport" content="width=device-width,initial-scale=1">
[error]    8 |       
[error] >  9 |         <meta name="description" content="Zero-crossing "edge detection" raster function for image processing.">
[error]      |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[error]   10 |       
[error]   11 |       
[error]   12 |       

-> quotes in quotes issue

r.patch.html
[error] r.patch.html: SyntaxError: Opening tag "meta" not terminated. (9:9)
[error]    7 |       <meta name="viewport" content="width=device-width,initial-scale=1">
[error]    8 |       
[error] >  9 |         <meta name="description" content="Creates a composite raster map layer by using known category values from one (or more) map layer(s) to fill in areas of "no data" in another map layer.">
[error]      |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[error]   10 |       
[error]   11 |       
[error]   12 |       

-> also quotes in quotes issue

manual_gallery.html
[error] manual_gallery.html: SyntaxError: Unexpected closing tag "div". It may happen when the tag has already been closed by another tag. For more info see https://www.w3.org/TR/html5/syntax.html#closing-elements-that-have-implied-end-tags (631:4)
[error]   629 | </p>
[error]   630 |
[error] > 631 | <p></div>
[error]       |    ^^^^^^
[error]   632 | <script id="init-glightbox">const lightbox = GLightbox({"touchNavigation": true, "loop": false, "zoomable": true, "draggable": true, "openEffect": "zoom", "closeEffect": "zoom", "slideEffect": "slide"});
[error]   633 | document$.subscribe(() => { lightbox.reload() });
[error]   634 | </script></body>

-> not sure here where this comes from.

Also raster/r.li/r.li.html suffers from a missing </div>, probably the original HTML file contains a bug (I suppose the header stuff therein should be removed, as seen in other "meta page description" pages).

[neteler]

I have taken liberty to change the pandoc code in the HOWTO above to generate Github-Flavored Markdown -t gfm.

Example: the table in r.sun.md is now syntactically accepted and readable (just the width needs to be adjusted somehow):

Background of supported flavors:

• https://pandoc.org/MANUAL.html#extensions
• https://pandoc.org/MANUAL.html#markdown-variants

[landam]

I have taken liberty to change the pandoc code in the HOWTO above to generate Github-Flavored Markdown -t gfm.

Great! Please feel free to create new PR for HTML to MD conversion. This PR (changing build system to support mkdocs/MD) is ready for review.

[echoix]

Looks great to me! What will be the steps needed in regards to grass-addons after that change? Will they work out of the box?

[landam]

Looks great to me! What will be the steps needed in regards to grass-addons after that change? Will they work out of the box?

This PR is just an initial step. It adds support for markdown-based documentation in build system. HTML-based documentation is still default.

And yes, we have to convert also Addons documentation to markdown, similarly as done for GRASS core modules, see #4620

Steps:

1. Merge this PR
2. Finish & merge docs: script to convert HTML manual pages to markdown #4620
3. Convert Addons documentation to Markdown
4. Remove HTML-related rules from build system and HTML files

It would be also nice to remove RST-related code which was introduced in the past and never finished.