docs: Generate manual from markdown using mkdocs #3849

landam · 2024-06-16T13:28:17Z

Goal

Build user manual from Markdown using mkdocs

How to test

Install requirements, see man/mkdocs/requirements.txt
Create file /tmp/code.lua:

function CodeBlock (cb)
    return pandoc.RawBlock('markdown', '```bash\n' .. cb.text .. '\n```\n')
end

Convert HTML files to Markdown: -> see docs: script to convert HTML manual pages to markdown #4620

for f in `find . -name *.html`; do echo $f; cat $f | sed 's#<div class="code"><pre>#<pre><code>#g' | sed 's#</pre></div>#</code></pre>#g' | pandoc --from=html --to=markdown -t gfm --lua-filter /tmp/code.lua > ${f%%.html}.md;done

Uncomment parts commented in 6743f1a and ab67825
Recompile GRASS (make distclean; ./configure ...; make)
Build documentation from markdown files: (cd man; make build-mkdocs)
Open dist.x86_64-pc-linux-gnu/docs/mkdocs/site/index.html in web browser

Tasks

Critical:

Important:

Nice to have:

implement class_graphical (https://grass.osgeo.org/grass84/manuals/class_graphical.html is empty ???)
manual_gallery: different number of images (md vs html), fix layout (run in man dir: GISBASE="/home/martin/src/grass/dist.x86_64-pc-linux-gnu" ARCH="x86_64-pc-linux-gnu" ARCH_DISTDIR="/home/martin/src/grass/dist.x86_64-pc-linux-gnu" VERSION_NUMBER=8.5.0dev VERSION_DATE=2024 python3 ./build_manual_gallery.py) and see dist.x86_64-pc-linux-gnu/docs/markdown/site/manual_gallery.html and https://grass.osgeo.org/grass84/manuals/manual_gallery.html

Tasks after merge(?)

remove rest-related scripts
generate man from markdown
remove HTML-based documentation and clean-up scripts

Co-authored-by: Edouard Choinière <[email protected]>

wenzeslaus · 2024-11-08T16:13:01Z

I get an error with !ENV:

... > make build-mkdocs

MkDocs encountered as error parsing the configuration file: could not determine a constructor for the tag '!ENV'
  in ".../dist.x86_64-pc-linux-gnu/docs/mkdocs/mkdocs.yml", line 2, column 12
make: *** [Makefile:247: build-mkdocs] Error 1

After deleting !ENV from the referenced file:

... > make build-mkdocs
ERROR   -  Config value: 'theme'. Error: Unrecognised theme name: 'material'. The available installed themes are: mkdocs, readthedocs 
ERROR   -  Config value: 'markdown_extensions'. Error: Failed loading extension "pymdownx.details". 
ERROR   -  Config value: 'plugins'. Error: The "glightbox" plugin is not installed 

Aborted with 3 Configuration Errors!
make: *** [Makefile:247: build-mkdocs] Error 1

landam · 2024-11-08T16:23:35Z

I get an error with !ENV:

Environmental variables are supported by mkdocs: https://www.mkdocs.org/user-guide/configuration/#environment-variables

@wenzeslaus Which version of mkdocs do you have?

landam · 2024-11-08T16:26:11Z

... > make build-mkdocs
ERROR - Config value: 'theme'. Error: Unrecognised theme name: 'material'. The available installed themes are: mkdocs, readthedocs
ERROR - Config value: 'markdown_extensions'. Error: Failed loading extension "pymdownx.details".
ERROR - Config value: 'plugins'. Error: The "glightbox" plugin is not installed

please install requirements: man/mkdocs/requirements.txt

See How To, step 1.

landam · 2024-11-08T16:29:10Z

@wenzeslaus Which version of mkdocs do you have?

Probably you have to install https://github.com/waylan/pyyaml-env-tag

landam · 2024-11-08T16:33:20Z

@wenzeslaus Which version of mkdocs do you have?

Probably you have to install https://github.com/waylan/pyyaml-env-tag

If so, we have to update requirements file accordingly. Let us now.

wenzeslaus · 2024-11-08T17:44:31Z

I used the requirements file. I have mkdocs 1.1.2. I also added pyyaml-env-tag and reinstalled. I will try to trash everything and start over, but I think I'm still the three mkdocs plugins.

wenzeslaus · 2024-11-08T20:14:58Z

Thanks. Starting from scratch with the pyyaml-env-tag package helped. The result (for core repo) looks great!

wenzeslaus

This creates a great minimal working product (assuming a conversion is done on the side). It does not break the current HTML doc build. while there is a lot of places to improve, we can work on them as PRs against main.

I have only the 2-3 comments about dependencies.

I would be also curious to see a list of special cases you encountered, i.e., what special pages we have. It is sort of hard to keep track. We have the standard options table, intro pages, family, topic and keyword lists, anything else?

man/mkdocs/requirements.txt

REQUIREMENTS.md

landam · 2024-11-10T19:32:37Z

Thanks. Starting from scratch with the pyyaml-env-tag package helped. The result (for core repo) looks great!

Ok, requirements file updated in d45c0b6

Co-authored-by: Vaclav Petras <[email protected]>

neteler · 2024-11-13T20:59:05Z

@landam Overall it looks good to me, thanks for your hard work!

One thing I don't understand yet: if we keep both HTML and MD in parallel for a short period, how to deal with the comment parts as seen in 6743f1a? Should this now be activated in the PR?

landam · 2024-11-14T13:33:08Z

One thing I don't understand yet: if we keep both HTML and MD in parallel for a short period, how to deal with the comment parts as seen in 6743f1a? Should this now be activated in the PR?

@neteler No, otherwise the GRASS compilation fails (and breaks CI) since there are no MD files in repo. It can be activated in #4620 or in follow-up PR which adds generated MD files to git.

neteler

Tested on Fedora 40, the result, i.e. the generated manual pages based on markdown look great!

wenzeslaus

Thanks for addressing the dependencies. I tested the previous version as discussed in my previous review.

neteler · 2024-11-21T23:07:06Z

It seems that this change was missing in this PR:

--- a/lib/gis/parser_rest_md.c
+++ b/lib/gis/parser_rest_md.c
@@ -517,7 +517,7 @@ void print_escaped_for_md_keywords(FILE *f, const char *str)
                 fputc(*s, f);
             }
         }
-        fprintf(f, ".html)");
+        fprintf(f, ".md)");
     }
     else { /* first and other than second keyword */
         if (st->n_keys > 0 && strcmp(st->module_info.keywords[0], str) == 0) {
@@ -532,13 +532,13 @@ void print_escaped_for_md_keywords(FILE *f, const char *str)
                     fputc(*s, f);
                 }
             }
-            fprintf(f, ".html)");
+            fprintf(f, ".md)");
         }
         else {
             /* keyword index */
             char *str_link;
             str_link = G_str_replace(str_s, " ", "%20");
-            fprintf(f, "[%s](keywords.html#%s)", str_s, str_link);
+            fprintf(f, "[%s](keywords.md#%s)", str_s, str_link);
             G_free(str_link);
         }
     }

This fixes

[...]
WARNING -  Doc file 'd.what.rast.md' contains a link 'display.html', but the target is not found among documentation files. Did you mean 'display.md'?
WARNING -  Doc file 'd.what.rast.md' contains a link 'topic_vector.html', but the target is not found among documentation files. Did you mean 'topic_vector.md'?
WARNING -  Doc file 'd.what.vect.md' contains a link 'display.html', but the target is not found among documentation files. Did you mean 'display.md'?
WARNING -  Doc file 'd.what.vect.md' contains a link 'topic_vector.html', but the target is not found among documentation files. Did you mean 'topic_vector.md'?
WARNING -  Doc file 'd.where.md' contains a link 'display.html', but the target is not found among documentation files. Did you mean 'display.md'?
WARNING -  Doc file 'd.where.md' contains a link 'topic_sampling.html', but the target is not found among documentation files. Did you mean 'topic_sampling.md'?

[...]
WARNING -  Doc file 'g.copy.md' contains a link 'general.html', but the target is not found among documentation files. Did you mean 'general.md'?
WARNING -  Doc file 'g.copy.md' contains a link 'topic_map_management.html', but the target is not found among documentation files.
WARNING -  Doc file 'g.copy.md' contains a link 'keywords.html#copy', but the target 'keywords.html' is not found among documentation files. Did you mean
           'keywords.md#copy'?
WARNING -  Doc file 'g.dirseps.md' contains a link 'general.html', but the target is not found among documentation files. Did you mean 'general.md'?
[...]

I am not fully sure if this change is correct - it helps to get rid of above error messages but still dist.x86_64-pc-linux-gnu//docs/mkdocs/site/sql.html contains .md suffixes (e.g. dist.x86_64-pc-linux-gnu/docs/mkdocs/site/grass-sqlite.md) which should be .html.

So I kindly ask @landam and others for support.

landam · 2024-11-22T14:50:14Z

So I kindly ask @landam and others for support.

For record: #4740

The initial commit simply uncoments the Markdown compilation lines from OSGeo#3849. This requires the actual Markdown files to be present, so this breaks the CI. This will stay like this for now. The PR is mean to be tested together with the Markdown files generated by script in OSGeo#4620 (switch to this branch after generating the files). Subsequent commits will add fixes of the compilation (missing tools, files, etc.).

The initial commit simply uncomments the Markdown compilation lines from #3849. (This requires the actual Markdown files to be present which are now part of the source code.) Subsequent commits add fixes of the compilation (missing tools, files, etc.): * Add compilation of multi-program directories like r.colors, r.mapcalc, and r3.flow * Generate colortable thumbnails. This makes the script little more general and assumes that HTML will be eventually removed (the thumbnails are generated twice now, once for HTML and once of Markdown). * Copy barscale and northarrow files for Markdown doc source. * Presence of spaces vs underscores in key may differ between HTML and Markdown, so always do replacements. * Do not add dash to visible text for keyword. Additionally, remove underscore just like in the HTML version. * Add projectionintro and grass_database pages from doc dir. * Add compilation for GUI, icons, and additional non-Markdown files. * Add basic port of graphical index (removes broken links and provides some content). * Strip keywords from whitespace and skip empty ones The complex r.mapcalc build would fail on the first run attempt presumably because of a raise condition with color table build. This removes it from standard SUBDIR and relies on the extra pre-builld of r.mapcalc dir. However, this fails to do distclean properly and needs to be addressed later. g.extension now creates md from html on the fly if no md is present. The build of Markdown and HTML doc creates a race condition while getting the metadata though git clone, so the build now checks for presence of grass-addons clone and uses the directory as a cache after a successful clone. The code assumes all directory move errors are only the good case of a cache being created in the meantime. The code also creates two (or more) clones if multiple processes run together (git clone runs twice, but the clash in the cache is avoided). --------- Co-authored-by: Anna Petrasova <[email protected]>

landam added 2 commits June 16, 2024 15:09

very first draft

a01ab38

re-enable html builds

307432f

landam marked this pull request as draft June 16, 2024 13:28

landam self-assigned this Jun 16, 2024

landam added the docs label Jun 16, 2024

landam added this to the 8.5.0 milestone Jun 16, 2024

landam added 2 commits June 16, 2024 15:39

rename "md" dir to "markdown"

5aa3564

add converted md file from html using pandoc

d33bcb9

landam added 7 commits June 16, 2024 16:49

fix various compilation issues

e2fd51c

run mkdocs

78cbd32

fix md newline

40b6de9

short commit hash

6d49dae

--md-description: skip main header

f0c5311

avoid duplicated header for now

f725352

move mkdocs from doc/ to man/

bbbeee2

micro white space fix

92898df

Co-authored-by: Edouard Choinière <[email protected]>

echoix previously approved these changes Nov 7, 2024

View reviewed changes

wenzeslaus requested changes Nov 8, 2024

View reviewed changes

man/mkdocs/requirements.txt Outdated Show resolved Hide resolved

REQUIREMENTS.md Outdated Show resolved Hide resolved

add pyyaml-env-tag to man/mkdocs/requirements.txt

d45c0b6

landam dismissed echoix’s stale review via d45c0b6 November 10, 2024 19:32

landam and others added 3 commits November 10, 2024 20:33

Update man/mkdocs/requirements.txt

11011e2

Co-authored-by: Vaclav Petras <[email protected]>

remove duplicated pyyaml-env-tag from man/mkdocs/requirements.txt

ad6170a

Update REQUIREMENTS.md

9be5d69

Co-authored-by: Vaclav Petras <[email protected]>

landam requested a review from wenzeslaus November 10, 2024 19:36

Merge branch 'main' into markdown_docs

f59b0db

neteler approved these changes Nov 14, 2024

View reviewed changes

wenzeslaus approved these changes Nov 14, 2024

View reviewed changes

landam merged commit 346c8a2 into OSGeo:main Nov 14, 2024
27 checks passed

landam deleted the markdown_docs branch November 14, 2024 15:58

neteler mentioned this pull request Nov 23, 2024

[Bug] docs: remaining issues with markdown manual using mkdocs #4748

Closed

16 tasks

wenzeslaus mentioned this pull request Feb 4, 2025

doc: Enable Markdown doc compilation #5048

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: Generate manual from markdown using mkdocs #3849

docs: Generate manual from markdown using mkdocs #3849

landam commented Jun 16, 2024 •

edited by neteler

Loading

wenzeslaus commented Nov 8, 2024

landam commented Nov 8, 2024

landam commented Nov 8, 2024 •

edited

Loading

landam commented Nov 8, 2024

landam commented Nov 8, 2024

wenzeslaus commented Nov 8, 2024

wenzeslaus commented Nov 8, 2024

wenzeslaus left a comment

landam commented Nov 10, 2024

neteler commented Nov 13, 2024

landam commented Nov 14, 2024

neteler left a comment

wenzeslaus left a comment

neteler commented Nov 21, 2024 •

edited

Loading

landam commented Nov 22, 2024

docs: Generate manual from markdown using mkdocs #3849

docs: Generate manual from markdown using mkdocs #3849

Conversation

landam commented Jun 16, 2024 • edited by neteler Loading

Goal

How to test

Tasks

Tasks after merge(?)

wenzeslaus commented Nov 8, 2024

landam commented Nov 8, 2024

landam commented Nov 8, 2024 • edited Loading

landam commented Nov 8, 2024

landam commented Nov 8, 2024

wenzeslaus commented Nov 8, 2024

wenzeslaus commented Nov 8, 2024

wenzeslaus left a comment

Choose a reason for hiding this comment

landam commented Nov 10, 2024

neteler commented Nov 13, 2024

landam commented Nov 14, 2024

neteler left a comment

Choose a reason for hiding this comment

wenzeslaus left a comment

Choose a reason for hiding this comment

neteler commented Nov 21, 2024 • edited Loading

landam commented Nov 22, 2024

landam commented Jun 16, 2024 •

edited by neteler

Loading

landam commented Nov 8, 2024 •

edited

Loading

neteler commented Nov 21, 2024 •

edited

Loading