DRY versatile cmdline doc generation (manpage, -h msg, html etc) via a DSL

[timotheecour]

/cc @a-mr
our docs generally look good, but cmdline docs are bad:

not DRY

these are manually (IIRC) maintained and generated:

• tools/nim.zsh-completion
• tools/nim.bash-completion
• doc/nimgrep.rst vs tools/nimgrep.nim repeats cmdline flags (and they're out of sync)

nim-lang#17415 attempts to make nimgrep cmdline docs DRY (which is good) but at cost of introducing artifacts (as noted in PR comments)

ugly/hard to read

compare https://nim-lang.github.io/Nim/nimc.html vs docs for other projects, see below

good looking command line reference docs

• D: https://dlang.org/dmd-osx.html
• clang: https://clang.llvm.org/docs/ClangCommandLineReference.html
• python: https://docs.python.org/3/using/cmdline.html

proposal

provide tooling allowing to write cmdline docs that are:

• DRY (nothing is ever repeated)
• decent looking or even great-looking
• can be rendered in several modes:
  • as --help msg in cmdline applications (eg nim --fullhelp, nimgrep -h, etc)
  • in rendered html docs
  • can generate bash/zsh completion scripts (IIRC there are tools we can invoke based on -h output)
  • can generate man pages (ditto: IIRC there are tools we can invoke based on -h output)
• allow for easy tooling, eg we should be able to use doc links pointing to where a flag is defined, eg:

# in foo.nim
proc bar*() =
  ## see cmdline.dynlibOverrideAll_

implementation

generate the doc string in pure nim using a DSL (similar to karax https://github.com/pragmagic/karax but much simpler), leveraging full power of nim to handle things like inserting variable names, dates, auto-generated content, etc

the DSL logic would be in some module std/cmdlineutils

then CI auto-generates the various artifacts for each rendering target:

• html rendered docs
• bash/zsh completion
• man pages
• doc string for --help msg

example std/cmdlineutils DSL usage

# in nimgrep.nim:

import std/cmdlineutils

proc help(): auto = 
  title("nimgrep")
  section("Represent results")
  flag("beforeContext", "b"):
    "print N lines of leading context before every match"
  flag("group", "g"):
    "group matches by file"

  flag(long = "group", short = "g", txt = "group matches by file") # also supported
  ...

now we can use help in different rendering contexts:

• for cmdline help, use help().toCmdline
• for html rendering of cmdline, use help().toHtml
• for bash/zsh/manpages, use help().toManpages etc
• the cmdline application can use an enum generated from help to handle cmdline input and look like this:

case handle(help)
of Flag.beforeContext: ...
of Flag.group: ...

links

• see man nim : No manual entry for nim nim-lang/Nim#7401

The official Debian package provides manpages for nim, nimble, nimgrep, nimsuggest
They are autogenerated from the help commands. You can reuse them in "brew" and other.

indeed, for autocompletion and man page generation this works: c-blake/cligen#23 (comment)
just a matter of integrating these in nim distributions (incl homebrew formula)

• nim: generate man page Homebrew/homebrew-core#60518

system "help2man", "bin/nim", "-o", "nim.1", "-N"
man1.install "nim.1"

[beef331]

I've started on a solution which I think addresses a fair bit of these issues. Mostly just posting here for posterity, and at least a look at a possible DSL solution.

[timotheecour]

thanks, looks like a good start, I'll comment in your repo later

[a-mr]

You did not specify what exactly is so ugly about our cmdline docs.

I see only one defect: that descriptions for long options can be confused together because they merge visually. We should solve it at the CSS style level IMHO.

I don't understand what you see in "good looking examples". They all waste too much of page space for gaps between options. Summary:

• D — just on par with Nim
• clang — actually it's worse
• Python — somewhat better, it certainly has its bells and whistles

[timotheecour]

the 2 column layout we have is ugly and wastes horizontal space, leaving very little room for the actual explanation; note that all other examples i gave instead an interleaved layout:

• each cmdline flag on it's own line
• the description goes below, eg:

[timotheecour]

A DSL would be the most flexible option, allowing to render differently for for rst, html cmdline help output or even integrating the command line processing logic, etc, as well as being reusable for other cmdline tools (nimgrep, nimsuggest, nimfind etc) as well as 3rd party packages (nimble etc). Note that writing directly the cmdline to RST is less flexible (conditional generation per target, string processing)

examples where our 2 column layout is bad

[a-mr]

examples where our 2 column layout is bad

The first example is just a syntax error and it's already been fixed in nim-lang@8db93fd

Second example can be cured by inserting HTML <wbr> tags before every |. I inserted them manually into html file to check how it will look like. Here it is:

[timotheecour]

it's already been fixed in

yes, good, i saw that.

Second example can be cured

even then i still don't like the layout, it feels cramped, wastes space that could be used to explain our cmdline better without a ton of wasted space, both in the html rendering and in the cmdline output of nim --fullhelp.

With a very simple modification to match to the style i mentioned earlier (that is followed by all others i mentioned, python clang, D, etc), i already improve the results:

rst after reflowing away from 2 columns: https://gist.github.com/timotheecour/3fcc84cde4481e7a7849c937e0f832c3

html output after this modification:

(it also looks good when dumped via --fullhelp on cmdline)

it could be improved further by ensuring each flag appears in bold; but this should be done automatically and the rst source with flags+description shouldn't have to mentioning styling (hence the DSL i suggested, but happy to discuss other options that improve styling)

current html output from https://nim-lang.github.io/Nim/nimc.html:

note

note that the rst renders good in github (https://gist.github.com/timotheecour/3fcc84cde4481e7a7849c937e0f832c3) with each option in bold.

if github does it, can nim rst2html do it too?