autogenerate docs

[efiop]

Seems like sphinx is the standard (Kudos @pmrowla for the suggestion), but dvc-task and dvc-render use mkdocs. Need to choose.

[efiop]

@daavoo Were there any particular reasons for choosing mkdocs over the other options?

[daavoo]

@daavoo Were there any particular reasons for choosing mkdocs over the other options?

Note my points are not really sphinx vs mkdocs but more sphinx vs (mkdocs + ecosystem) .

Mainly chose mkdocs because:

• Used by doc pages I like (FastAPI, Pydantic, httpx)
• Complexity of our dvc-X docs doesn't require 90% of sphinx features (and features is the most commonly used point in favor of sphinx).
• Markdown. (IMO) mkdocs has simpler syntax. Consistency with doc pages (although sphinx can be extended with a custom parser for markdown)
• Easier deploy to gh pages
• Love the default appearance and features of mkdocs-material / kinda hate most sphinx themes

[daavoo]

Also:

• First thing I tried and worked. Tbh I don't see much value of in-depth debate/research about docs tooling, at least given our current needs.

[shcheklein]

@casperdcl has introduced some tooling in the shtab and PyDrive2 recently and there was a discussion I believe around it. Unless there is a really strong reason, I would try to keep it unified if possible.

[casperdcl]

• shtab: GHA workflow + /docs/ subdir (using pydoc-markdown + custom preprocessor + mkdocs renderer + material theme + markdown_extensions) -> gh-pages branch + CNAME -> https://docs.iterative.ai/shtab/
  • same as what tqdm uses :)
• PyDrive2: sphinx was a pain, I only changed the theme to furo recently.

[skshetry]

This is a discussion to have in https://github.com/iterative/py-template. Right now, using markdown in docs is inconsistent with the rest of the cookie-cutter template. Also, we don't need to force docs in all the generated repositories, it should be conditional.

I usually just use sphinx, coupled with the furo theme, which is nice and used in lots of Python libraries. There's also myst-parser that allows you to use markdown with Python and sphinx (which I find to be useful when including readme contents in the docs but use sphinx's autogeneration features.

[casperdcl]

using markdown in docs is inconsistent with the rest of the cookie-cutter template

https://github.com/iterative/py-template/blob/main/%7B%7Bcookiecutter.project_name%7D%7D/docs/index.md looks like markdown to me?

[skshetry]

using markdown in docs is inconsistent with the rest of the cookie-cutter template

https://github.com/iterative/py-template/blob/main/%7B%7Bcookiecutter.project_name%7D%7D/docs/index.md looks like markdown to me?

template has README.rst, CONTRIBUTING.rst and CODE_OF_CONDUCT.rst.

[shcheklein]

So, let's try to make it a bit more constructive please? What are the pros and cons of each? I only see some good arguments from @daavoo (thanks, David 🙏 ), everything else so far is not very constructive.

From the top of my head. We mush use MD (so we'll have to introduce additional processing? custom things? - not sure how complicated it is)?

@casperdcl what was your experience with sphinx was a pain ? could you elaborate a bit?

@skshetry what are the cons of using mkdocs?

can we compare the implementation complexity of some examples that we like ?

[casperdcl]

my understanding:

• both sphinx & mkdocs are mature with nearly identical feature matrices, and we're not considering any other options. However, primary intended use is different:
  • sphinx is rst-first & config via a py file
  • mkdocs is md-first & config via a yml file
• IMO because md is more widely adopted outside the Python world and is less html-code-like (i.e. ironically more Pythonically zen), it's more actively developed.
  • I switched projects I maintain from sphinx to mkdocs due to nicer themes, extensions & less boilerplate code
  • Also note PyPI Topic :: Documentation :: Sphinx has 465 entries, while Topic :: Documentation has 2051 (implying circa 3x more non-sphinx Python docs projects).

Compare:

mkdocs

• plugins/extensions/themes
  • https://github.com/facelessuser/pymdown-extensions <- ~25 extensions
  • https://squidfunk.github.io/mkdocs-material <- theme + ~15 feature categories, Free + $10/month versions
  • https://github.com/mkdocs/mkdocs/wiki/MkDocs-Plugins <- ~150 projects
• complexity
  • https://github.com/iterative/shtab/blob/master/docs/pydoc-markdown.yml
  • inline extension directives (=== tab, !!! tip)
• example result
  • https://docs.iterative.ai/shtab/

sphinx

• plugins/extensions/themes
  • https://www.sphinx-doc.org/en/master/usage/extensions/index.html <- short builtin list
  • https://github.com/sphinx-contrib/ <- ~100 repos
  • https://github.com/yoloseem/awesome-sphinxdoc <- last update >1 year ago, ~65 entries
  • Framework :: Sphinx <- 118 projects
• complexity
  • https://github.com/iterative/PyDrive2/blob/master/docs/conf.py
  • inline rst directives (e.g. .. toctree:: :maxdepth: 2 <list of pages>)
• example result
  • https://docs.iterative.ai/PyDrive2/

[0x2b3bfa0]

(My personal preference¹ is for Sphinx with Furo — like @skshetry in #19 — plus Napoleon and Google-style documentation strings for the sake of quasiliterate programming)

Projects using Sphinx

• NumPy
• Python
• black²
• pandas
• requests

Projects using MkDocs

• FastAPI
• tqdm

Projects using other systems

• spaCy — does something weird

Footnotes

1. Shameless plug ↩

2. Written in Markdown instead of reStructuredText; doesn't have any API reference, just user guides ↩

[0x2b3bfa0]

Disclaimer: my real-world experience with MkDocs is close to zero. 🤷🏼‍♂️

While reStructuredText and Sphynx are “the canonical choice” and have lots of specific syntax to represent Python concepts, they're also a bit niche and overcomplicated. I wonder if MkDocs will end up being the “hypermodern” replacement for them.

I've rescued some links from an old direct message conversation with @casperdcl:

• https://github.com/NiklasRosenstein/pydoc-markdown
• https://github.com/mkdocstrings/mkdocstrings
• https://github.com/davidenunes/mkgendocs (https://davidenunes.com/mkgendocs)

[0x2b3bfa0]

36% of Python programmers use documentation tools. The most popular one is Sphinx.¹ #pythondevsurvey https://lp.jetbrains.com/python-developers-survey-2021

Footnotes

1. Sphinx (61%) 🥇 followed by MKDocs (22%) 🥈 ↩

[jorgeorpinel]

In case it's useful, we have a page that lists docs-related repos (and what they use) as of now:

https://www.notion.so/iterative/Docs-related-repositories-list-c23f24dca6154dc49026923ff67d20f4

[casperdcl]

I would prefer all docs-related repos have a topic documentation added to them to show up in https://github.com/iterative/?q=topic%3Adocumentation#org-repositories (similar to how we do for topic:example)

[jorgeorpinel]

Agree. But that doesn't give us details on what engines they use, unless we start including that metadoc in the README or CONTRIBUTE files of every repo.

[casperdcl]

give us details on what engines they use

oh is that the intention? Why?