DOC: WIP: Convert markdown to rehype's hast at build time.

[tomprince]

Originally posted by @d0sboots in #1489 (comment)

Related to that was my thought that we should probably render the MD pages to html as part of building. There's a noticable pause in loading some of those docs because of the client-side rendering it's doing. That would eliminate the need for MathJax in the docs.

I was curious to see how hard it would be to do this. At first I attempted to convert directly to html at webpack time. However, that ended up with essentially unstyled output.

This is my second attempt, which converts markdown to hast (which is the HTML ast used by react-markdown) at webpack-time. Then, at runtime, it loads the JSON hast, and tricks react-markdown into using that directly, rather than parsing the markdown at runtime.

I'll likely leave this to somebody else to turn into something that can be merged.

[d0sboots]

One thing I'll note is that the speed issue largely depends on full preprocessing, since looking at the profiles showed that at least half the time was spent dealing with MathJax. (The slow pages are the ones with lots of formulas.)

[tomprince]

I added a second commit which uses hast-util-to-jsx-runtime directly, rather than tricking react-markdown into using the pregenerated hast. However, that library doesn't pass all the props that react-markdown@8 does. in particular, li, ul and code blocks don't have props that we use, so those have broken styling with that commit.

Note that the prop changes are also in react-markdown@9. See https://github.com/remarkjs/react-markdown/blob/main/changelog.md#remove-extra-props-passed-to-certain-components

[tomprince]

One thing I'll note is that the speed issue largely depends on full preprocessing, since looking at the profiles showed that at least half the time was spent dealing with MathJax. (The slow pages are the ones with lots of formulas.)

I've not done any profiling or benchmarking, but the mathjax processing is happening at webpack-time. The result hast-json of the SVG elements generated by mathjax.

[d0sboots]

One thing I'll note is that the speed issue largely depends on full preprocessing, since looking at the profiles showed that at least half the time was spent dealing with MathJax. (The slow pages are the ones with lots of formulas.)

I've not done any profiling or benchmarking, but the mathjax processing is happening at webpack-time. The result hast-json of the SVG elements generated by mathjax.

Oh, nice!

I'm still a bit confused about the fundamental issue, though: Why can't you go all the way down to HTML? I was envisioning we just embed them in an iframe or something.

[tomprince]

I'm still a bit confused about the fundamental issue, though: Why can't you go all the way down to HTML? I was envisioning we just embed them in an iframe or something.

There is nothing fundamental preventing us from compiling down to html. As I mentioned, I had a version that did that (and then displayed it inline using dangerouslySetInnerHTML). It would take some work to get styling correct, and also to handle links and the corrupted text hiding advanced concepts.

If somebody is looking to do that, you can add rehype-stringify (and maybe, if using an iframe, rehype-document) as additional plugins to remark-loader. There are also a bunch of other plugins that might be useful in adding the appropriate styles.

Mostly I went with the approach I did to get something working with the correct styling with minimal effort.