🔄 Redefine schema in terms of @types/mdast

[agoose77]

This PR redefines the MyST schema in terms of types exposed by @types/mdast, and generates the MyST JSON schema from these types.

By making this change, we no longer need to vendor the CommonMark schema itself, and we can directly author the types that the rest of the MyST ecosystem consumes (rather than writing indirect JSON schema).

Tasks

• Re-implement YML tests in TypeScript
• Update docs
• Publish schema in CI

Schema

Warning

This PR changes our spec slightly, to rectify breaking changes with MDAST and to make better use of types. These changes are as follows:

• Table.align change from string | undefined to string[] | undefined | null (this comes from @types/mdast
• crossReference, mystRole, mystDirective change from Node to Literal & Parent (these attributes already were used, but not at the required-type level)

Docs

To build the docs, run npm run docs:prepare, and then myst start in the docs directory.

With this PR, it's now the case that the code (rather than any IDL definition or schema) is the reference spec for working with MyST's AST. Consequently, this PR reworks the docs to focus on people who want to reason about e.g. MyST JSON AST, such as people writing plugins in Python.¹

I chose to split each node into its own page:

Complex types (only arrays presently) are moved to footnotes:

This might be a controversial change, but I think on balance it does help readability.

Schema

The downside of this PR is that we have to use https://github.com/YousefED/typescript-json-schema, because of the way that mdast types are designed to be extended (via interface declaration merging). This mechanism is not handled by any AST-driven TS -> JSON Schema generators, so we need something that actually invokes the compiler (see above).

Footnotes

1. This is not solely a pragmatic take on who needs the docs -- it's also hard to preserve all syntactic type information when exporting the spec to a non TS format (see also Schema). As we lose types like Array<BlockContent | DefinitionContent> (which is simply expanded to an unnamed literal intersection), I opted to focus exclusively on the parts of the AST that end up in JSON, i.e. node types, rather than e.g. named array types. ↩

[agoose77]

I don't think this is correct w.r.t the CommonMark spec:

- one

two

produces this CommonMark AST:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE document SYSTEM "CommonMark.dtd">

<document xmlns="http://commonmark.org/xml/1.0">
  <list type="bullet" tight="true">
    <item>
      <paragraph>
        <text>one</text>
      </paragraph>
    </item>
  </list>
  <paragraph>
    <text>two</text>
  </paragraph>
</document>

[rowanc1]

Isn't this saying that we default to having a paragraph instead of text directly in the list item? Which is what is created?

[agoose77]

Our documentation states that

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE document SYSTEM "CommonMark.dtd">

<document xmlns="http://commonmark.org/xml/1.0">
  <list type="bullet" tight="true">
    <item>
      <text>one</text>
    </item>
  </list>
</document>

is permitted by the CommonMark spec. I don't see any evidence that that is the case. What I did notice is that the HTML renderer doesn't show the <p>, but I think that might be a rendering difference rather than a spec difference.

I noticed this because whilst the spec page shows <li>TEXT</li> in the inline example, the actual examples file does not. EDIT: oops, off-by-one error :P

[agoose77]

To add -- if my conclusion here is incorrect, then typing this proves quite hard because it is not easy/possible to extend an existing type and change the type of children in that manner.

[fwkoch]

Wow... yeah, wild that the html doesn't show the <p> despite it being in the AST! Oof, good catch.

[rowanc1]

Looks like a good direction. Excited to have proper typescript types.

Docs and a downstream PR in mystmd consuming these changes are the major things to see before merging/releasing.

[agoose77]

TODO: move to @types/mdast==4 in order to work with most other remark releases.

[fwkoch]

I think this looks great. It sure is nice to define actual typescript types - I think my original reasoning for JSON schema was (a) language agnostic (a bit too lofty a goal so early in the project, I suspect...) and (b) I wasn't super happy with any TS -> JSON options. Looks like you have one that works, though.

I'm approving this, but I agree with Rowan that we should probably consume it in mystmd before merging/releasing just to iron out any type oddities that may come up (I assume it will substantially improve things and allow us to basically scrub GenericNode/GenericParent from the codebase).