Generate documentation page with JSDoc

[eulertour]

Use JSDoc to generate a documentation page in svelte so that it can be added to the documentation page here. The goal is to have something similar to the one here. The webpage should:

• Have a menu on the left that links to all pages
• Contain 1-2 freeform pages that aren't generated from a template (placeholder text)
• Include a page for each class within
  • geometry.ts
  • animation.ts
  • diagram.ts
  • text.ts
• Each page should have
  • The name of the class
  • A summary taken from the docstring (placeholder text)
  • An enumeration of the class arguments and types

Don't worry about the content of the website or complex styling, the main goal is just to be able to generate the site with templated pages for the classes. I'll also want a walkthrough on the tools used to generate the website and how I can extend/customize it with new classes or default values.

[niktek]

Status update:

So the two options of JSDoc and Sveld for getting the docs stuff out quickly and easily are not going to work in this situation. JSdoc is only going to work on .js files or TS that has been transpiled like you would get in a Vite/esbuild pipeline and likewise Sveld runs as a scanner of Svelte components which is not really applicable in this pure TS environment.

So the obvious route forward is to look into TSDoc with it's API extractor https://api-extractor.com/pages/overview/intro/ (the doc model) and the typedoc project https://typedoc.org - both of which would give us a json summary of the classes/doc comments for displaying in the pages.

I've setup both of them and they both fail. They start failing on:

build/bundle.d.ts:523:86 - error TS2304: Cannot find name 'WidthSetupConfig'.

523     constructor(UserScene: Class<StudioScene>, canvasRef: HTMLCanvasElement, config: WidthSetupConfig | HeightSetupConfig | undefined);
                                                                                         ~~~~~~~~~~~~~~~~

build/bundle.d.ts:523:105 - error TS2304: Cannot find name 'HeightSetupConfig'.

523     constructor(UserScene: Class<StudioScene>, canvasRef: HTMLCanvasElement, config: WidthSetupConfig | HeightSetupConfig | undefined);

And then continues with a lot more errors. Basically the @ts-nocheck might stop things being annoying during dev, but the errors it is reporting are real and need to be solved if we want to use any of the TS tooling to assist us.

I don't know if this is a quick fix where fixing 1 or 2 references/types or something like that solves the bulk of the 27 issues reported in geometry.ts :

So for now I'm going to pause on looking any further into this until I get some feedback on the TS issues.

[niktek]

Hi,

Just pushed the feat/doc-generation branch as a status update.

You can run <package manager> api-extractor run which will generate /temp/studio.api.json
The next step will be to build out the genDocs script which uses api-extractor-model to summarise the generated .json file to create the output needed for the docsite and delete the intermediate files/folders that api-extractor has generated.

I had hoped that it would be able to do parsing with just a file or two, but because it is also doing full type checking, it has to consume all the referenced files, so it encountered some more TS issues. I've made some modifications to the source files and annotated them with TODO: to show where those changes are - as well as a hackish change on the bundle.d.ts imports that will get overwritten the next time the build script is run.

As another test I also added typedoc (<package manager> typedoc build/bundle.d.ts) to see if was a bit more lenient on the generation, but it also had issues and doesn't even get to the generated output stage.

These changes are purely to get the api extraction working so that I can use the outputted .json file. They are not the changes that would actually be needed for the project. When you get a chance, it would probably be best to bring a copy of /src/* from the main branch over the version in this branch and resolve the TS issues.

Cheers,
Nik

[ghostdevv]

Worth looking into:

• https://github.com/ghostdevv/extractinator
• or https://npmjs.com/sveltedoc-parser

[niktek]

Hey @ghostdevv !

Yeah that looks great for the next step (how it has zero stars is beyond me), but I think all the tools are kind of stuck till the TS issues are resolved.

~/code/studio.js bun extractinator extract src/ temp/
{
  dts_path: '/Users/nik/code/studio.js/.extractinator/dts-1703126283078/MeshLineGeometry.d.ts',
  path: '/Users/nik/code/studio.js/src/MeshLine/MeshLineGeometry.ts'
}
file:///Users/nik/code/studio.js/node_modules/extractinator/dist/cli.js:345
      throw new Error(`Unable to find dts path for "${input_path}"`);
            ^

Error: Unable to find dts path for "/Users/nik/code/studio.js/src/MeshLine/MeshLineGeometry.ts"
    at emit_dts (file:///Users/nik/code/studio.js/node_modules/extractinator/dist/cli.js:345:13)
    at async extractinator (file:///Users/nik/code/studio.js/node_modules/extractinator/dist/cli.js:362:15)
    at async file:///Users/nik/code/studio.js/node_modules/extractinator/dist/cli.js:419:27

Node.js v21.4.0
error: "extractinator" exited with code 1 (SIGHUP)

[eulertour]

I just pushed fixes for the TS errors and circular imports. api-extractor still seems to just fail outright:

> @eulertour/[email protected] apiExtract
> api-extractor run --local --verbose


api-extractor 7.39.0  - https://api-extractor.com/

Using configuration from ./config/api-extractor.json
Analysis will use the bundled TypeScript version 5.3.3

ERROR: Internal Error: Unable to analyze the export "SceneController" in
/Users/devinneal/GitHub/studio.js/build/bundle.d.ts
You have encountered a software defect. Please consider reporting the issue to the maintainers of this application.

And the gendocs.ts script seems to dislike something about the files in dist/, though I don't know how those were created/how to recreate them.

Still, it feels good for those stubborn TS errors to be gone from VsCode. Let me know if there's more I need to do to unblock you.

[niktek]

So I just want to preface this update, this is turning about to be quite a different approach than what we took with the Skeleton docs where we basically used a vite plugin to extract the docs and we were also extracting from svelte components. Having said that - this is still relevant to the stuff I'm working on and I'm happy to keep exploring this.

Thanks for getting the TS issues fixed! There is still something going on with the d.ts being generated into build/bundle.d.ts - the latest commit has the slight correction in it, but lets walk through what's happening.

Running bun run build to generate the bundle.d.ts file will show some errors on lines 787-788 where it is trying to export things unnecessarily because they have already been exported via the Scene namespace. Changing those lines to be:

export { Geometry, Animation, Text, Scene, THREE, Utils, Diagram, Constants, setCameraDimensions, setCanvasViewport };
//export type { StudioScene, AnimationRepresentation };

Will now allow api-extractor run to go ahead and do it's thing: the main one of interest being the DocModel (/temp/studio.api.json). I have temporarily left on it's d.ts rollup functionality which is responsible for the /dist/studio.d.ts file for comparisons sake - it is differently formatted, but it's hard to say whether it is for better or worse spec wise.

At this point I recommend clear ing the console to make it easier to see. We can now run bun genDocs which ingests the /temp/studio.api.json docmodel via @microsoft/api-extractor-model where I have got it displaying the full APIEntity hierarchy, types and nested members. To save us all from console spam hell, I have adjusted it to only show the inner data available to us for the first occurrence of each type that is found, so if you scroll towards the top you will see the json outputs interspersed.

At the top of the console you will see that I've added some dummy TSDoc annotations to the Animation constructor to see how they would come through. And whilst it comes through in the docComment field - none of the tags have been split out or presented in any useful manner.

@ghostdevv I noticed that Extractinator uses the @microsoft/tsdoc and @micrsoft/tsdoc-config packages directly, which I had avoided due to the probably misleading blurb from Microsoft. Are you able to shed any light on whether they get a better representation of the Docs themselves ?

I've also stepped back and just tried to find any other tools that are oriented towards this problem and haven't really turned up much. But I'm completely open to any suggestions of other routes to look into - but my inclination is that sticking close to the TSDoc packages makes the most sense.

Tomorrow i'll continue to explore the tsdoc packages as well and check out how Extractinator works too.

[eulertour]

I'm glad you were able to make some more progress! And that you got me to start taking those type errors more seriously 😅.

You both have more experience with this than I do, but I'm getting the feeling that this task is for some reason more difficult with this codebase than it is on average. Does it seem that way to you? And if so, do you have any idea what might be causing it?

When I started writing this I mostly just picked whatever typescript settings would get me up and running the fastest, so I don't know if it'd be better if I was to use "alwaysStrict": true or replace rollup with api-extractor for the bundling, or something like that.

[ghostdevv]

how it has zero stars is beyond me

Thanks! We only started making it recently

Yeah that looks great for the next step (how it has zero stars is beyond me), but I think all the tools are kind of stuck till the TS issues are resolved.

Could you try with version 0.3.0?

[niktek]

Running Extractinator on this repo seems to get pretty far now with the following output:

Along with the emits of the d.ts files in the .extractinator folder.