Bug: Out-of-Memory error when parsing many snippets with `compileSnippets`

[SgtPooki]

When running compileSnippets on markdown files with a lot of typescript code blocks (e.g. in https://github.com/ipfs/helia-verified-fetch documentation), typescript-docs-verifier loads and compiles all TypeScript code blocks concurrently. This causes high memory pressure and can lead to the Node.js process running out of memory.

We experienced this issue directly when running aegir's document-check against documentation with many inline TypeScript blocks. See ipfs/aegir#1816 for a related report. aegir uses compileSnippets at https://github.com/ipfs/aegir/blob/3ee2f685d643442fbea4fa206235a4bf8c2734a4/src/document-check.js#L70

I've got this fixed locally with a patch (see proof at ipfs/aegir#1818) and will submit a PR with changes to typescript (, but the JS changes are below:

diff --git a/node_modules/typescript-docs-verifier/dist/src/SnippetCompiler.js b/node_modules/typescript-docs-verifier/dist/src/SnippetCompiler.js
index 83d2759..545da13 100644
--- a/node_modules/typescript-docs-verifier/dist/src/SnippetCompiler.js
+++ b/node_modules/typescript-docs-verifier/dist/src/SnippetCompiler.js
@@ -39,10 +39,10 @@ class SnippetCompiler {
         this.workingDirectory = workingDirectory;
         this.packageDefinition = packageDefinition;
         const configOptions = SnippetCompiler.loadTypeScriptConfig(packageDefinition.packageRoot, project);
-        this.compilerConfig = {
+        this.service = TSNode.create({
             ...configOptions.config,
             transpileOnly: false,
-        };
+        });
     }
     static loadTypeScriptConfig(packageRoot, project) {
         var _a;
@@ -57,16 +57,56 @@ class SnippetCompiler {
     static escapeRegExp(rawString) {
         return rawString.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
     }
-    async compileSnippets(documentationFiles) {
-        try {
-            await this.cleanWorkingDirectory();
-            await fsExtra.ensureDir(this.workingDirectory);
-            const examples = await this.extractAllCodeBlocks(documentationFiles);
-            return await Promise.all(examples.map(async (example) => await this.testCodeCompilation(example)));
+    async compileSnippets(documentationFiles, { concurrency = 4 } = {}) {
+      try {
+        await this.cleanWorkingDirectory();
+        await fsExtra.ensureDir(this.workingDirectory);
+
+        const results = [];
+        const tasks = [];
+        let active = 0;
+
+        const importSubstituter = new LocalImportSubstituter_1.LocalImportSubstituter(this.packageDefinition);
+
+        for (const file of documentationFiles) {
+          const blocks = await this.extractFileCodeBlocks(file, importSubstituter);
+          tasks.push(...blocks)
+        }
+
+        const self = this;
+        let hasFailed = false;
+
+        async function runNext() {
+          if (hasFailed || tasks.length === 0) return;
+
+          const block = tasks.shift();
+          active++;
+
+          try {
+            const result = await self.testCodeCompilation(block);
+            results.push(result);
+
+            if (result.error) {
+              hasFailed = true;
+            } else {
+              await runNext();
+            }
+          } finally {
+            active--;
+          }
         }
-        finally {
-            await this.cleanWorkingDirectory();
+
+        const workers = [];
+        for (let j = 0; j < Math.min(concurrency, tasks.length); j++) {
+          workers.push(runNext());
         }
+
+        await Promise.all(workers);
+
+        return results;
+      } finally {
+        await this.cleanWorkingDirectory();
+      }
     }
     async cleanWorkingDirectory() {
         return await fsExtra.remove(this.workingDirectory);
@@ -96,8 +136,7 @@ class SnippetCompiler {
         const id = process.hrtime.bigint().toString();
         const codeFile = path.join(this.workingDirectory, `block-${id}.${type}`);
         await fsExtra.writeFile(codeFile, code);
-        const compiler = TSNode.create(this.compilerConfig);
-        compiler.compile(code, codeFile);
+        this.service.compile(code, codeFile);
     }
     removeTemporaryFilePaths(message, example) {
         const escapedCompiledDocsFolder = SnippetCompiler.escapeRegExp(path.basename(this.workingDirectory));

---

This patch resolves the issue by:

• Replacing unbounded Promise.all() parallelism with a concurrency-limited task queue (default: 4).

• Reusing a single ts-node service instance across all snippet compilations to avoid repeated setup cost.

• Failing fast when a snippet compilation returns an error.

• No external dependencies are introduced, and API behavior remains unchanged. However, an optional options object is added as a parameter to compileSnippets with default concurrency set to 4.

[SgtPooki]

it looks like #30 and #33 mean we should not be using a single TSNode service...

[SgtPooki]

ok I was going to try to get a typescript version fixed, but pulling down your repo, and making no changes, results in a bunch of Property 'should' does not exist on type 'Promise<SnippetCompilationResult[]>'. errors in test/TypeScriptDocsVerifierSpec.ts

I did modify that file so i could add a test, and even only 50 snippets results in OOM err… kind of wild

[paulbrimicombe]

Thanks for raising this @SgtPooki , I'll take a look. The chai problem is down to a breaking change in the types that didn't come with a major semver bump (DefinitelyTyped/DefinitelyTyped#72101) — versioning for types is weird because it kind of has to match the library that the types are for, but that means that you can accidentally bring in breaking changes.

[paulbrimicombe]

I did modify that file so i could add a test, and even only 50 snippets results in OOM err… kind of wild

I've had a quick look at this and ts-node seems pretty wild in its memory usage — something like this gets an OOM error pretty fast:

while (i < 1_000) {
  i += 1;
  const compiler = TSNode.create(config);
  compiler.compile("const something = true;", "index.ts");
  console.log(i);
}

I'll have a look at your PR — I suspect that re-using the TSNode Server is the main thing to fix (or even switching to a different compiler that's less hungry but that might be a bigger change).

EDIT: If you don't make the compile() call, there are no memory issues at all.

[paulbrimicombe]

@SgtPooki , thanks so much for raising this. I've found some time today to completely pull ts-node out of the codebase. I've published 3.0.0-beta.2 — can you try that out please (you can use @next when you install it) and see if it solves at least some of the problems? I've tried it on your codebase and it seems to work fine.

Thanks for #36 — I'm really reluctant to use the same ts-node instance across multiple compilations because of being burned in the past by snippets not compiling independently but I'll look at the other changes / suggestions.

Now that we're not writing snippets to disk and given that all of the internal file operations during TypeScript compilation seem to be synchronous, I suspect that compiling the snippets in parallel in a single thread isn't going to help at all so I've gone completely sequential.

There are some breaking changes from dropping ts-node etc. (hence the major version bump):

• Minimum Node.js version is now 20
• Minimum TypeScript version is now 4.7.2
• The errors included in the compilation results are no longer TSNode.TSError instances but the CompilationError class has basically the same API so unless you're using instanceof, it shouldn't be a problem.

Let me know how you get on!

[SgtPooki]

I'm prepping for eth.cc and a little busy with my own stuff but ill try to give this a shot when i get a chance