Anchor link checker

[Josh-Cena]

I sent mdn/content#22140 to fix a lot of anchor links. Here's the script I used:

import fs from "node:fs/promises";
import path from "node:path";
import { load } from "cheerio";

async function* getFiles(dir) {
  const dirents = await fs.readdir(dir, { withFileTypes: true });
  for (const dirent of dirents) {
    const res = path.join(dir, dirent.name);
    if (dirent.isDirectory()) {
      yield* getFiles(res);
    } else if (res.endsWith(".html")) {
      yield Promise.all([res, fs.readFile(res, "utf8")]);
    }
  }
}

const files = getFiles("build");
/** @type {Record<string, string[]>} */
const pathToIds = { __proto__: null };
/** @type {Record<string, string[]>} */
const pathToLinks = { __proto__: null };

for await (const [filename, content] of files) {
  const $ = load(content);
  pathToLinks[filename] = $("a")
    .map((i, el) => $(el).attr("href"))
    .get()
    .map(
      (url) =>
        new URL(
          url.startsWith("#")
            ? `${filename.replace(/build\/|\/index.html/g, "")}${url}`
            : url,
          "https://developer.mozilla.org"
        )
    )
    .filter(
      (url) =>
        url.hostname === "developer.mozilla.org" &&
        url.hash &&
        url.hash !== "#languages-switcher-button"
    );
  pathToIds[filename] = $("[id]")
    .map((i, el) => $(el).attr("id"))
    .get();
}

for (const [filename, urls] of Object.entries(pathToLinks)) {
  const failures = [];
  for (const url of urls) {
    const referencedPath = path.join(
      "build",
      url.pathname
        .replaceAll("*", "_star_")
        .replaceAll("::", "_doublecolon_")
        .replaceAll(":", "_colon_")
        .replaceAll("?", "_question_")
        .toLowerCase(),
      "index.html"
    );
    const ids = pathToIds[referencedPath];
    if (ids) {
      const target = ids.find((id) => id === decodeURI(url.hash.slice(1)));
      if (!target) {
        failures.push(
          `Anchor not found: ${url
            .toString()
            .replace("https://developer.mozilla.org", "")}`
        );
      }
    } else if (
      !["/", "/en-US/docs/MDN/Contribute/Feedback", "/docs/MDN/About"].includes(
        url.pathname
      )
    ) {
      failures.push(
        `Page not found: ${url
          .toString()
          .replace("https://developer.mozilla.org", "")}`
      );
    }
  }
  if (failures.length) {
    const sourceFile = path.resolve(
      filename
        .replace("build/en-us/docs", "files/en-us")
        .replace(".html", ".md")
    );
    console.log(`In ${sourceFile}:
${failures.map((e) => `  - ${e}`).join("\n")}
`);
  }
}

You can build a mostly self-contained part of the site, like:

yarn build files/en-us/web/javascript/**/*.md

and then run this script:

node analyze-html.mjs > analyze.log

(It's ESM, so use .mjs extension)

analyze.log would look like this:

In $PWD/files/en-us/web/javascript/guide/index.md:
  - Anchor not found: /en-US/docs/Web/JavaScript/Guide/Expressions_and_Operators#conditional_(ternary)_operator
  - Anchor not found: /en-US/docs/Web/JavaScript/Guide/Indexed_collections#array_object
  - Anchor not found: /en-US/docs/Web/JavaScript/Guide/Iterators_and_Generators#generators

In $PWD/files/en-us/web/javascript/guide/modules/index.md:
  - Page not found: /en-US/docs/Web/HTML/Element/script#attributes

In $PWD/files/en-us/web/javascript/guide/using_promises/index.md:
  - Page not found: /en-US/docs/Web/API/HTML_DOM_API/Microtask_guide/In_depth#tasks_vs_microtasks

And now is time to fix all the errors!

A caveat: it doesn't follow redirects. It's a feature, not a bug: we should update the redirected links to the new location anyway. Redirect links are also reported as a flaw by Yari.

[Josh-Cena]

I ran the script over the entire corpus and found 2664 broken anchors (with duplicates) and 116 broken links. Good luck!

[yin1999]

    const referencedPath = path.join(
      "build",
      url.pathname.toLowerCase().replace(":", "_colon_"),
      "index.html"
    );

the rule of slug to folder replacement is here:

https://github.com/mdn/yari/blob/ce67d66ecd57752801745e9e4d77944925a85c5e/libs/slug-utils/index.js#L3-L20

[Josh-Cena]

Thanks! Updated

[teoli2003]

This is great, and we should totally fix all broken anchors.

To maintain quality, I wonder if we should morph this into a CI test for each PR, checking the anchors on the pages touched by the PR:

[Josh-Cena]

That sounds like a good idea. I don't know how tricky it gets (because we need to build the page that the link points to in order to know if the anchor exists), but that would be ideal.

[Rumyra]

This is incredibly useful @Josh-Cena - thank you!