Build System: Implementation

Overview

The build scripts are designed to make module compilation faster and more flexible. The scripts are written in Typescript, compiled to Javscript and then executed using NodeJS.

The system is designed around asynchronous Javascript, making full use of Promises and the async/await syntax to maximize concurrency.

Command Execution

Parsing Command Line Arguments

The modules build system is intended to work seamlessly with other commands run from the command line via yarn. Scripts are defined under package.json:

"scripts": {
  "scripts": "node --no-warnings --max-old-space-size=8192 scripts/bin/index.js",
  "build": "yarn scripts build"
}

yarn scripts serves as a shorthand to run module build scripts. In the example above, yarn build is short for yarn scripts build. The actual command executed is yarn node --no-warnings --max-old-space-size=8192 scripts/bin/index.js build.

scripts/bin/index.js serves as the entry point script for the build system. Command line arguments are passed to the Commander module, which parses them and handles the execution of the appropriate command.

// scripts/bin/index.js
const parser = new Command()
  .addCommand(buildAllCommand)
  .addCommand(createCommand)
  .addCommand(getLintCommand())
  .addCommand(getPrebuildCommand())
  .addCommand(getTscCommand())
  .addCommand(watchCommand);

await parser.parseAsync();
process.exit();

Each Command object is usually returned using its respective get function. (This is necessary for testing with jest: Each Command object is designed to be used only once. Once its argument values have been populated, they cannot be modified, so each test requires a new instance of that command. For this reason, commands that aren't tested using jest don't have a get function and are used directly)

For more information, refer to the documentation for Commander.

Modularity

To abide by DRY and reuse as much code as possible, code that performs the compilation has been separated from code that interacts with the user. For example:

// scripts/src/docs/json.ts

// Responsible for actually outputting json files
// Does not print to console
export const buildJsons = async (project: ProjectReflection, { outDir, bundles }: BuildJsonOpts) => {
  await fs.mkdir(`${outDir}/jsons`, { recursive: true });
  if (bundles.length === 1) {
    // If only 1 bundle is provided, typedoc's output is different in structure
    // So this new parser is used instead.
    const [bundle] = bundles;
    const { elapsed, result } = await buildJson(bundle, project as any, outDir);
    return [['json', bundle, {
      ...result,
      elapsed,
    }] as UnreducedResult];
  }

  return Promise.all(
    bundles.map(async (bundle) => {
      const { elapsed, result } = await buildJson(bundle, project.getChildByName(bundle) as DeclarationReflection, outDir);
      return ['json', bundle, {
        ...result,
        elapsed,
      }] as UnreducedResult;
    }),
  );
};

// Responsible for creating the `build jsons` command
// Performs all the printing to console and user interaction
const getJsonCommand = () => createBuildCommand('jsons', false)
  .option('--tsc', 'Run tsc before building')
  .argument('[modules...]', 'Manually specify which modules to build jsons for', null)
  .action(async (modules: string[] | null, { manifest, srcDir, outDir, verbose, tsc }: Omit<BuildCommandInputs, 'modules' | 'tabs'>) => {
    const [bundles] = await Promise.all([
      retrieveBundles(manifest, modules),
      createOutDir(outDir),
    ]);

    if (bundles.length === 0) return;

    if (tsc) {
      const tscResult = await runTsc(srcDir, {
        bundles,
        tabs: [],
      });
      logTscResults(tscResult);
      if (tscResult.result.severity === 'error') process.exit(1);
    }

    const { elapsed: typedocTime, result: [, project] } = await initTypedoc({
      bundles,
      srcDir,
      verbose,
    });


    logTypedocTime(typedocTime);
    printList(chalk.magentaBright('Building jsons for the following modules:\n'), bundles);
    const jsonResults = await buildJsons(project, {
      bundles,
      outDir,
    });
    logResult(jsonResults, verbose);
    exitOnError(jsonResults);
  })
  .description('Build only jsons');

The buildJsons() function can then be reused by the build docs command:

// scripts/src/docs/index.ts
const getBuildDocsCommand = () => createBuildCommand('docs', true)
  .argument('[modules...]', 'Manually specify which modules to build documentation', null)
  .action(async (modules: string[] | null, { manifest, srcDir, outDir, verbose, tsc }: Omit<BuildCommandInputs, 'modules' | 'tabs'>) => {
    const [bundles] = await Promise.all([
      retrieveBundles(manifest, modules),
      createOutDir(outDir),
    ]);

    if (bundles.length === 0) return;

    if (tsc) {
      const tscResult = await runTsc(srcDir, {
        bundles,
        tabs: [],
      });
      logTscResults(tscResult);
      if (tscResult.result.severity === 'error') process.exit(1);
    }

    printList(`${chalk.cyanBright('Building HTML documentation and jsons for the following bundles:')}\n`, bundles);

    const { elapsed, result: [app, project] } = await initTypedoc({
      bundles,
      srcDir,
      verbose,
    });
    const [jsonResults, htmlResult] = await Promise.all([
      // Reused here!
      buildJsons(project, {
        outDir,
        bundles,
      }),
      buildHtml(app, project, {
        outDir,
        modulesSpecified: modules !== null,
      }),
    // app.generateJson(project, `${buildOpts.outDir}/docs.json`),
    ]);

    logTypedocTime(elapsed);
    if (!jsonResults && !htmlResult) return;

    logHtmlResult(htmlResult);
    logResult(jsonResults, verbose);
    exitOnError(jsonResults, htmlResult.result);
  })
  .description('Build only jsons and HTML documentation');

Note also that typedoc initialization is shared between buildJsons and buildHtml (the code for which is found in docUtils.ts).

A similar principle has been applied to the code that handles building tabs and bundles.

Common Script Options

The command system has been designed with flexibility in mind. There are various options to customize the behaviour of the scripts such as --srcDir and --outDir which allow for custom source and output directories.

Script Compilation

The directory structure for the scripts folder is as follows:

.
├── bin               // Compiled Javascript
└── src               // Typescript source code
    ├── build
    │   ├── docs      // Scripts for building documentation
    │   ├── modules   // Scripts for building bundles and tabs
    │   └── prebuild  // Scripts for linting and typechecking
    ├── templates     // Scripts for bundle and tab creation
    ├── index.ts      // Scripts execution entrypoint
    └── tsconfig.json // Separate tsconfig for script code

Running yarn build:scripts runs the following commands:

1. yarn test:scripts: Run jest to ensure that the scripts pass unit tests
2. yarn lint:scripts: Run eslint to make sure script source code is properly formatted
3. rimraf scripts/bin: Clean the scripts build directory
4. tsc --project scripts/src/tsconfig.json: Compile the script code to Javascript
5. copyfiles -f \"scripts/src/templates/templates/*\" scripts/bin/templates/templates": Copy the module template files

Folder Structure

Build Assets

There are two categories of build 'assets': modules (bundles and tabs) and documentations (jsons and html). The code for building each resides in its corresponding folder.

scripts/src
├── docs
│   ├── json.ts    // Code for building jsons only
│   ├── html.ts    // Code for building HTML only
│   └── index.ts   // Code for building both jsons and html
└── modules
    ├── bundle.ts  // Code for building bundles only
    ├── tab.ts     // Code for building tabs only
    └── index.ts   // Code for building both tabs and bundles

The Command definitions also reside in those files. For example, the build modules subcommand can be found inside modules/index.ts. Each folder also contains its on set of utility functions (in docUtils.ts and moduleUtils.ts respectively).

Prebuild Commands

The prebuild commands are commands that execute before any compilation takes place, namely linting (eslint) and type checking (tsc).

scripts/src
└── prebuild
    ├── lint.ts  // Code for running eslint
    ├── tsc.ts   // Code for running tsc
    └── index.ts // Code for running the prebuild command (both tsc and eslint)

Linting and type checking currently take a significant amount of time to execute, hence they are given as separate commands. Linting and type checking can be carried out while running the main build commands using the --lint and --tsc flags. Since most developers will be using IDEs that provide linting and type checking, by default, running a build command will not trigger either to save time.

Script Utilities

There are several helper functions that are reused throughout the build system:

• wrapWithTimer: (func: Function) => Promise<{ elapsed: number, result: ReturnType<typeof func> }>: Wraps a function (that returns a Promise) with calls to performance.now(). Returns a new function that calls the original and returns that result along the time taken for the original function to execute.

• exitOnError: To properly emulate behaviour of node commands, when an error occurs, a call to process.exit(1) to make sure that the process exits with a non zero error code. This allows tools like Github workflows to identify when an error has occurred and indicate that deployment is unsuccessful. This function is called at the end of the execution of each Command.

• retrieveBundles, retrieveTabs and retrieveBundlesAndTabs: Certain commands are allow the user to specify which modules to compile. These functions take the user input, compare them against the modules manifest, and then return the appropriate bundles and tabs for the system to compile.

• createBuildCommand: Convenience function for creating the build [something] commands.

Helper Types

Severity

type Severity = 'error' | 'warn' | 'success'

Used to represent the build result of each asset.