-
Notifications
You must be signed in to change notification settings - Fork 76
monorepo
Calcite development occurs in a monorepo to simplify the continuous integration and issue management workflows. Calcite uses turbo to manage packages, lerna for versioning and next releases, and release-please for latest releases.
The monorepo uses NPM workspaces. You can run NPM scripts for specific packages from anywhere in the repo using the --workspace flag. Workspace names are specified in the "name" field in package.json. For example, to watch Calcite Component tests from the repo's root directory, run:
npm run --workspace=@esri/calcite-components test:watchIf your PWD is within a package's directory, e.g. packages/calcite-components, you can run NPM scripts for that package without the --workspace flag.
However, make sure to always use the workspace flag when installing dependencies. This may not be necessary, but dependency hell is a scary place so better safe than sorry. Read the #dependencies section for more info.
Many of the NPM scripts in the root package.json run turbo pipelines. See the #task-execution section for more info.
NOTE: In order to take advantage of Turbo's task execution, Calcite Component's
testNPM script no longer has a baked inbuild. However, abuildwas added before thetest:watchscript, which is more commonly used by developers.
Monorepos are difficult to manage manually, especially at scale. Management tools help coordinate complex local dependency graphs so everything runs smoothly. Monorepo tools can also provide build caching and package versioning functionality.
Turbo is a package-based monorepo tool that simplifies workspace dependency management and improves build and task execution using caching, multitasking, and other strategies. Here is a breakdown of the difference between package-based and integrated monorepos by Turbo's competitor, nx. Checkout Turbo's monorepo handbook to learn more.
Lerna is the O.G. monorepo management tool for Node, and is now maintained by the creators of Nx. Nx provides the build caching and task execution, while Lerna helps with versioning and publishing packages. Nx and Lerna are often used side-by side. Calcite uses Lerna as a dependency of release-please's node-workspace plugin and for #prereleases. However, Calcite uses Turbo instead of Nx for the caching and task execution.
If Turbo adds the ability to generate a dependency graph between workspaces in Node, a custom release-please plugin can be created to replace Lerna. This looks like Turbo's only Node API for now. release-please would also need to add better support for #prereleases to fully remove Lerna from the repo.
- nx: https://github.com/nrwl/nx
- changesets: https://github.com/changesets/changesets
- auto: https://github.com/intuit/auto
All packages published to NPM are in the packages directory. Once added, unpublished packages will live in a different directories, such as documentation.
Turbo Repo pipelines, such as build/test/lint, are specified in turbo.json and have their outputs cached to improve performance. Turbo uses a hash to determine if a package changed since storing the cached output. By default, Turbo will rerun the task when any file in a package changes. Glob patterns for files that should cause a cache miss when changed are specified in a task's "inputs" field in turbo.json. Calcite has not set inputs yet to prevent caching issues. If things run smoothly for a while inputs can be specified to improve performance.
Turbo supports remote caching which can improve performance in GitHub Actions and other CI that runs in containers or has an ephemeral filesystem. Calcite is not using remote caching right now since it appears to be a paid service. There is internal documentation for self hosted remote caching, which may be implemented for Calcite in the future.
Make sure to explicitly specify the workspace if you are installing a dependency to a non-root package.json. Workspace names are specified in the "name" field in package.json files. For example, to install @stencil/core to packages/calcite-components, run:
npm install --workspace=@esri/calcite-components @stencil/coreUse the following rules to determine where the dependency should be installed:
-
Install devDependencies in the root
package.json, except when installing packages local to the monorepo. For example, if@esri/calcite-componentsuses@esri/calcite-design-tokensas a devDependency, it should be installed inpackages/calcite-components/package.json. Local dependencies need to be in the package's workspace so task execution order and versioning work correctly. -
Install dependencies to the package's workspace, remembering to use the
--workspaceflag.
There can be exceptions to the rule when build issues arise. For example, there were build errors because two Stencil versions were being installed. @stencil/core is a dependency of @esri/calcite-components so it was installed in the workspace's package.json. However, some Stencil devDependencies (which were installed in the root) had @stencil/core as a dependency too, which resulted in multiple installations. The issue was resolved by moving the Stencil devDeps to the @esri/calcite-components workspace's package.json.
A new package should not be added to the monorepo if the new package has deployable changes. This ensures the changelog for the package's next release isn't missing entries that were committed in the previous repo. If the package has deployable changes, create a release before continuing.
Make the following changes and submit a PR:
- Move devDependencies to the root directory (besides local packages). You will need to regenerate the
package-lock.jsonif you copy and paste frompackage.jsonfiles. - Move GitHub Actions, git hooks, or other CI to the root directory, if applicable. GitHub Actions should follow the the naming convention of
what-it-does_scope.yml, e.g.pr-tests_eslint-plugin-calcite-components.yml - Add the path to the package and its current version to
.release-please-manifest.json. - In
release-please-config.jsonunder thepackagesfield, add the new package's path as well as any package-specific configurations. The only required field is the package's name, taken from thenamefield in itspackage.json. - If the new package needs to be linked to Calcite Component's version, add its name to the
LINKED_VERSIONS_TRACKING_PACKAGESarray insupport/syncLinkedPackageVersions.ts. - Potentially rename the new package's NPM scripts so they match the pipeline names in
turbo.json(build, test, clean, etc.). Note: having all of the NPM scripts that are specified inturbo.jsonis not required. - If present and when possible, the
testNPM script should not build first. Turbo will make sure thebuildscript runs first and will cache the results. - Potentially rename directories for consistency with the other packages:
-
src/- source code -
dist/- directory created when building -
support/- package-specific scripts such as build patches
- If there is an existing changelog, make sure its heading matches the changelogs of the other packages in the monorepo.
Make the following changes once the PR is installed, which will ensure the release CI generates changelog entries starting at the correct commit:
-
Create a #git-tags on the commit that adds the package.
git tag -a "<package_name>@<current_version>" "<git_sha>" -m "<package_name>@<current_version>" npm run util:push-tags
-
Create a #github-releases using the git tag you created. The title should be the same as the git tag. The body can say: "Move package to the Calcite Design System monorepo"
Calcite uses release-please, which creates release PRs that are automatically updated after pushes to the target branch using a GitHub Action. release-please takes care of changelog generation, GitHub releases, git tags, and package versioning. release-please also has plugins and file updaters. Calcite uses the generic file updater to bump the Calcite Components CDN link version in the package's readme. A manifest file in the root directory configures release-please, as well as JSON file with the current versions of the monorepo's packages. The version JSON file is automatically updated in the release PR, along with the versions in package.json files.
NOTE:
release-please's config options are out of date in their manifest config file documentation. Check the JSON schema for up to date options instead. The options and their values will autocomplete in VS Code when editing the config file in Calcite's repo. You can also install VS Code's JSON language server if your editor has an LSP client (ask Ben for help).
Calcite has a slightly more complex versioning pattern than other monorepos. Commonly, monorepos either maintain all the packages under the same version, or all as different versions. When versioning separately, packages bump a patch version when a dependency updates, even if a major update for the dependency occurred. For example, if [email protected] depends on [email protected] and there is a major design-tokens release to v3x. calcite-components will be bumped to v1.3.3 and released with the new design-tokens. This makes sense for most packages with the assumption that the consuming devs resolve breaking changes so they won't affect downstream users.
calcite-components-react (CCR) is a wrapper of calcite-components (CC), which makes versioning trickier. When CC releases a breaking change, CCR also needs to release a breaking change. release-please has a plugin called linked-versions, which makes sure the specified packages always stay the same version. It works by checking the conventional commits of the specified packages, and then bumps all linked packages to the highest version of the bunch. Moving forward, the CC and CCR versions will always be the same. The design-tokens and other packages can still be versioned separately.
Using linked-versions means a bug fix in CCR will also bump CC a patch version, even if there are no deployable changes. This doesn't make sense because CC isn't dependent on CCR. To keep versions tidy, any deployable changes to CCR are not released until changes of the equivalent semver type are installed in CC. However, there has never been a deployable change to CCR that wasn't from CC, so this shouldn't be an issue.
release-please has a couple issues (#510 and #1355) that block Calcite from using it for next releases. Calcite will use Lerna for next release versioning and changelog generation until the issues are resolved.
The git tags will contain the package name and version in the format of <package_name>@<package_version> to prevent collision between tags for different packages. The <package_name> is the name field in its package.json file, e.g. @esri/calcite-components. Each bumped package will have their own tag on the release commit. The tag format is Lerna's default, and how Lerna determines which commit to end at when generating changelog entries. The default tag format for release-please is different, but configurable. release-please looks for GitHub releases when determining the commit where the changelog entries should end.
release-please creates a GitHub release for each deployed package. Lerna's version command has a --create-release flag which creates GitHub releases, but Calcite is not using it to prevent confusing release-please.
Calcite uses the conventional commits changelog format, which is the default for release-please. Lerna uses the angular format by default. Calcite configured Lerna to use the conventional commits changelog preset so the formats match between tools.
Lerna generates a changelog section for each next version. Before latest releases, a script removes Lerna's next changelog sections, which are then replaced with a single latest section by release-please. This means edits to changelog entries under next headers will not automatically carry over to the upcoming latest release section.
To update a changelog entry, you must edit the merged PR's body following the steps documented by release-please. This will ensure the message is fixed in the subsequent latest release.
Optionally, you can create a PR that fixes the changelog entry under the next section. This isn't necessary if the latest release will occur soon, but may be a good idea at the beginning of a sprint so next users see the correct message in the changelog.
The release CI consists of three GitHub Actions and three scripts.
next releases happen in the deploy-next.yml GitHub Action, which runs on pushes to main. The Action runs the isNextDeployable.ts script to determine whether any fixes, feats, or breaking changes were installed since the most recent #git-tags (aka release). If there are deployable changes, Lerna versions the relevant packages and generates the new changelog entries.
The syncLinkedPackageVersions.ts executes after Lerna versions the packages. The script makes sure CCR's (and potentially other packages in the future) semver version isn't greater or less than CC's version. CCR's version is bumped to CC's version if it is behind. If CCR's version is greater, next releases for all packages will be blocked until CC's version catches up.
After versioning, a commit is created, which is tagged for each released package. Lerna publishes to NPM in topological order, which ensures local dependencies are published before the packages that depend on them. Lastly, the commit and tags are pushed to main.
A deploy-latest.yml GitHub Action runs release-please, which creates the release PR and updates it when there are new deployable changes pushed to main. The PR contains the following changes:
- new #changelog section and entries
- version updates in
package.jsonfor the package itself and/or local dependencies which were updated - version updates in
.release-please-manifest.json - version update in the CDN link in
packages/calcite-components/readme.md
After installing the PR, the Action creates #git-tags and #github-releases for each bumped package, and then deploys to NPM.
A remove-prerelease-changelog-entries.yml GitHub Action runs the removePrereleaseChangelogEntries.ts script every time release-please pushes changes to its branch. This ensures all next, hotfix, and rc changelog sections created by Lerna are removed before a dev installs the PR.
For the most part the releases are automated in the CI (see the sections above). However, there are a few manual steps, which are described in the releasing documentation.