Road to the next supermajor

[arcanis]

I'm opening this issue to share my vision and hear potential feedback regarding the next supermajor Yarn release. I’ve been working on a prototype intermittently for the past few months, but I expect to be able to allocate more time to it soon. Hopefully I'd like to release a first beta next summer.

Note

I mean by supermajor a release that will contain more significant changes in how we recommend to use Yarn than typical major releases (which are often prompted by technical requirements more than philosophical one; for example bumping the minimal Node.js version).

• P0 features are those I’m most excited about and feel confident will be implemented.
• P1 features are desirable but less certain, and will be included if feasible.
• P2 features are good candidates for minor releases.

[arcanis]

P0 - Smooth Migration

The migration from Yarn 1.0 to 2.0 was painful, and I still have that in mind. That's perhaps the most important item: whatever we do, it needs not only to have a smooth migration path, but also a clear one. We want to avoid the kind of confusion that occurred back then.

One thing I think will mitigate most risks is that unlike the aforementionned migration, we now have a solid test framework that can support us and guarantee we don't ship unforeseen breaking changes. This wasn't the case before, as the 1.0 testsuites were comprised of unit tests that couldn't survive the heavy refactoring we did for 2.0.

[arcanis]

P0 - Auto Installs

You can achieve today "zero-installs" by enabling PnP and checking-in both your dependencies and the Yarn-generated Node.js dependency loader inside your repository. Because this loader uses relative paths, this setup removes the need to perform installs during branch switches or main updates.

While this approach offers reliability, speed, and a streamlined developer experience, it requires adding binary artifacts to repositories. Though I once strongly advocated that Git could handle this approach, I have greater reservations after being exposed to the downsides of this approach on some very large projects. While I still think it's a simple and "good enough" strategy for many projects, I now want to explore alternative strategies that would achieve most the benefits for only a fraction of the downsides.

My current plan is to introduce a new pattern, tentatively called "auto installs." In this mode - potentially as a default - Yarn would automatically install dependencies if it detects they’re out-of-sync when running yarn run or yarn node. Because installs would become transparent from the user this would offer some of the benefits of zero-installs without the need for keeping binary artifacts in your repository.

Incidentally, it seems this is something that pnpm is also considering in some way.

[jwoo0122]

I'm really looking forward to seeing this. Currently using git post-checkout hook.

[arcanis]

P0 - Faster Installs

To support auto installs, I believe install times must become nearly instantaneous. I have some promising ideas for achieving this but am unsure of their full feasibility. I’m working on a prototype to clarify the approach.

[arcanis]

P0 - Reduced yarn run Overhead

The few seconds of overhead in yarn run have been frustrating and, with auto installs, any delay could become even more noticeable. I’m focused on making this command feel as responsive as if you were calling binaries directly.

[arcanis]

P1 - Task Management

The success of tools like Turbo show the value there is in having efficient task management when working with monorepo solutions. While Yarn attempts to provide a similar feature via yarn workspaces foreach, it's not as performant, easy to use, or flexible as it could be. I'd like to explore how we could improve this experience.

[arcanis]

P1 - Reduced Redundancy in Dependency Management

Yarn restricts packages to only their declared dependencies, ensuring consistency in versions and stability for developers. However, this can lead to redundancy, particularly for large monorepos where multiple workspaces depend on the same packages (e.g., React). Maintaining duplicate listings can become tedious, especially if other dependencies such as @types/node also need to be shared across workspaces.

One possible solution is a shared dependency concept, similar to pnpm's catalog feature. I’d like to explore whether Yarn could implement something like catalogs or an equivalent feature.

[wojtekmaj]

Among the features described here this excites me the most. I pretty much made a habit of running yarn dedupe every time I modify the dependencies somehow.

[arcanis]

P1 - Corepack Alternative

With the Node.js project reconsidering its stance on enabling Corepack by default, we’ll need to revisit our distribution strategy and decrease our reliance on third-party projects¹.

Whatever we do it'll for sure be compatible with the existing packageManager syntax, so the migration should be relatively painless.

Footnotes

1. Of course, if Node.js figures out how to move forward with Corepack it will be great; I'm still supportive of the project. But it became clear that we can't exclusively rely on that as a mean of distribution. ↩

[wojtekmaj]

I really really really wouldn't like to see corepack go, but I also don't understand the drama regarding whether to include it with Node.js or not.

Typing, for example, npm i corepack --global once instead of corepack enable wouldn't make a difference and corepack could still thrive.

[arcanis]

P1 - Plug'n'Play v2

Ever since we introduced Yarn Plug'n'Play in 2018, it remained mostly the same. The loader improved a lot, but the core concepts stayed the same, and so did the data format. Six years later I'm interested to explore evolutions we could bring to the table.

One particular idea I have in mind is to leverage import maps to simplify integrations. Yarn would generate an enhanced import-map-compatible JSON file along with the classic .pnp.cjs file, allowing third-party files to import packages without needing to know about the PnP loader. This isn't entirely trivial, as import maps don't support many of the resolution features you'd expect from package.json, so we'll see.

[arcanis]

P2 - Better gitignore management

Yarn now generates quite a few artifacts, and knowing which one to gitignore can be a bit of a pain. I'd like to explore ways to make this process more straightforward.

My current idea is to add a .yarn/ignore folder to the generated files. This folder would contain a preconfigured .gitignore file with a single * rule. Yarn would then put all temporary files into this folder, thus ensuring they get gitignored without you having to carefully manage your ignore list.

[arcanis]

P2 - Release Workflow

The release workflow introduced in Yarn 2 has been quite nice for us maintainers, but it's not as user-friendly as it could be. I'd like to spend some cycles improving it, potentially through an official GitHub Action workflow. Some ideas include:

• Checkboxes we can tick to indicate whether a package must be released
• Labels to automatically cherry-pick PRs to the patch branches
• A way to add release notes as PR comments