[Bug?]: Aliased installs with node_modules are many times larger than the unaliased versions

[jpm4]

Self-service

• I'd be willing to implement a fix

Describe the bug

On an attempted upgrade to 3.2.0-rc.10 in a large monorepo, our node_modules size went from ~1.8G to ~2.4G. The most obvious examples of installs that increased in size were aliased ones - the most extreme case I've seen is [email protected], from 2.3M -> 260M.

There is a large "self reference" directory (node_modules/webpack4/node_modules/webpack) as the deleted test here would normally test for: https://github.com/yarnpkg/berry/pull/3631/files#diff-ca05954739eec8e45eefa4b801ae3909a5069348db95ca5fb1c7be9ff57cf54aL344 (We must not create self-reference directory 'node_modules/no-deps2/node_modules/no-deps'). That PR is relevant to us for the issue it fixes, v3.1.1 breaks.

To reproduce

{
  "packageManager": "[email protected]",
  "dependencies": {
    "webpack": "^4.46.0",
    "webpack4": "npm:webpack@^4.46.0"
  }
}

Observe large size difference between webpack and webpack4. This can also be observed with separate installs, which would be more realistic anyway. I'd guess this would work with many/all packages; another one I tried at random was selenium-webdriver.

Environment

System:
    OS: macOS 11.6.2
    CPU: (8) x64 Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz
  Binaries:
    Node: 12.22.8 - /path/to/node
    Yarn: 3.2.0-rc.10 - /path/to/yarn
    npm: 6.14.15 - ~/.nvm/versions/node/v12.22.8/bin/npm

Additional context

No response

[paul-soporan]

Creating self-reference directories for aliases when using the NM linker is required to fulfill the require promise (if we didn't, webpack4 wouldn't be able to require itself as webpack), the fact that we didn't do it before was a correctness issue.

The solution is to use nmMode: hardlinks-local (which will be the new default starting from 4.x) so that the files in the self-reference directory are hardlinked rather than duplicated.

[jpm4]

Nice thanks for that tip! On 3.2.0-rc.10 that brings the node_modules size back down to what it was or even slightly lower. The node_modules/aliased_webpack4 dir in our project (not in the minimal repro) is still much bigger than before (~77MB), but maybe that doesn't matter if the entirety of node_modules is smaller?

This also looks to have sped up installs a ton, from 2m -> 1m with node_modules deleted, or 6m -> 3m with .yarn/cache and install-state deleted as well.

According to #3591 there's no real downside to this approach that you've seen?

[yarnbot]

Hi! 👋

This issue looks stale, and doesn't feature the reproducible label - which implies that you didn't provide a working reproduction using Sherlock. As a result, it'll be closed in a few days unless a maintainer explicitly vouches for it or you edit your first post to include a formal reproduction (you can use the playground for that).

Note that we require Sherlock reproductions for long-lived issues (rather than standalone git repositories or similar) because we're a small team. Sherlock gives us the ability to check which bugs are still affecting the master branch at any given point, and decreases the amount of code we need to run on our own machines (thus leading to faster bug resolutions). It helps us help you! 😃

If you absolutely cannot reproduce a bug on Sherlock (for example because it's a Windows-only issue), a maintainer will have to manually add the upholded label. Thanks for helping us triaging our repository! 🌟