[Bug?]: "ENOENT: no such file or directory, link ..." when configured for nodeLinker: pnpm and running in Docker with mounted filesystem

[broofa]

Self-service

• I'd be willing to implement a fix

Describe the bug

As the title says, yarn throws a "no such file or directory" error while trying to hard-link a file from the globalStore into a project's node_modules/.store when configured for pnpm mode, and when running in Docker and mounting a host volume

See the attached MRE for reproduction recipe. In addition to the reproduction below, it's worth noting...

• If you just cd into the example directory and run yarn on your host (i.e. not in Docker), it runs with no issue. (Or at least for me it did, on my Mac system.)
• Running yarn on the host (successfully) causes the problem to go away when running in Docker... presumably because the globalStore and .store have been properly linked up.
• Deleting ./node_modules and ./yarn-cache should return the example back to a state that produces the error when installing in Docker

To reproduce

With Docker Desktop installed...

• Download and unpack yarn_pnpm_link_test.zip
  (contains 3 files: Dockerfile, package.json, and yarnrc.yml)
• cd into the unpacked directory, then ...
• docker build -t yarntest .
• docker run -v .:/app yarntest

On my system, the above results in the following:

---

$ #### Build Docker image
$ docker build -t yarntest .
[+] Building 0.1s (7/7) FINISHED                                                                                                      docker:desktop-linux
 => [internal] load build definition from Dockerfile                                                                                                  0.0s
 => => transferring dockerfile: 162B                                                                                                                  0.0s
 => [internal] load metadata for docker.io/library/node:20                                                                                            0.0s
 => [internal] load .dockerignore                                                                                                                     0.0s
 => => transferring context: 2B                                                                                                                       0.0s
 => [1/3] FROM docker.io/library/node:20                                                                                                              0.0s
 => CACHED [2/3] WORKDIR /app                                                                                                                         0.0s
 => CACHED [3/3] RUN corepack enable     && corepack use yarn                                                                                         0.0s
 => exporting to image                                                                                                                                0.0s
 => => exporting layers                                                                                                                               0.0s
 => => writing image sha256:03d450b54cf9a124e345e1cd7e44b4aecee2bd402721e8b79594c8de8ee76b15                                                          0.0s
 => => naming to docker.io/library/yarntest                                                                                                           0.0s

View build details: docker-desktop://dashboard/build/desktop-linux/desktop-linux/xwii14idy8g1ja85bvcyj8mby

What's next:
    View a summary of image vulnerabilities and recommendations → docker scout quickview

$ #### Run Docker container
$ docker run -v .:/app yarntest
➤ YN0000: · Yarn 4.4.0
➤ YN0000: ┌ Resolution step
➤ YN0000: └ Completed
➤ YN0000: ┌ Fetch step
➤ YN0013: │ 117 packages were added to the project (+ 8.58 MiB).
➤ YN0000: └ Completed in 0s 904ms
➤ YN0000: ┌ Link step
➤ YN0001: │ Error: ENOENT: no such file or directory, link '/app/yarn-cache/index/57/578785efdec6fa01dae357fb1e7675ccac00861e.dat' -> '/app/node_modules/.store/@babel-helper-validator-identifier-npm-7.24.7-748889c8d2/package/LICENSE'
➤ YN0000: └ Completed in 2s 490ms
➤ YN0000: · Failed with errors in 3s 425ms
node:internal/process/promises:391
    triggerUncaughtException(err, true /* fromPromise */);
    ^

[Error: ENOENT: no such file or directory, link '/app/yarn-cache/index/57/578785efdec6fa01dae357fb1e7675ccac00861e.dat' -> '/app/node_modules/.store/@babel-helper-validator-identifier-npm-7.24.7-748889c8d2/package/LICENSE'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'link',
  path: '/app/yarn-cache/index/57/578785efdec6fa01dae357fb1e7675ccac00861e.dat',
  dest: '/app/node_modules/.store/@babel-helper-validator-identifier-npm-7.24.7-748889c8d2/package/LICENSE'
}

Node.js v20.16.0

Environment

System:
    OS: macOS 14.5
    CPU: (10) arm64 Apple M1 Max
  Binaries:
    Node: 20.16.0 - /private/var/folders/h3/trw0qb4n5t39jdpfqnt20qy80000gn/T/xfs-c29d33b7/node
    Yarn: 4.4.0 - /private/var/folders/h3/trw0qb4n5t39jdpfqnt20qy80000gn/T/xfs-c29d33b7/yarn
    npm: 10.8.1 - ~/.nvm/versions/node/v20.16.0/bin/npm

Additional context

No response

[broofa]

@camrionnvmff Thanks for the input, but I don't see how any of what you're suggesting would seem to apply here. Specifically:

• Not using Docker is not an option. (Large, production system where Docker is baked into the infrastructure.)
• Internalizing the volume within the container is not an option. (the main point of this setup is to share the globalStore across many hosts / containers)
• It's not an issue with the file not existing. See below.
• Creating hard links in Docker isn't an issue. See below.
• File permissions aren't an issue. See below.
• Caching is not an issue. (I'm not sure how caching would be relevant here.)

The problem is that yarncomplains about a "missing file" that isn't actually missing. Furthermore, the hard link it's trying to create can be created just fine from w/in Docker, as shown here ...

---

$ #### In host, return file system back to initial example state (just in case)
$ rm -fr .yarn node_modules/ yarn-cache yarn.lock
$ ls -Fa
./            ../           .yarnrc.yml   Dockerfile    package.json

$ #### Start interactive Docker session (assumes "yarntest" image is already built per instructions in issue description)
$ docker run -it -v .:/app --entrypoint bash yarntest

root@394ea5ab1794:/app# #### (Now inside of Docker ...)

root@394ea5ab1794:/app# #### Run yarn (generates error below)
root@394ea5ab1794:/app# yarn

( snip )

[Error: ENOENT: no such file or directory, link '/app/yarn-cache/index/5a/5aaf48196ddd4d007a3067aa7f30303ca8e4b29c.dat' -> '/app/node_modules/.store/resolve-from-npm-4.0.0-f758ec21bf/package/license'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'link',
  path: '/app/yarn-cache/index/5a/5aaf48196ddd4d007a3067aa7f30303ca8e4b29c.dat',
  dest: '/app/node_modules/.store/resolve-from-npm-4.0.0-f758ec21bf/package/license'
}

( snip )

root@394ea5ab1794:/app# #### Verify "path" file named in error actually exists 
root@394ea5ab1794:/app# ls /app/yarn-cache/index/5a/5aaf48196ddd4d007a3067aa7f30303ca8e4b29c.dat
/app/yarn-cache/index/5a/5aaf48196ddd4d007a3067aa7f30303ca8e4b29c.dat

root@394ea5ab1794:/app# #### Verify "dest" file named in error does not exist (expected)
root@394ea5ab1794:/app# ls /app/node_modules/.store/resolve-from-npm-4.0.0-f758ec21bf/package/license
ls: cannot access '/app/node_modules/.store/resolve-from-npm-4.0.0-f758ec21bf/package/license': No such file or directory

root@394ea5ab1794:/app# #### Create hard link from "source" to "path" (what yarn is trying to do)
root@394ea5ab1794:/app# ln /app/yarn-cache/index/5a/5aaf48196ddd4d007a3067aa7f30303ca8e4b29c.dat /app/node_modules/.store/resolve-from-npm-4.0.0-f758ec21bf/package/license

root@394ea5ab1794:/app# #### Verify hard link succeeded
root@394ea5ab1794:/app# ls /app/node_modules/.store/resolve-from-npm-4.0.0-f758ec21bf/package/license
/app/node_modules/.store/resolve-from-npm-4.0.0-f758ec21bf/package/license

[broofa]

FWIW, my suspicion is this is caused by a race condition of some sort, where yarn is creating the "source" file, and then attempting to create a hard link for it into the ".store" directory before the create operation is fully complete. 'Not sure how else to explain the behavior I'm seeing.

[broofa]

Quick breadcrumb comment ...

I reached out to Yarn support on discord, and @arcanis responded with the following:

I don't have much time atm to dig into this, but it looks strange, I don't see an obvious error in the implementation
we perform the write into the store here: https://github.com/yarnpkg/berry/blob/master/packages/yarnpkg-fslib/sources/algorithms/copyPromise.ts#L233-L238
and we create the link right after, here: https://github.com/yarnpkg/berry/blob/master/packages/yarnpkg-fslib/sources/algorithms/copyPromise.ts#L267
I'd suggest to use yarn set version from sources --no-minify to get a yarn.js you can edit, and add some console.log around those places to 1/ confirm those are the impacted places (with a try/catch around the linkPromise, for example) and 2/ double-check whether that's an issue caused by Docker or not (perhaps it doesn't immediately create the file?)

I looked at the yarn code in question and, on the surface, it seems to be properly awaiting everything. (No surprise, that.)

@quezo and I have been digging into this a bit further on the Docker side and suspect this may be something to do with Docker Desktop's handling of the file system on Mac. We tested the above example on an EC2 Linux instance and it worked without error, fwiw.