Long node_modules paths cannot be found on Windows when LongPathsEnabled enabled

[karlhorky]

Version

v20.9.0

Platform

Microsoft Windows NT 10.0.20348.0 x64

Subsystem

No response

What steps will reproduce the bug?

I created a reproduction repo at https://github.com/karlhorky/node-js-max_path-windows-bug . The steps in the GitHub Actions workflow file reproduce the bug:

# Test whether Windows support for long paths is enabled
(Get-ItemProperty "HKLM:System\CurrentControlSet\Control\FileSystem").LongPathsEnabled
# 1

# Clone, setup Node.js

# Install dependencies with npm
npm install

# Run program in short path (✅ works)
node eslint.config.js

# Make directory with long path and copy all files there
mkdir long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\
cp eslint.config.js long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\
cp index.js long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\
cp package.json long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\

# Change to directory with long path
cd long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\

# Install dependencies with npm
npm install

# Run program in long path (💥 crashes)
node eslint.config.js

The error that shows up is this:

node:internal/process/esm_loader:[4](https://github.com/karlhorky/node-js-max_path-windows-bug/actions/runs/6890011394/job/18742165029#step:11:5)0
      internalBinding('errors').triggerUncaughtException(
                                ^

Error: Cannot find package 'D:\a\node-js-max_path-windows-bug\node-js-max_path-windows-bug\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\node_modules\eslint-plugin-import\package.json' imported from D:\a\node-js-max_path-windows-bug\node-js-max_path-windows-bug\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\eslint.config.js
Did you mean to import eslint-plugin-import/lib/index.js?
    at legacyMainResolve (node:internal/modules/esm/resolve:189:26)
    at packageResolve (node:internal/modules/esm/resolve:776:14)
    at moduleResolve (node:internal/modules/esm/resolve:838:20)
    at defaultResolve (node:internal/modules/esm/resolve:1043:11)
    at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:383:12)
    at ModuleLoader.resolve (node:internal/modules/esm/loader:3[5](https://github.com/karlhorky/node-js-max_path-windows-bug/actions/runs/6890011394/job/18742165029#step:11:6)2:25)
    at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:228:38)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/module_job:85:39)
    at link (node:internal/modules/esm/module_job:84:3[6](https://github.com/karlhorky/node-js-max_path-windows-bug/actions/runs/6890011394/job/18742165029#step:11:7)) {
  code: 'ERR_MODULE_NOT_FOUND'
}

Node.js v20.[9](https://github.com/karlhorky/node-js-max_path-windows-bug/actions/runs/6890011394/job/18742165029#step:11:10).0
Error: Process completed with exit code 1.

How often does it reproduce? Is there a required condition?

Always

What is the expected behavior? Why is that the expected behavior?

If Windows support for long paths is turned on (see LongPathsEnabled check above showing that it's enabled) I would expect Node.js to also support long paths (longer than the Windows ~260 characters MAX_PATH limit)

What do you see instead?

The crash and error above

Additional information

More information about LongPathsEnabled :

• MAX_PATH lengths on Windows environment actions/runner-images#1052 (comment)

Context:

I'm here after originally reporting the bug in ESLint (I thought that this comment by @bzoz meant that Node.js is not affected by the 260 character path limit on Windows :

• Bug: Long node_modules plugin paths cannot be found on Windows eslint/eslint#17763

[MrJithil]

cc: @nodejs/platform-windows

[StefanStojanovic]

I took a look at it, but found another issue instead. @karlhorky I forked your repo and the failing run can be found here.

I made long path longer, because the original one was shorter then 260 characters. After I did it, Even running npm install (or anything else, example here) started failing. Since I'm seeing same error regardless if I'm using CMD or PowerShell it seems safe to assume this is something Windows specific.

Back to the issue you reported, importer uses file URLs internally, thus cannot use \\?\ prefix required for handling long paths. I'll investigate this further, and add another comment based on my findings.

[karlhorky]

Thanks for the response, and for the investigation! 🙌

I made long path longer, because the original one was shorter then 260 characters. After I did it, Even running npm install (or anything else, example here) started failing

I guess you mean "anything else Node.js-related"? Because the cp command on the previous step doesn't fail:

cp eslint.config.js long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\

But maybe the way that GitHub Actions is launching PowerShell + CMD has a problem - maybe a separate GitHub Actions-related problem cc @al-cheb @MaksimZhukov @chrispat:

An error occurred trying to start process 'C:\Program Files\PowerShell\7\pwsh.EXE' with working directory 'D:\a\node-js-max_path-windows-bug\node-js-max_path-windows-bug\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\'. The directory name is invalid.

I'll try to run various commands from a directory with a long path on my Windows machine locally and see what I find.

[karlhorky]

Ok, I ran some commands on a Windows 11 Pro x64 machine (both node and also cmd) I can confirm that running commands from within deep directories fails with The directory name is invalid:

PS C:\> node -v
v20.5.1

PS C:\> mkdir long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\


    Directory: C:\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\l
    ong-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-p
    ath\long-path\long-path\long-path\long-path


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d-----        17.11.2023     16:27                long-path


PS C:\> cd long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\
PS C:\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path> node -v
Program 'node.exe' failed to run: The directory name is invalidAt line:1 char:1
+ node -v
+ ~~~~~~~.
At line:1 char:1
+ node -v
+ ~~~~~~~
    + CategoryInfo          : ResourceUnavailable: (:) [], ApplicationFailedException
    + FullyQualifiedErrorId : NativeCommandFailed

Commands like cd still work:

PS C:\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path> cd .

cmd.exe also doesn't run:

PS C:\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path> cmd
Program 'cmd.exe' failed to run: The directory name is invalidAt line:1 char:1
+ cmd
+ ~~~.
At line:1 char:1
+ cmd
+ ~~~
    + CategoryInfo          : ResourceUnavailable: (:) [], ApplicationFailedException
    + FullyQualifiedErrorId : NativeCommandFailed

Also when running C:\Windows\System32\cmd.exe with the full path - still has a problem with the current directory being invalid:

PS C:\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path> C:\Windows\System32\cmd.exe
Program 'cmd.exe' failed to run: The directory name is invalidAt line:1 char:1
+ C:\Windows\System32\cmd.exe
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~.
At line:1 char:1
+ C:\Windows\System32\cmd.exe
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ResourceUnavailable: (:) [], ApplicationFailedException
    + FullyQualifiedErrorId : NativeCommandFailed

Wonder if someone from the Windows team involved in interop / this MAX_PATH limitation could confirm that running - I don't know of contacts who work on the Windows team, but maybe @JeremyKuhne (because of this blog post) or @iSazonov (because of involvement in PowerShell/PowerShell#17197 and PowerShell/PowerShell#13955) or @SteveL-MSFT (because of PowerShell/PowerShell involvement) ?

---

Anyway, back to the import / long path file reading issue, maybe at least this part could be supported by Node.js...

[StefanStojanovic]

Ok, I ran some commands on a Windows 11 Pro x64 machine (both node and also cmd) I can confirm that running commands from within deep directories fails with The directory name is invalid:

That is correct. cp is a command, not an executable, so I assume only applications are facing that issue.

Back to the import with long paths problem. Investigating a bit longer got me to one of my first PRs in this repo, especially to changes in the manifest file. Applying that change solved an issue for me, but I'd like to investigate a bit more to find the root cause. The original PR was dropped in favor of this one as that was enough to fix the issue and something similar should probably be possible here. I'll get back to this later this week to continue my work.

[karlhorky]

Regarding the "PowerShell unable to launch programs from long path" problem:

I've commented in an existing MAX_PATH issue in the PowerShell/PowerShell repo:

• Incorrect behavior running executable if current location path is longer than 256 characters PowerShell/PowerShell#13955 (comment)

I may need to open a new issue for this as well.

[StefanStojanovic]

After further investigation, I saw that some parts of the code responsible for module loading, do not add the required \\?\ prefix for Windows long path support. I was able to make a PoC fixing it without the changes to the manifest file, but that's just a starting point for further work.

[karlhorky]

Workaround (for pnpm)

pnpm install can sometimes create file / folder structures that have long paths, which can lead to confusing errors only on Windows such as "Cannot find package" with ESLint:

$ pnpm eslint . --max-warnings 0

Oops! Something went wrong! :(

ESLint: 8.53.0

Error: Cannot find package 'C:\Users\Victor\upleveled\testing-projects\next-js-example-winter-2024-atvie\node_modules\.pnpm\[email protected]_@[email protected]_@[email protected]_@[email protected]_@ty_lxumkk76msp4b3xunkneyh2dsu\node_modules\@next\eslint-plugin-next\package.json' imported from C:\Users\Victor\upleveled\testing-projects\next-js-example-winter-2024-atvie\node_modules\.pnpm\[email protected]_@[email protected]_@[email protected]_@[email protected]_@ty_lxumkk76msp4b3xunkneyh2dsu\node_modules\eslint-config-upleveled\index.js
Did you mean to import "@next/eslint-plugin-next/dist/index.js"?
    at legacyMainResolve (node:internal/modules/esm/resolve:215:26)
    at packageResolve (node:internal/modules/esm/resolve:841:14)
    at moduleResolve (node:internal/modules/esm/resolve:927:18)
    at defaultResolve (node:internal/modules/esm/resolve:1157:11)
    at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:390:12)
    at ModuleLoader.resolve (node:internal/modules/esm/loader:359:25)
    at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:234:38)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/module_job:87:39)
    at link (node:internal/modules/esm/module_job:86:36)

These errors are also caused by the Node.js incompatibility with long paths on Windows.

If you're encountering this, one thing you can try is to use pnpm with node-linker=hoisted in an .npmrc file in your project:

.npmrc

node-linker=hoisted

Source: https://github.com/upleveled/preflight-test-project-next-js-passing/blob/419e89d0eb4067d63373aa7daf703114f949896d/.github/workflows/ci.yml#L77-L81

cc @zkochan

[karlhorky]

It looks like the Node.js v22 PR contains the fix "module,win: fix long path resolve (by @StefanStojanovic) #51097":

• 2024-04-24, Version 22.0.0 (Current) #52505

So maybe Node.js has support for long paths on Windows now!

(make sure that you have LongPathsEnabled set to true on the Windows machine you test on)

[karlhorky]

@StefanStojanovic thanks for the PR btw, excited for this to be eventually fixed!

I just upgraded to Node.js 22.1.0 in my reproduction repo in the PR below:

• Upgrade to Node.js v22 karlhorky/node-js-max_path-windows-bug#1

However, the test run of the GitHub Actions workflow still failed:

Run (Get-ItemProperty "HKLM:System\CurrentControlSet\Control\FileSystem").LongPathsEnabled
1

Run actions/setup-node@v4
Attempting to download >=22.1.0...
Not found in manifest. Falling back to download directly from Node
Acquiring 22.1.0 - x64 from https://nodejs.org/dist/v22.1.0/node-v22.1.0-win-x64.7z
Extracting ...
Adding to the cache ...
Done
Environment details
  node: v22.1.0
  npm: 10.7.0
  yarn: 1.22.22

$ node eslint.config.js
node:internal/modules/run_main:125
    triggerUncaughtException(
    ^

Error: Cannot find package 'D:\a\node-js-max_path-windows-bug\node-js-max_path-windows-bug\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\node_modules\eslint-plugin-import\package.json' imported from D:\a\node-js-max_path-windows-bug\node-js-max_path-windows-bug\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\eslint.config.js
Did you mean to import "eslint-plugin-import/lib/index.js"?
    at legacyMainResolve (node:internal/modules/esm/resolve:214:26)
    at packageResolve (node:internal/modules/esm/resolve:832:14)
    at moduleResolve (node:internal/modules/esm/resolve:918:18)
    at defaultResolve (node:internal/modules/esm/resolve:1148:11)
    at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:541:12)
    at ModuleLoader.resolve (node:internal/modules/esm/loader:510:25)
    at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:240:38)
    at ModuleJob._link (node:internal/modules/esm/module_job:126:49) {
  code: 'ERR_MODULE_NOT_FOUND'
}

Node.js v22.1.0
Error: Process completed with exit code 1.

Here's the file it's running (this file runs fine in a location in a shorter path):

eslint.config.js

import eslintImport from 'eslint-plugin-import';

const config = [
  {
    plugins: {
      import: eslintImport,
    },
    rules: {
      'no-unused-vars': 'warn',
    },
  },
];

export default config;

Is there still a bug here? Maybe this issue should be reopened?

[karlhorky]

Friendly ping @StefanStojanovic - should this issue be reopened?

(I could also open a new issue, but would like to avoid that duplication if possible)

[StefanStojanovic]

Friendly ping @StefanStojanovic - should this issue be reopened?

(I could also open a new issue, but would like to avoid that duplication if possible)

I was out of the office last week. Thanks for bringing this to my attention again. I'm working on some other things now, but I'll take a look once I have some free cycles.

[karlhorky]

Great, thanks!

Is it possible to reopen the issue too? To keep it on the radar?

[karlhorky]

@StefanStojanovic Or I can open another issue with the same issue description and title and link it to this one, if that's better...

[StefanStojanovic]

@karlhorky we can do it both ways. I do not have time to work on this now, but I've talked to my colleague @huseyinacacak-janea and he'll take a look.

In my PR that addressed this issue, I added a new test test/es-module/test-GH-50753.js, which passes without any problems. Do you see some way to change that test and make it experience same issue you are facing with GitHub Action in your repo?

[karlhorky]

Ok great, thanks for the info!

I see the current version of the test at test/es-module/test-GH-50753.js here:

node/test/es-module/test-GH-50753.js

Lines 1 to 42 in 2aaeaa8

	'use strict';
	// Flags: --expose-internals
	const common = require('../common');
	if (!common.isWindows) {
	common.skip('this test is Windows-specific.');
	}
	const fs = require('fs');
	const path = require('path');
	const tmpdir = require('../common/tmpdir');
	// https://github.com/nodejs/node/issues/50753
	// Make a path that is more than 260 chars long.
	// Module layout will be the following:
	// package.json
	// main
	// main.js
	const packageDirNameLen = Math.max(260 - tmpdir.path.length, 1);
	const packageDirName = tmpdir.resolve('x'.repeat(packageDirNameLen));
	const packageDirPath = path.resolve(packageDirName);
	const packageJsonFilePath = path.join(packageDirPath, 'package.json');
	const mainDirName = 'main';
	const mainDirPath = path.resolve(packageDirPath, mainDirName);
	const mainJsFile = 'main.js';
	const mainJsFilePath = path.resolve(mainDirPath, mainJsFile);
	tmpdir.refresh();
	fs.mkdirSync(packageDirPath);
	fs.writeFileSync(
	packageJsonFilePath,
	`{"main":"${mainDirName}/${mainJsFile}"}`
	);
	fs.mkdirSync(mainDirPath);
	fs.writeFileSync(mainJsFilePath, 'console.log("hello world");');
	require('internal/modules/run_main').executeUserEntryPoint(packageDirPath);
	tmpdir.refresh();

Current version looks good, but low-level (whereas my reproduction was more on the "end to end" / "integration" end of the spectrum) - maybe there is something else in Node.js that is also failing... 🤔

[karlhorky]

@huseyinacacak-janea could you reopen this issue as a tracking issue for this work?

[karlhorky]

Oh, apparently @huseyinacacak-janea doesn't have access to reopen the issue

@StefanStojanovic could you do it instead?

[WangJincheng4869]

Can it be merged into other TLC versions?

[karlhorky]

@huseyinacacak-janea thanks for the PR fixing the problem in #53294, and thanks @RafaelGSS and other Node.js contributors for the Node.js 22.7.0 release PR #54452 where the fix landed

I just upgraded to Node.js 22.7.0 in my reproduction repo in the PR below:

• Upgrade to Node.js 22.7.0 karlhorky/node-js-max_path-windows-bug#2

I can confirm that the workflow run of the GitHub Actions workflow was successful 🎉

So for my reproduction case, this is now resolved 🙌

If there are any other further long path issues with Node.js on Windows, I'll be sure to open a different issue and mention it here (if the issue is not locked over time).

[ibrahimqasim-abtrace]

@karlhorky

I tried to follow this fix, and it seemed to work. However, trying longer paths still causes this to fail. In your example, the path length is still below 260. If you try creating a path that is longer than 260 and npm init in there, what happens then?

[karlhorky]

@ibrahimqasim-abtrace I have not noticed any failures, but it's possible that I'm not testing the same way you mean.

Can you fork my repo and create a reproduction that fails on GitHub Actions?

https://github.com/karlhorky/node-js-max_path-windows-bug

Then it will be a clear, reproducible failure that Node.js can look at.