Parallel node-gyp builds failing due to concurrent file access (Windows only)

[MRayermannMSFT]

Hi! I'm looking for help solving an issue we're starting to see more often on Windows, when npm install builds our native node modules.

The issue is that because npm install runs native module builds in parallel, and each needs exclusive access to some files, they end up failing due to not being able to get that exclusive access. We believe we're seeing this issue more often because we recently standardized on using node-addon-api for all five of our native modules. Thus, we have five parallel builds, all trying to access a lot of the same files. I've provided a series of example errors below (if someone wants full logs, I can provide those as well).

To be clear, I don't think this is necessarialy node-gyp's fault, however it may be part of the solution. For now, I am here to see if anyone in the node-gyp community has any ideas/thoughts/suggestions on how to proceed next. So far here's some ideas I've had:

1. Use --install-strategy=nested to de-deduplicate node-addon-api, thus reducing (maybe even eliminating) the chance of conflicting file access. Our project is ultimately bundled, so this will probably work out ok, but there are likely going to be other complications. Would be nice if we could only de-deduplicate node-addon-api.
2. Combine some of the five native modules, thus reducing the total number. This won't fix the problem, but it'll reduce the chance of it happening.
3. Solicit npm to not do parallel native node module installs.
4. Add a retry mechanism to node-gyp.
5. Prebuild the binaries for as many of our native node_modules as we can, thus avoiding having to rebuild them for the main project.
6. Write our own build-time code to de-deduplicate node-addon-api.
7. Use --ignore-scripts, and then use npm rebuild for any package which has an installScript: true in package-lock. Currently using this.
8. Use --foreground-scripts. Need to test this.

The specified task executable "CL.exe" could not be run. System.IO.IOException: The process cannot access the file 'PATH_TO_NODE_MODULES\node-addon-api\Release\obj\nothing\nothing.tlog\CL.command.1.tlog' because it is being used by another process. [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at System.IO.StreamReader..ctor(String path, Encoding encoding, Boolean detectEncodingFromByteOrderMarks, Int32 bufferSize, Boolean checkHost) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at System.IO.StreamReader..ctor(String path) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at System.IO.File.OpenText(String path) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at Microsoft.Build.CPPTasks.TrackedVCToolTask.MapSourcesToCommandLines() [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at Microsoft.Build.CPPTasks.TrackedVCToolTask.GenerateSourcesOutOfDateDueToCommandLine() [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at Microsoft.Build.CPPTasks.CL.ComputeOutOfDateSources() [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at Microsoft.Build.CPPTasks.TrackedVCToolTask.SkipTaskExecution() [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at Microsoft.Build.Utilities.ToolTask.Execute() [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]

node-gyp\src\win_delay_load_hook.cc(1,1): error C1083: Cannot open compiler generated file: 'PATH_TO_NODE_MODULES\node-addon-api\Release\obj\nothing\win_delay_load_hook.obj': Permission denied [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]

npm error LINK : fatal error LNK1181: cannot open input file 'Release\obj\nothing\\node-addon-api\nothing.obj' [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]

The specified task executable "CL.exe" could not be run. System.IO.IOException: The process cannot access the file 'PATH_TO_NODE_MODULES\node-addon-api\Release\obj\nothing\nothing.tlog\CL.write.1.tlog' because it is being used by another process. [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at System.IO.StreamWriter.CreateFile(String path, Boolean append, Boolean checkHost) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding, Int32 bufferSize, Boolean checkHost) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at System.IO.File.InternalWriteAllText(String path, String contents, Encoding encoding, Boolean checkHost) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at System.IO.File.WriteAllText(String path, String contents, Encoding encoding) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at Microsoft.Build.Utilities.CanonicalTrackedOutputFiles.SaveTlog(DependencyFilter includeInTLog) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at Microsoft.Build.CPPTasks.CL.PostExecuteTool(Int32 exitCode) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at Microsoft.Build.CPPTasks.TrackedVCToolTask.ExecuteTool(String pathToTool, String responseFileCommands, String commandLineCommands) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at Microsoft.Build.CPPTasks.CL.ExecuteTool(String pathToTool, String responseFileCommands, String commandLineCommands) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at Microsoft.Build.Utilities.ToolTask.Execute() [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]

npm error LINK : fatal error LNK1181: cannot open input file 'Release\obj\nothing\win_delay_load_hook.obj' [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]

npm error C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppBuild.targets(386,5): error MSB3374: The last access/last write time on file "Release\obj\nothing\nothing.tlog\unsuccessfulbuild" cannot be set. Could not find file 'PATH_TO_NODE_MODULES\node-addon-api\Release\obj\nothing\nothing.tlog\unsuccessfulbuild'.  [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]

npm error FileTracker : error FTK1011: could not create the new file tracking log file: PATH_TO_NODE_MODULES\node-addon-api\Release\obj\nothing\nothing.tlog\CL.write.1.tlog. The file exists. [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]

[lukekarrys]

@MRayermannMSFT I think 3 is likely the best option since npm is in charge of how and when those scripts get run.

There is also a hack/workaround that uses the npm flag --foreground-scripts which forces npm scripts to run sequentially: npm/cli#3034 (comment). Can you try and see if this flag does the same to native module builds?

[MRayermannMSFT]

There is also a hack/workaround that uses the npm flag --foreground-scripts which forces npm scripts to run sequentially:

Oh nice! I had actually just implemented a workaround using --ignore-scripts, where we use that flag, and then we use npm rebuild for any package which has an installScript: true in package-lock. I'll give --foreground-scripts a try as well though.