Analyzing NPM projects fails the DependencyGraphBuilder with "The following references do not actually refer to packages"

[sschuberth]

Running NPM analysis on https://github.com/doubleopen-project/dos fails with

Exception in thread "main" java.lang.IllegalArgumentException: The following references do not actually refer to packages: [Identifier(type=NPM, namespace=, name=database, version=), Identifier(type=NPM, namespace=, name=s3-helpers, version=), Identifier(type=NPM, namespace=, name=spdx-validation, version=), Identifier(type=NPM, namespace=, name=validation-helpers, version=)].
	at org.ossreviewtoolkit.model.utils.DependencyGraphBuilder.checkReferences(DependencyGraphBuilder.kt:204)
	at org.ossreviewtoolkit.model.utils.DependencyGraphBuilder.build(DependencyGraphBuilder.kt:177)
	at org.ossreviewtoolkit.model.utils.DependencyGraphBuilder.build$default(DependencyGraphBuilder.kt:176)
	at org.ossreviewtoolkit.plugins.packagemanagers.node.npm.Npm.createPackageManagerResult(Npm.kt:146)
	at org.ossreviewtoolkit.analyzer.PackageManager.resolveDependencies(PackageManager.kt:326)
	at org.ossreviewtoolkit.analyzer.PackageManagerRunner$run$3.invokeSuspend(Analyzer.kt:321)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)
	at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:113)
	at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:89)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:586)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:820)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:717)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:704)

This used to work before and was probably broken by the NPM packager manager rewrite.

[sschuberth]

For the record, it seems this regression did not surface before #9616, although it was introduced before that.

[klaxa]

Hello,

I can confirm this issue for our project and did a superficial bisect over the tags. For our project at least the analysis produces results up until 39.0.0, but starting with 40.0.0 it aborts with the same error type (different packages) as the opening post. Sadly I am not yet familiar enough with the code base to assist much further at this point.

Best regards

[sschuberth]

the analysis produces results up until 39.0.0, but starting with 40.0.0 it aborts

Thank for this analysis!

Sadly I am not yet familiar enough with the code base to assist much further at this point.

I started looking into the issue, but got distracted and wasn't able to continue so far.

[Etsija]

I managed to run full git bisect for this issue: started from v39.0.0 (GOOD) and from the other end, pretty widely from v45.0.0 (BAD). According to the bisect, this commit seems to break the Analyzer:

03560a58b357023745b1717b29330437e837b476 is the first bad commit

commit 03560a58b357023745b1717b29330437e837b476
Author: Frank Viernau <[email protected]>
Date:   Thu Nov 7 14:00:17 2024 +0100

    refactor(node)!: Make `Npm` separate from `Yarn`
    
    Analog to 0eb1eea, remove the inheritance between the two managers and
    re-write large parts of `Npm` to extract all needed information solely
    based on the output of the `npm` CLI command, instead of relying on the
    file hierarchy under the `node_modules` directory.
    
    This reduces complexity and makes the implementation(s) easy to
    understand, maintain and change in isolation.
    
    Note: The handling of the `installIssues` in `Yarn` has been used only
    for `Npm`, which is why that code is moved from `Yarn` to `Npm`.
    
    Signed-off-by: Frank Viernau <[email protected]>

What I did and what the criterias were for GOOD and BAD:

1. For testing, I used our DOS monorepo, which was anchored to this revision: doubleopen-project/dos@62a5d31
2. For each bisect step, I ran

> ./gradlew installDist
> ort --info analyze -i <input_dir> -o <output_dir>

3. Criterias:

• GOOD: Analyzer runs successfully and produces analyzer-report.yml which is roughly 500kB and includes dependencies.
• BAD: either

(a) Closer to v39.0.0, Analyzer writes only a 7kB results file, which is completely missing the dependencies, and logs this issue:

      'NPM::package.json:':
      - timestamp: "2025-02-10T08:25:03.062115412Z"
        source: "NPM"
        message: "NPM failed to resolve dependencies for path 'package.json': IOException:\
          \ Running 'npm list --depth Infinity --json --long' in '/input' failed with\
          \ exit code 1:\nnpm error code ELSPROBLEMS\nnpm error invalid: @typescript-eslint/[email protected]\
          \ /input/node_modules/@typescript-eslint/eslint-plugin\nnpm error A complete\
          \ log of this run can be found in: /home/ort/.npm/_logs/2025-02-10T08_25_00_475Z-debug-0.log\n"
        severity: "ERROR"

(b) Closer to v45.0.0, Analyzer completely fails to resolve the projects of our monorepo and fails with this error log:

12:14:54.147 [DefaultDispatcher-worker-1] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'npm info --json [email protected]' in '/home/jyrki/dos/node_modules/scanner-worker'...
12:14:55.255 [DefaultDispatcher-worker-1] WARN  org.ossreviewtoolkit.plugins.packagemanagers.node.npm.Npm - Error getting details for [email protected] in directory /home/jyrki/dos/node_modules/scanner-worker: Running 'npm info --json [email protected]' in '/home/jyrki/dos/node_modules/scanner-worker' failed with exit code 1:
{
  "error": {
    "code": "E404",
    "summary": "Not Found - GET https://registry.npmjs.org/scanner-worker - Not found",
    "detail": "\n '[email protected]' is not in this registry.\n\nNote that you can also install from a\ntarball, folder, http url, or git url."
  }
}

12:14:55.259 [DefaultDispatcher-worker-1] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'npm info --json @types/[email protected]' in '/home/jyrki/dos/node_modules/@types/jest'...
12:14:56.031 [DefaultDispatcher-worker-1] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'npm info --json [email protected]' in '/home/jyrki/dos/node_modules/spdx-exceptions'...
12:14:56.658 [DefaultDispatcher-worker-1] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'npm info --json [email protected]' in '/home/jyrki/dos/node_modules/spdx-license-ids'...
12:14:57.261 [DefaultDispatcher-worker-1] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'npm info --json @types/[email protected]' in '/home/jyrki/dos/node_modules/@types/is-glob'...
12:14:59.532 [DefaultDispatcher-worker-1] INFO  org.ossreviewtoolkit.utils.common.DirectoryStash - Moving back directory from '/home/jyrki/dos/.stash11363407600901117333/node_modules' to '/home/jyrki/dos/node_modules'.
12:14:59.532 [DefaultDispatcher-worker-1] INFO  org.ossreviewtoolkit.analyzer.PackageManager - NPM resolved dependencies for path 'package.json' in 10m 38.513115124s.
Exception in thread "main" java.lang.IllegalArgumentException: The following references do not actually refer to packages: [Identifier(type=NPM, namespace=, name=database, version=), Identifier(type=NPM, namespace=, name=s3-helpers, version=), Identifier(type=NPM, namespace=, name=spdx-validation, version=), Identifier(type=NPM, namespace=, name=validation-helpers, version=)].
	at org.ossreviewtoolkit.model.utils.DependencyGraphBuilder.checkReferences(DependencyGraphBuilder.kt:204)
	at org.ossreviewtoolkit.model.utils.DependencyGraphBuilder.build(DependencyGraphBuilder.kt:177)
	at org.ossreviewtoolkit.model.utils.DependencyGraphBuilder.build$default(DependencyGraphBuilder.kt:176)
	at org.ossreviewtoolkit.plugins.packagemanagers.node.npm.Npm.createPackageManagerResult(Npm.kt:146)
	at org.ossreviewtoolkit.analyzer.PackageManager.resolveDependencies(PackageManager.kt:326)
	at org.ossreviewtoolkit.analyzer.PackageManagerRunner$run$3.invokeSuspend(Analyzer.kt:321)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)
	at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:113)
	at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:89)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:586)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:820)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:717)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:704)

Now this two-fold failure manifestation might have set the bisecting off, but at least it's a start I believe. For info, @fviernau.

[lamppu]

Yeah, it seems there's a couple issues here, the analysis crashes completely from 81f58ea on for the DOS monorepo, and as @Etsija found, the NPM packages aren't found after 03560a5.

Now that I found this thread #9551 though, I'm testing to see if recreating the package-lock.json for DOS will resolve these issues.

[lamppu]

Interestingly, after recreating the package-lock.json for DOS, the analyzer now crashes already with 03560a5 but 81f58ea doesn't fix it, it still crashes.