Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

"NoDataException: No documents exist" when local database does not exist yet #7227

Closed
marcelstoer opened this issue Dec 4, 2024 · 5 comments
Closed

"NoDataException: No documents exist" when local database does not exist yet #7227

marcelstoer opened this issue Dec 4, 2024 · 5 comments
Labels
bug

Comments

@marcelstoer
Copy link
Contributor

Describe the bug
If you don't have an existing data directory (e.g. ~/.m2/repository/org/owasp/dependency-check-data/11.0), the Maven plugin will fail to properly initialize a new database. It succeeds creating it but then fails the scan with

NoDataException: No documents exist

odc.mv.db will be around 300KB when instead it should be around 180MB.

However, if I already haven an older database in the 11.0 data dir, the plugin properly updates it.

Updated the CPE ecosystem on 141898 NVD records

Version of dependency-check used
The problem occurs using version 11.1.1 of the Maven plugin.

Log file
Excerpt of the Maven console output below. What I think we are seeing is that creating the local database completes successfully. No indication of a malfunction anywhere.

...
[DEBUG] Settings.getDataFile() - file: '[JAR]/../../dependency-check-data/11.0'
[DEBUG] Settings.getDataFile() - transforming filename
[DEBUG] Settings.getDataFile() - jar file: '/Users/marcelstoer/.m2/repository/org/owasp/dependency-check-utils/11.1.1'
[DEBUG] Settings.getDataFile() - returning: '/Users/marcelstoer/.m2/repository/org/owasp/dependency-check-utils/11.1.1/../../dependency-check-data/11.0'
[DEBUG] Data directory: /Users/marcelstoer/.m2/repository/org/owasp/dependency-check-utils/11.1.1/../../dependency-check-data/11.0
[DEBUG] Connection String: 'jdbc:h2:file:/Users/marcelstoer/.m2/repository/org/owasp/dependency-check-data/11.0/odc;AUTOCOMMIT=ON;CACHE_SIZE=65536;RETENTION_TIME=1000;MAX_COMPACT_TIME=10000;'
[DEBUG] Settings.getDataFile() - file: '[JAR]/../../dependency-check-data/11.0'
[DEBUG] Settings.getDataFile() - transforming filename
[DEBUG] Settings.getDataFile() - jar file: '/Users/marcelstoer/.m2/repository/org/owasp/dependency-check-utils/11.1.1'
[DEBUG] Settings.getDataFile() - returning: '/Users/marcelstoer/.m2/repository/org/owasp/dependency-check-utils/11.1.1/../../dependency-check-data/11.0'
[DEBUG] Lock file created (main) 37afd16e5985ac7aa1e1eb21ae5c0d43 @ 2024-12-04 17:48:33.989
[DEBUG] Loading driver 'org.h2.Driver'
[DEBUG] Settings.getDataFile() - file: '[JAR]/../../dependency-check-data/11.0'
[DEBUG] Settings.getDataFile() - transforming filename
[DEBUG] Settings.getDataFile() - jar file: '/Users/marcelstoer/.m2/repository/org/owasp/dependency-check-utils/11.1.1'
[DEBUG] Settings.getDataFile() - returning: '/Users/marcelstoer/.m2/repository/org/owasp/dependency-check-utils/11.1.1/../../dependency-check-data/11.0'
[DEBUG] Data directory: /Users/marcelstoer/.m2/repository/org/owasp/dependency-check-utils/11.1.1/../../dependency-check-data/11.0
[DEBUG] Connection String: 'jdbc:h2:file:/Users/marcelstoer/.m2/repository/org/owasp/dependency-check-data/11.0/odc;AUTOCOMMIT=ON;CACHE_SIZE=65536;RETENTION_TIME=1000;MAX_COMPACT_TIME=10000;'
[DEBUG] Settings.getDataFile() - file: '[JAR]/../../dependency-check-data/11.0'
[DEBUG] Settings.getDataFile() - transforming filename
[DEBUG] Settings.getDataFile() - jar file: '/Users/marcelstoer/.m2/repository/org/owasp/dependency-check-utils/11.1.1'
[DEBUG] Settings.getDataFile() - returning: '/Users/marcelstoer/.m2/repository/org/owasp/dependency-check-utils/11.1.1/../../dependency-check-data/11.0'
[DEBUG] Need to create DB Structure: false
[DEBUG] Loading database connection
[DEBUG] Connection String: jdbc:h2:file:/Users/marcelstoer/.m2/repository/org/owasp/dependency-check-data/11.0/odc;AUTOCOMMIT=ON;CACHE_SIZE=65536;RETENTION_TIME=1000;MAX_COMPACT_TIME=10000;
[DEBUG] Database User: dcuser
[DEBUG] Database product: h2
[DEBUG] DC Schema: 5.5
[DEBUG] DB Schema: 5.5
[INFO] Checking for updates
[DEBUG] starting getUpdatesNeeded() ...
[INFO] NVD API Cache requires several updates; this could take a couple of minutes.
[INFO] Download Started for NVD Cache - https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2012.json.gz
[DEBUG] Temporary directory is `/var/folders/pl/nxddb61s6y7_hdjqbr2l7pfm0000gn/T/dctemp6ee4a524-2432-48ee-92a7-c9ea279940c0`
[INFO] Download Started for NVD Cache - https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2023.json.gz
[INFO] Download Started for NVD Cache - https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2011.json.gz
[INFO] Download Started for NVD Cache - https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2022.json.gz
[INFO] Download Started for NVD Cache - https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2010.json.gz
[INFO] Download Started for NVD Cache - https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2021.json.gz
[INFO] Download Started for NVD Cache - https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2020.json.gz
[INFO] Download Started for NVD Cache - https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2009.json.gz
[INFO] Download Started for NVD Cache - https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2008.json.gz
[INFO] Download Started for NVD Cache - https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2019.json.gz
[INFO] Download Started for NVD Cache - https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2007.json.gz
[INFO] Download Started for NVD Cache - https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2018.json.gz
[INFO] Download Started for NVD Cache - https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2006.json.gz
[INFO] Download Started for NVD Cache - https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2017.json.gz
[INFO] Download Started for NVD Cache - https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2005.json.gz
[INFO] Download Started for NVD Cache - https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2016.json.gz
[INFO] Download Started for NVD Cache - https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2004.json.gz
[INFO] Download Started for NVD Cache - https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2015.json.gz
[INFO] Download Started for NVD Cache - https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2003.json.gz
[INFO] Download Started for NVD Cache - https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2014.json.gz
[INFO] Download Started for NVD Cache - https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2002.json.gz
[INFO] Download Started for NVD Cache - https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2013.json.gz
[INFO] Download Started for NVD Cache - https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2024.json.gz
[INFO] Download Started for NVD Cache - https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-modified.json.gz
[DEBUG] Updating the ecosystem cache
[DEBUG] Corrected the ecosystem for 0 ecoSystemCache entries
[INFO] Begin database maintenance
[INFO] End database maintenance (2 ms)
[DEBUG] Begin Engine Version Check
[DEBUG] Last checked: 0
[DEBUG] Now: 1733330966
[DEBUG] Current version: 11.1.1
[DEBUG] Checking web for new version.
[DEBUG] Current Release: 11.1.1
[DEBUG] Upgrade not needed
[DEBUG] Settings.getDataFile() - file: '[JAR]/../../dependency-check-data/11.0'
[DEBUG] Settings.getDataFile() - transforming filename
[DEBUG] Settings.getDataFile() - jar file: '/Users/marcelstoer/.m2/repository/org/owasp/dependency-check-utils/11.1.1'
[DEBUG] Settings.getDataFile() - returning: '/Users/marcelstoer/.m2/repository/org/owasp/dependency-check-utils/11.1.1/../../dependency-check-data/11.0'
[DEBUG] Begin RetireJS Update
[DEBUG] Settings.getDataFile() - file: '[JAR]/../../dependency-check-data/11.0'
[DEBUG] Settings.getDataFile() - transforming filename
[DEBUG] Settings.getDataFile() - jar file: '/Users/marcelstoer/.m2/repository/org/owasp/dependency-check-utils/11.1.1'
[DEBUG] Settings.getDataFile() - returning: '/Users/marcelstoer/.m2/repository/org/owasp/dependency-check-utils/11.1.1/../../dependency-check-data/11.0'
[DEBUG] Lock file created (main) d527030a2f4b6a0474144a05d05640cd @ 2024-12-04 17:49:26.83
[DEBUG] RetireJS Repo URL: https://raw.githubusercontent.com/Retirejs/retire.js/master/repository/jsrepository.json
[DEBUG] Lock released (main) d527030a2f4b6a0474144a05d05640cd @ 2024-12-04 17:49:27.062
[DEBUG] Settings.getDataFile() - file: '[JAR]/../../dependency-check-data/11.0'
[DEBUG] Settings.getDataFile() - transforming filename
[DEBUG] Settings.getDataFile() - jar file: '/Users/marcelstoer/.m2/repository/org/owasp/dependency-check-utils/11.1.1'
[DEBUG] Settings.getDataFile() - returning: '/Users/marcelstoer/.m2/repository/org/owasp/dependency-check-utils/11.1.1/../../dependency-check-data/11.0'
[DEBUG] Begin Hosted Suppressions file update
[DEBUG] Settings.getDataFile() - file: '[JAR]/../../dependency-check-data/11.0'
[DEBUG] Settings.getDataFile() - transforming filename
[DEBUG] Settings.getDataFile() - jar file: '/Users/marcelstoer/.m2/repository/org/owasp/dependency-check-utils/11.1.1'
[DEBUG] Settings.getDataFile() - returning: '/Users/marcelstoer/.m2/repository/org/owasp/dependency-check-utils/11.1.1/../../dependency-check-data/11.0'
[DEBUG] Lock file created (main) 5c1cda1259d98cb9b8e1c5f422c05a31 @ 2024-12-04 17:49:27.109
[DEBUG] Hosted Suppressions URL: https://jeremylong.github.io/DependencyCheck/suppressions/publishedSuppressions.xml
[DEBUG] Lock released (main) 5c1cda1259d98cb9b8e1c5f422c05a31 @ 2024-12-04 17:49:27.129
[INFO] Updating CISA Known Exploited Vulnerability list: https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json
[INFO] Begin database defrag
[INFO] End database defrag (80 ms)
[DEBUG] Closing database
[DEBUG] Cache cleared
[DEBUG] Connection closed
[DEBUG] Resources released
[DEBUG] Begin deregister driver
[DEBUG] End deregister driver
[INFO] Check for updates complete (52408 ms)
[DEBUG] Settings.getDataFile() - file: '[JAR]/../../dependency-check-data/11.0'
[DEBUG] Settings.getDataFile() - transforming filename
[DEBUG] Settings.getDataFile() - jar file: '/Users/marcelstoer/.m2/repository/org/owasp/dependency-check-utils/11.1.1'
[DEBUG] Settings.getDataFile() - returning: '/Users/marcelstoer/.m2/repository/org/owasp/dependency-check-utils/11.1.1/../../dependency-check-data/11.0'
[DEBUG] Data directory: /Users/marcelstoer/.m2/repository/org/owasp/dependency-check-utils/11.1.1/../../dependency-check-data/11.0
[DEBUG] Connection String: 'jdbc:h2:file:/Users/marcelstoer/.m2/repository/org/owasp/dependency-check-data/11.0/odc;AUTOCOMMIT=ON;CACHE_SIZE=65536;RETENTION_TIME=1000;MAX_COMPACT_TIME=10000;'
[DEBUG] Settings.getDataFile() - file: '[JAR]/../../dependency-check-data/11.0'
[DEBUG] Settings.getDataFile() - transforming filename
[DEBUG] Settings.getDataFile() - jar file: '/Users/marcelstoer/.m2/repository/org/owasp/dependency-check-utils/11.1.1'
[DEBUG] Settings.getDataFile() - returning: '/Users/marcelstoer/.m2/repository/org/owasp/dependency-check-utils/11.1.1/../../dependency-check-data/11.0'
[DEBUG] copying database /Users/marcelstoer/.m2/repository/org/owasp/dependency-check-utils/11.1.1/../../dependency-check-data/11.0/odc.mv.db to /var/folders/pl/nxddb61s6y7_hdjqbr2l7pfm0000gn/T/dctemp6ee4a524-2432-48ee-92a7-c9ea279940c0
[DEBUG] Setting: data.h2.directory='/var/folders/pl/nxddb61s6y7_hdjqbr2l7pfm0000gn/T/dctemp6ee4a524-2432-48ee-92a7-c9ea279940c0'
[DEBUG] Setting: data.connection_string='jdbc:h2:file:%s;AUTOCOMMIT=ON;CACHE_SIZE=65536;RETENTION_TIME=1000;MAX_COMPACT_TIME=10000;ACCESS_MODE_DATA=r'
[DEBUG] Setting: odc.autoupdate='false'
[DEBUG] Loading driver 'org.h2.Driver'
[DEBUG] Settings.getDataFile() - file: '/var/folders/pl/nxddb61s6y7_hdjqbr2l7pfm0000gn/T/dctemp6ee4a524-2432-48ee-92a7-c9ea279940c0'
[DEBUG] Data directory: /var/folders/pl/nxddb61s6y7_hdjqbr2l7pfm0000gn/T/dctemp6ee4a524-2432-48ee-92a7-c9ea279940c0
[DEBUG] Connection String: 'jdbc:h2:file:/private/var/folders/pl/nxddb61s6y7_hdjqbr2l7pfm0000gn/T/dctemp6ee4a524-2432-48ee-92a7-c9ea279940c0/odc;AUTOCOMMIT=ON;CACHE_SIZE=65536;RETENTION_TIME=1000;MAX_COMPACT_TIME=10000;ACCESS_MODE_DATA=r'
[DEBUG] Loading database connection
[DEBUG] Connection String: jdbc:h2:file:/private/var/folders/pl/nxddb61s6y7_hdjqbr2l7pfm0000gn/T/dctemp6ee4a524-2432-48ee-92a7-c9ea279940c0/odc;AUTOCOMMIT=ON;CACHE_SIZE=65536;RETENTION_TIME=1000;MAX_COMPACT_TIME=10000;ACCESS_MODE_DATA=r
[DEBUG] Database User: dcuser
[DEBUG] Database product: h2
[DEBUG] DC Schema: 5.5
[DEBUG] DB Schema: 5.5
[DEBUG] Lock released (main) 37afd16e5985ac7aa1e1eb21ae5c0d43 @ 2024-12-04 17:49:27.498
[ERROR] Unable to continue dependency-check analysis.
[DEBUG] 
org.owasp.dependencycheck.exception.NoDataException: No documents exist
    at org.owasp.dependencycheck.Engine.ensureDataExists (Engine.java:1163)
    at org.owasp.dependencycheck.Engine.analyzeDependencies (Engine.java:641)
    at org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.runCheck (BaseDependencyCheckMojo.java:1959)
    at org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.execute (BaseDependencyCheckMojo.java:1157)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:77)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:568)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)
[DEBUG] Closing database
[DEBUG] Cache cleared
[DEBUG] Connection closed
[DEBUG] Resources released
[DEBUG] Begin deregister driver
[DEBUG] End deregister driver
[INFO] Element event queue destroyed: org.apache.commons.jcs3.engine.control.event.ElementEventQueue@6e53bb4f

To Reproduce

  • Remove any potentially cached data directory e.g. ~/.m2/repository/org/owasp/dependency-check-data/11.0
  • Take any valid Maven project and run this: mvn -e org.owasp:dependency-check-maven:11.1.1:aggregate -DnvdDatafeedUrl="https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-{0}.json.gz"

Expected behavior
The plugin should initialize a new database in the 11.0 directory. It should contain all CVEs pulled in from the data feed JSON files.

Additional context

  • We can consistently reproduce this both on Mac and Linux.
  • Please see the attached 11.0.zip. It contains the entire 11.0 directory including the more or less empty database.
@marcelstoer marcelstoer added the bug label Dec 4, 2024
@marcelstoer
Copy link
Contributor Author

@aikebah @jeremylong I'd be happy to help with debugging but at this point I wouldn't know where to start.

@aikebah
Copy link
Collaborator

aikebah commented Dec 5, 2024

@marcelstoer git feel: cveb.in is not using the correct file format (old style NVD datafeed JSON, where ODC requires a per-year file with API v2 json formatting (e.g from the vulnz-cli mirroring as documented on the ODC documentation))
I think it would we reasonable to expect a clearer error reported on the dowloaded files for such abuse-cases though.

@marcelstoer
Copy link
Contributor Author

marcelstoer commented Dec 5, 2024 •
edited
Loading

Oh...oh.... I went back and checked https://jeremylong.github.io/DependencyCheck/dependency-check-maven/configuration.html -> nvdDatafeedUrl

The URL for the NVD API Data feed that can be generated using https://github.com/jeremylong/Open-Vulnerability-Project/tree/main/vulnz#caching-the-nvd-cve-data - example value https://internal.server/cache/nvdcve-{0}.json.gz

So, the files on that mirror are "per year" but the "1.1" in nvdcve-1.1-{0}.json.gz indicates that the format is <v2? The only feed files currently supported by ODC are those produced by the vulnz project? Do you have an explanation why updating an existing database from those mirror feed files works, but initializing a new one doesn't?

Wild idea: since the NVD API is still unavailable, could I translate the mirror feed files to the new format?

I'm still a bit confused, sorry. (How) is this related to #7211 (comment)? It says

Many users simply utilize https://cveb.in/

Doesn't that indicate that using cveb.in is expected to work?

@jeremylong
Copy link
Owner

My mistake - from earlier posts it seemed like the cveb.in worked. Apparently it does not. There is a cache located at https://dependency-check.github.io/DependencyCheck_Builder/nvd_cache/

This is updated nightly and is in the format required by ODC.

@marcelstoer
Copy link
Contributor Author

I still don't fully understand what's going on with the different feed files but I now switched caches as an interim fix. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@marcelstoer @jeremylong @aikebah