NVD API request failures #7178
Comments
|
Found the same at version 10.0.2 2024-11-22T01:49:43.0836082Z [ERROR] Error updating the NVD Data |
|
I got the same issues with AZ pipelines. |
|
Yeah, seems they are having problems. As normal, probably nothing this project can do other than wait. Root cause are persistent
|
|
+1 with Version 11.1.0 |
|
Gee folks, can you please leave (the maintainers of) this project alone with such issues? HTTP 503 means "unavailable", literally nothing this project can do. Besides, the top 2 results when searching for "nvd api 503" in your favorite search engine are previous issues with this project. --> issue should be closed (As for the root cause, it appears to me as if NIST had bulk-updated a ton of CVEs in their database. ODC reports that "NVD API has 245,142 records in this update" even though we update our local database very frequently. No wonder their API gets flooded with requests...) |
|
The actual issue is #6535, raised the last time this happened. |
|
Totally agree with @marcelstoer. There's no issue the Dependency-Check project could resolve in this situation. The software reacts correctly by retrying and reporting the 503 errors. As the NVD API is unavailable every now and then I decoupled updating and analyzing dependencies in my builds some time ago. Documentation is available for Caching the database, especially useful for multiple jobs/pipelines running on the same build server, as well as Using a central database, which might be more appropriate if you build on different servers or e.g. on Kubernetes pods (via Gitlab CI/CD pipelines and the like). |
|
Easiest way to avoid this issue is just to use the open-vulnerability-data-mirror docker container. Then configure ODC to use the mirror. |
|
But isn't the mirror only good for that point-in-time and thus would need to be regularly updated? We currently have a worker that runs daily and rebuilds the data directory, with the CI jobs downloading the cache and running with If we are already doing that, I assume there is no benefit to mixing in the NVD data mirror since the job that builds the cache would likely be the same job building the mirror and they'd both hit any API limitations... or is there a meaningful way to combine these two options? |
|
|
@cbertoldi I understand that. We are already using an outdated version by caching the whole data directory and using
I would assume since we are rebuilding the cache on a regular schedule that there's no benefit to using the mirror, since we'd have to rebuild that anyways. |
|
I run my vulnerability scans on GitHub Actions and use its caching mechanisms. A full db redownload is required when the local db is invalidated, for whatever reason:
When NIST feeds are struggling/erroring, it might make sense to bump the nvd max retry count up. More retries might allow the db to actually be fully downloaded and cached and would avoid the full db refetch attempt again on the next run. This would mean much less traffic to NIST for subsequent runs, benefiting everyone. |
Unfortunately, that's not how it works. When the NVD is screwed like this you want fewer retries, so you don't sit there hammering their servers for hours until dependency-check finally gives up and fails anyway. What is needed is the ability for it to continue analysis even if an update fails. |
|
@OrangeDog, thanks for responding. It is a trade-off, no? With low retries:
With elevated retries:
Once we get that db downloaded, we are only hitting NVD for small deltas. I agree that #6535 would be great, if practical, but we don't have that yet. |
|
Where are you getting these numbers from? Yesterday every run fails after downloading <1% of db, regardless of how many retries. All elevated retires does is make it take longer for you, and make the problem worse for everyone else,. |
|
FWIW it took us 2.5 hours, with retries set to 20 and the delay bumped to 6000 (using an api key) to finally update our cache due to the NVD bulk updates. Everything is smooth sailing now, updates are practically instantaneous again |
|
@OrangeDog when you say:
I think you are right for the initial fetch of the db, but if elevated retries allow the db to be fully downloaded (which they did for me), then subsequent runs will only fetch the necessary very small delta, which is much less ongoing stress on the NIST servers. Maybe not so clear cut?
I was getting much further in than that before failing. My default delay is 5s. Maybe that had something to do with it. Or maybe I just got "lucky". Or maybe the servers were a little healthier when I tried. |
|
I totally agree that erroring NIST servers is a NIST issue and not a DependencyCheck issue. But given that NIST servers do sometimes get into a bad state, would it make sense to explore more aggressive delay algorithms? Perhaps some variant of an exponential delay algorithm? An exponential delay could reduce stress on NIST servers and maybe allow more successful NIST db downloads? If the idea is interesting and not already been explored/rejected, I could open an issue to work out details. |
|
For people facing this issue - please see @pinkfloydx33's post above: #7178 (comment) |
|
which settings specifically did you update @pinkfloydx33? I've tried: However the build task in ADO still fails after 30 minutes with a 504:
|
|
That looks like what I did, I also have an NVD api key though. I'll note that it did fail similarly the first time I tried it. But I did it again. You'll also have to uodate the job's timeout which defaults to an hour, assuming it's going to take as long as it took ours. |
|
@pinkfloydx33 |
|
@stephenruda if you are using Actions for ODC - have you considered just using https://github.com/dependency-check/Dependency-Check_Action - it uses an image that is rebuilt nightly and has an updated database (although the data is slightly stale due to ongoing issues - but should be fixed soon). Otherwise, your update process should just run daily. Unless the NVD makes a mass update (which we are dealing with now) the update will only download a small json file containing the modified entries for the last 7 days. |
|
Hello, ended up here, not sure if it's the right place for my question. However, it takes forever to complete. Here's GitHub output:
The first time I ran it (without an API key) it took 4 hours, then I set up an NVD API key and it's still taking ages (40 minutes and still running). Is it normal? It was much faster when I used version 8. Is it like that now? Is there anything we could do to make it faster? |
|
|
You can create your own data feed using https://github.com/jeremylong/Open-Vulnerability-Project/tree/main/vulnz#docker-image |
|
thank you for feedback |
hello @jeremylong, |
Using this uri the download takes less than 1 minute 👍 |
|
@AkosHosszu yes - a cache of the data directory can be shared. |
I tried using the mirror with the Gradle plugin: I end up getting |
I used https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-recent.json.gz |
|
@saugion using |
I am running with maven and v11.1.0 but also get the |
|
@MatthewCooper-NHM the argument may need quoting or escaping for your shell, due to the |
Thank you @OrangeDog for responding - I appreciate the help! I tried that but unfortunately no change. I also tried this in my <plugin>
<groupId>org.owasp</groupId>
<artifactId>dependency-check-maven</artifactId>
<version>11.1.0</version>
<configuration>
<failBuildOnCVSS>0</failBuildOnCVSS>
<suppressionFiles>
<suppressionFile>src/main/resources/dependency-check-suppressions.xml</suppressionFile>
</suppressionFiles>
<nvdApiKey>${nvdApiKey}</nvdApiKey>
<nvdDatafeedUrl>https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-{0}.json.gz</nvdDatafeedUrl>
</configuration>
<executions>
<execution>
<goals>
<goal>check</goal>
</goals>
</execution>
</executions>
</plugin>I noticed that the |
|
Apparently |
@jeremylong would we utilise this by setting the |
|
@dwalba That's how I've got it working, e.g.:
That or (if you're using maven) adding it as below: |
|
Just set Keeps telling me |
|
The |
|
Closing this issue as the original problem has been resolved - the mass update at the NVD. |
|
If this problem appears again, I'd like to keep this as a workaround. So this is my build.gradle: Is there anything I have to modify to make |
and others
Describe the bug
A clear and concise description of what the bug is.
When starting the plugin, there are numerous messages like this:
[WARNING] NVD API request failures are occurring; retrying request for the 5 time
It does appear to download some updates, but there are many failures and it eventually crashes with the following stack trace.
[ERROR] Error updating the NVD Data
org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data
at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi (NvdApiDataSource.java:392)
at org.owasp.dependencycheck.data.update.NvdApiDataSource.update (NvdApiDataSource.java:116)
at org.owasp.dependencycheck.Engine.doUpdates (Engine.java:906)
at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase (Engine.java:711)
at org.owasp.dependencycheck.Engine.analyzeDependencies (Engine.java:637)
at org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.runCheck (BaseDependencyCheckMojo.java:1967)
at org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.execute (BaseDependencyCheckMojo.java:1150)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:103)
at java.lang.reflect.Method.invoke (Method.java:580)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Caused by: io.github.jeremylong.openvulnerability.client.nvd.NvdApiException: NVD Returned Status Code: 503
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient._next (NvdCveClient.java:412)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next (NvdCveClient.java:331)
at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi (NvdApiDataSource.java:352)
at org.owasp.dependencycheck.data.update.NvdApiDataSource.update (NvdApiDataSource.java:116)
at org.owasp.dependencycheck.Engine.doUpdates (Engine.java:906)
at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase (Engine.java:711)
at org.owasp.dependencycheck.Engine.analyzeDependencies (Engine.java:637)
at org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.runCheck (BaseDependencyCheckMojo.java:1967)
at org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.execute (BaseDependencyCheckMojo.java:1150)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:103)
at java.lang.reflect.Method.invoke (Method.java:580)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Version of dependency-check used
Maven plugin version: 11.1.0
Additional context
I noticed that NIST has recently changed their API. Perhaps this has broken the dependency checker?
The text was updated successfully, but these errors were encountered: