Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: add logging #123

Closed
wants to merge 5 commits into from
Closed

refactor: add logging #123

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
ca6bb7d
refactor: add logging
robincakeellis Jan 9, 2024
847b245
refactor: add logging
robincakeellis Jan 9, 2024
86bb4cb
refactor: add logging
robincakeellis Jan 9, 2024
2b5fd2e
Merge branch 'main' into robin-add-logging
robincakeellis Jan 14, 2024
21e1797
Merge branch 'main' into robin-add-logging
robincakeellis Feb 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions ...clients/src/main/java/io/github/jeremylong/openvulnerability/client/nvd/NvdCveClient.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -192,6 +192,7 @@ public class NvdCveClient implements PagedDataSource<DefCveItem> {
if (delay == 0) {
delay = apiKey == null ? 6500 : 600;
}
LOG.info("request endpoint={} threads={} delay={}", this.endpoint, threadCount, delay);
for (int i = 0; i < threadCount; i++) {
clients.add(new RateLimitedClient(maxRetryCount, delay, meter, httpClientSupplier));
}
Expand Down Expand Up @@ -257,7 +258,7 @@ private Future<RateLimitedCall> callApi(int clientIndex, int startIndex) throws
}
builder.addHeader("User-Agent", "vulnz");
URI uri = uriBuilder.build();
LOG.debug("requesting URI: {}", uri.toString());
LOG.info("requesting URI: {}", uri.toString());
final SimpleHttpRequest request = builder.setUri(uri).build();
return clients.get(clientIndex).execute(request, clientIndex, startIndex);
} catch (URISyntaxException e) {
Expand All @@ -281,7 +282,7 @@ public void close() {
try {
client.close();
} catch (Exception ex) {
LOG.debug("Error closing client during `close`", ex);
LOG.warn("Error closing client during `close`", ex);
}
}
clients = null;
Expand Down Expand Up @@ -352,8 +353,8 @@ public Collection<DefCveItem> next() {
return current.getVulnerabilities();
} else {
lastStatusCode = response.getCode();
LOG.debug("Status Code: {}", lastStatusCode);
LOG.debug("Response: {}", response.getBodyText());
LOG.warn("Status Code: {}", lastStatusCode);
LOG.warn("Response: {}", response.getBodyText());
throw new NvdApiException("NVD Returned Status Code: " + lastStatusCode);
}
}
Expand Down
6 changes: 4 additions & 2 deletions vulnz/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,15 @@

The cli is a spring-boot command line tool built with picocli. The example
below does run the setup - which creates both the `vulnz` symlink (in `/usr/local/bin`)
and a completion script. If using zsh, the completion will be added to
and a completion script. If using zsh, the completion will be added to
`/etc/bash_completion.d` or `/usr/local/etc/bash_completion.d` (depending
on if they exist); see [permanently installing completion](https://picocli.info/autocomplete.html#_installing_completion_scripts_permanently_in_bashzsh)
for more details. We may add a brew formula in the future.

After running `install` you may need to restart your shell for the completion to work.

```bash
./gradlew vulnz:build
./gradlew vulnz:build -Pversion=5.1.1
cd vulnz/build/libs
./vulnz-5.1.1.jar install
vulnz cve --cveId CVE-2021-44228 --prettyPrint
Expand Down Expand Up @@ -88,6 +88,8 @@ via a daily schedule to keep the cached data current:
vulnz cve --cache --directory ./cache
```

Download time might be reduced for the first cache by adding `--threads N` (where N is a number) but remote rate limits will still apply. It may be wise to specify the full path of the cache directory depending on your configuration.

Alternatively, without using the above install command:

```bash
Expand Down
26 changes: 22 additions & 4 deletions vulnz/src/main/java/io/github/jeremylong/vulnz/cli/commands/CveCommand.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,11 @@ public class CveCommand extends AbstractNvdCommand {
* Hex code characters used in getHex.
*/
private static final String HEXES = "0123456789abcdef";
/**
* Earliest year to acquire.
*/
private static final int FIRST_YEAR = 2002;

@CommandLine.ArgGroup(exclusive = true)
ConfigGroup configGroup;

Expand Down Expand Up @@ -118,6 +123,8 @@ public Integer timedCall() throws Exception {
String apiKey = getApiKey();
if (apiKey == null) {
LOG.info("NVD_API_KEY not found. Supply an API key for more generous rate limits");
} else {
LOG.info("NVD_API_KEY found");
}
NvdCveClientBuilder builder = NvdCveClientBuilder.aNvdCveApi().withApiKey(getApiKey());
if (getDelay() > 0) {
Expand Down Expand Up @@ -248,10 +255,11 @@ private Integer processRequest(NvdCveClientBuilder builder, CacheProperties prop
cves.put("modified", new HashMap<>());
final String prefix = properties.get("prefix", "nvdcve-");
// load existing cached files
for (int year = 2002; year <= Year.now().getValue(); year++) {
for (int year = FIRST_YEAR; year <= Year.now().getValue(); year++) {
File file = new File(properties.getDirectory(), prefix + year + ".json.gz");
cves.put(Integer.toString(year), new HashMap<>());
if (file.isFile()) {
LOG.info("cache found {}", file.getAbsolutePath());
CveApiJson20 data;
try (FileInputStream fileInputStream = new FileInputStream(file);
GZIPInputStream gzipInputStream = new GZIPInputStream(fileInputStream);) {
Expand All @@ -260,32 +268,41 @@ private Integer processRequest(NvdCveClientBuilder builder, CacheProperties prop
throw new CacheException("Unable to read cached data: " + file, exception);
}
collectCves(cves, data.getVulnerabilities());
} else {
LOG.info("no existing cache {}", file.getAbsolutePath());
}
}

final long requestStartTime = System.currentTimeMillis();
ZonedDateTime lastModified = null;
// retrieve from NVD API
int receivedSoFar = 0;
try (NvdCveClient api = builder.build()) {
while (api.hasNext()) {
Collection<DefCveItem> data = api.next();
receivedSoFar += data.size();
LOG.info("vulnerabilities received so far {}", receivedSoFar);
collectCves(cves, data);
lastModified = api.getLastUpdated();
}
} catch (Exception ex) {
LOG.debug("\nERROR", ex);
LOG.error("\nERROR", ex);
throw new CacheException("Unable to complete NVD cache update due to error: " + ex.getMessage());
}
if (lastModified != null) {
properties.set("lastModifiedDate", lastModified);
}
LOG.info("requests took ~{} seconds", (System.currentTimeMillis() - requestStartTime) / 1000L);

// write cache
// todo - get format and version from API
final String format = "NVD_CVE";
final String version = "2.0";

for (Map.Entry<String, HashMap<String, DefCveItem>> entry : cves.entrySet()) {
File file = new File(properties.getDirectory(), prefix + entry.getKey() + ".json.gz");
File meta = new File(properties.getDirectory(), prefix + entry.getKey() + ".meta");
List<DefCveItem> vulnerabilities = new ArrayList(entry.getValue().values());
List<DefCveItem> vulnerabilities = new ArrayList<>(entry.getValue().values());
LOG.info("CVEs {} vulnerabilities {}", entry.getKey(), vulnerabilities.size());
vulnerabilities.sort((v1, v2) -> {
return v1.getCve().getId().compareTo(v2.getCve().getId());
});
Expand Down Expand Up @@ -331,6 +348,7 @@ private Integer processRequest(NvdCveClientBuilder builder, CacheProperties prop
throw new CacheException("Unable to write cached meta-data: " + file, ex);
}
}
LOG.info("complete");
return 0;
}

Expand Down