IQ Success Metrics

Overview

Nexus IQ Server has a number of REST APIs that allow you to automate certain tasks as well as quickly retrieve IQ server data. One of those APIs is the Success Metrics Data API which collects all the violations and other measurements and shares them as counters inside a JSON dictionary. In order to better capture the results, we have developed a Python script to collect, aggregate and process the counters into outcome-based metrics. We can use these outcome-based metrics to measure progression toward your PDOs.

Understanding the Python script

Though the source code can be modified to suit your particular needs if necessary, the following is an explaination of the script in its current form.

The script is actually two different files: success_metrics.py and reports.py

success_metrics.py makes the API calls according to the command-line parameters and it will process the counters to generate the more relevant outcome-based Success Metrics, returning them as a JSON dictionary called successmetrics.json.

reports.py consumes the JSON file generated by success_metrics.py and produces different types of reports and graphs depending on the Primary Desired Outcome (PDO). The main output will be a successmetrics.pdf report containing graphs and data relevant to the chosen PDO. Additionally, all graphs are also saved to individual .png files for further re-use in presentations.

Usage

First we create a temporary folder to store the outputs:

mkdir /tmp/output

Then we pull the latest docker image from Docker Hub:

docker pull cmorenoserrano/iq_success_metrics:latest

Then we use the following docker command to generate the JSON file (switch -s stands for scope and it is the number of weeks that we want data from, in this case the past 50 weeks) and the desired PDF reports (-r stands for reports and will generate an executive report and an app-level detailed table report):

docker run --name iq_success_metrics --rm -it -v /tmp/output:/usr/src/app/output cmorenoserrano/iq_success_metrics:latest success_metrics.py -u 'http://<insert your IQ Server IP here>:8070' -a user:password -s 50 -r

If you want to generate the executive and table reports just for security violations, use the -rs switch instead of just -r. If you are only interested in licence obligations, you can generate such executive and table reports by using the -rl switch instead.

Please, ignore any warnings.

The Docker Hub repository for the docker image is here: https://hub.docker.com/repository/docker/cmorenoserrano/iq_success_metrics

For Windows users

The main difference for Windows users is in the path to your local folder to store the outputs. Below are the equivalent commands:

mkdir c:\temp

docker pull cmorenoserrano/iq_success_metrics:latest

docker run --name iq_success_metrics --rm -it -v c:\temp\:/usr/src/app/output cmorenoserrano/iq_success_metrics:latest success_metrics.py -u 'http://<insert your IQ Server IP here>:8070' -a user:password -s 50 -r

Explaining the Success Metrics Data API

The Success Metrics Data API returns policy evaluation, violation and remediation data, aggregated monthly or weekly. The API uses the following common language in its return values:

API Legend:

• Threat Level Low - Policy Threat Level 1
• Threat Level Moderate - Policy Threat Level 2 - 3
• Threat Level Severe - Policy Threat Level 4 - 7
• Threat Level Critical - Policy Threat Level 8 - 10
• Security Violation - Violation for which the policy constraint was on the Security Vulnerability Severity Score
• License Violation - Violation for which the policy constraint was on the License or License Threat Group
• Quality Violation - Violation for which the policy constraint was on the Age or Relative Popularity of a component
• Other Violation - Violation for which the policy constraint was something other than a Security, License, or Quality constraint, such as a label

Here are the actual values returned from the REST call:

Dimensional Data

• applicationId - Unique ID per application, assigned by IQ server
• applicationPublicId - ID, assigned by customer
• applicationName - Name, assigned by customer
• organizationId - Unique Organization ID, assigned by IQ server
• organizationName - Organization name, assigned by customer
• timePeriodStart - Start time period of aggregration of the data (usually weekly or monthly from this date). In ISO 8601 date format.

Scan Data

• evaluationCount - Number of evaluations or scans for a particular application

Violation Data

• Mean Time To Resolution (MTTR) in milliseconds for Low (Threat Level violation 1 ), Moderate (Threat Level violations 2-3), Severe (Threat Level violations 4-7), or Critical (Threat Level violations 8-10)
• mttrLowThreat
• mttrModerateThreat
• mttrSevereThreat
• mttrCriticalThreat

Number of newly discovered Security/License/Quality/Other violations during the time period for Low/Moderate/Severe/Critical threat levels (Note: does not include violations that existed in previous time periods)

• discoveredCountSecurityLow
• discoveredCountSecurityModerate
• discoveredCountSecuritySevere
• discoveredCountSecurityCritical
• discoveredCountLicenseLow
• discoveredCountLicenseModerate
• discoveredCountLicenseSevere
• discoveredCountLicenseCritical
• discoveredCountQualityLow
• discoveredCountQualityModerate
• discoveredCountQualitySevere
• discoveredCountQualityCritical
• discoveredCountOtherLow
• discoveredCountOtherModerate
• discoveredCountOtherSevere
• discoveredCountOtherCritical

Number of "fixed" Security/License/Quality/Other violations during the time period for Low/Moderate/Severe/Critical threat levels (Note: fixed is defined as a specific violation that existed in the immediately prior scan and now no longer appears in the subsequent scan)

• fixedCountSecurityLow
• fixedCountSecurityModerate
• fixedCountSecuritySevere
• fixedCountSecurityCritical
• fixedCountLicenseLow
• fixedCountLicenseModerate
• fixedCountLicenseSevere
• fixedCountLicenseCritical
• fixedCountQualityLow
• fixedCountQualityModerate
• fixedCountQualitySevere
• fixedCountQualityCritical
• fixedCountOtherLow
• fixedCountOtherModerate
• fixedCountOtherSevere
• fixedCountOtherCritical

Number of waived Security/License/Quality/Other violations during the time period for Low/Moderate/Severe/Critical threat levels.

• waivedCountSecurityLow
• waivedCountSecurityModerate
• waivedCountSecuritySevere
• waivedCountSecurityCritical
• waivedCountLicenseLow
• waivedCountLicenseModerate
• waivedCountLicenseSevere
• waivedCountLicenseCritical
• waivedCountQualityLow
• waivedCountQualityModerate
• waivedCountQualitySevere
• waivedCountQualityCritical
• waivedCountOtherLow
• waivedCountOtherModerate
• waivedCountOtherSevere
• waivedCountOtherCritical

Number of "open" Security/License/Quality/Other violations at the end of the time period for Low/Moderate/Severe/Critical threat levels.

Open counts accumulate from previous time periods (weeks/months) and constitute the technical debt backlog to fix/remediate. For example, if you discovered 10 Security Critical violations each week for 3 weeks (total of 30 violations) and you fixed and/or waived a total of 10 Security Critical violations at the end of those 3 weeks, the openCountAtTimePeriodEndSecurityCritical counter would show 20 (Security Critical open violations).

• openCountAtTimePeriodEndSecurityLow
• openCountAtTimePeriodEndSecurityModerate
• openCountAtTimePeriodEndSecuritySevere
• openCountAtTimePeriodEndSecurityCritical
• openCountAtTimePeriodEndLicenseLow
• openCountAtTimePeriodEndLicenseModerate
• openCountAtTimePeriodEndLicenseSevere
• openCountAtTimePeriodEndLicenseCritical
• openCountAtTimePeriodEndQualityLow
• openCountAtTimePeriodEndQualityModerate
• openCountAtTimePeriodEndQualitySevere
• openCountAtTimePeriodEndQualityCritical
• openCountAtTimePeriodEndOtherLow
• openCountAtTimePeriodEndOtherModerate
• openCountAtTimePeriodEndOtherSevere
• openCountAtTimePeriodEndOtherCritical

Understanding the successmetrics.json file

The successmetrics.json file is currently composed of four dictionaries:

• summary: this is the overall summary that collates and aggregates all the data together, giving the global view. This dictionary is the main one used for generating the global reports.
• apps: this is a list of all the applications within scope. It contains the raw data coming from the API call (aggregations) and also a summary view for that app, a licences view and a security view.
• licences: this is the same as summary but exclusively for licence violations.
• security: this is the same as summary but exclusively for security violations.

NOTE: adding the licences and security data together will not produce the overall summary data because there are also quality and other types of violations that are included in summary but not in licences or security.

Understanding summary

If we go inside summary we can see the following:

• appNames: this is a list of all the application names within scope.
• orgNames: this is a list of all the organization names within scope. The entries match one-for-one each one of the applications, so there will be duplicate organization names.
• weeks: this is the range of weeks in scope, in ISO format (week number). This was selected when running the success_metrics.py script and was set by default to 6 weeks, so if we were in the middle of week 38, we would request the IQ server for weeks 32, 33, 34, 35, 36 and 37 (the past six fully completed weeks).
• timePeriodStart: this is a list of the weeks in scope in normal date format instead of ISO format.
• appNumberScan: this is a list of the number of applications that have been scanned in each of the weeks in scope.
• appOnboard: this is a list of the number of applications onboarded in the IQ server in each of the weeks in scope.
• weeklyScans: this is a list of the total number of scans per week in scope.
• mttrLowThreat: this is a list of the overall MTTR (Mean Time To Resolution) measured in days for all Low Threat vulnerabilities per week.
• mttrModerateThreat: this is a list of the overall MTTR (Mean Time To Resolution) measured in days for all Moderate Threat vulnerabilities per week.
• mttrSevereThreat: this is a list of the overall MTTR (Mean Time To Resolution) measured in days for all Severe Threat vulnerabilities per week.
• mttrCriticalThreat: this is a list of the overall MTTR (Mean Time To Resolution) measured in days for all Critical Threat vulnerabilities per week.
• discoveredCounts: this is a dictionary containing all the combined (Security, License, Quality & Other) discovered vulnerabilities for each threat level. LIST is the aggregation of all threat level violations where each element of the list is one of the applications in scope. TOTAL is a list aggregating all threat level violations for all applications in scope combined where each element of the list is one of the weeks in scope.
• fixedCounts: this is a dictionary containing all the combined (Security, License, Quality & Other) fixed vulnerabilities for each threat level. LIST is the aggregation of all threat level violations where each element of the list is one of the applications in scope. TOTAL is a list aggregating all threat level violations for all applications in scope combined where each element of the list is one of the weeks in scope.
• waivedCounts: this is a dictionary containing all the combined (Security, License, Quality & Other) waived vulnerabilities for each threat level. LIST is the aggregation of all threat level violations where each element of the list is one of the applications in scope. TOTAL is a list aggregating all threat level violations for all applications in scope combined where each element of the list is one of the weeks in scope.
• openCountsAtTimePeriodEnd: this is a dictionary containing all the combined (Security, License, Quality & Other) vulnerabilities for each threat level that have not yet been fixed or waived (this is the current backlog or risk exposure). LIST is the aggregation of all threat level violations where each element of the list is one of the applications in scope. TOTAL is a list aggregating all threat level violations for all applications in scope combined where each element of the list is one of the weeks in scope.
• riskRatioCritical: this is a list calculating the Critical risk ratio (number of Critical vulnerabilities divided by the total number of applications onboarded) for each week in scope.
• riskRatioSevere: this is a list calculating the Severe risk ratio (number of Severe vulnerabilities divided by the total number of applications onboarded) for each week in scope.
• riskRatioModerate: this is a list calculating the Moderate risk ratio (number of Critical vulnerabilities divided by the total number of applications onboarded) for each week in scope.
• riskRatioLow: this is a list calculating the Low risk ratio (number of Critical vulnerabilities divided by the total number of applications onboarded) for each week in scope.

Understanding apps

If we go inside apps, we can see that first element in the list (number 0), has an applicationId, applicationPublicId, applicationName, organizationId, organizationName to be able to identify this particular application within a specific organization.

Then we can see the following:

• aggregations: this is the raw data collected by the API call. All the values inside aggregations have been explained in section 2. Explaining the Success Metrics Data API
• summary: this is the summary of all the outcome-based success metrics resulting from processing the raw data from the API call. More information later below.
• licences: this is the same as summary but exclusively for licence violations (for this particular app)
• security: this is the same as summary but exclusively for security violations (for this particular app)

Now it is time to explore the summary dictionary in more detail:

Below are each one of them explained:

• weeks: this is a list of all the weeks in ISO format that contain data. It is possible that a particular app was not scanned during one or more of the weeks in scope
• fixedRate: this is the YTD weekly rolling average (in percentage) of the Fixed Rate for Security/License/Quality/Other vulnerabilities combined, for all Low/Moderate/Severe/Critical threat levels combined for that particular app. fixedRate is calculated as fixedCounts / openCountsAtTimePeriodEnd (for the previous week) in percentage. For example if you fixed 5 Security Critical vulnerabilities in week 2 and at the end of week 1 you had left 50 open, the Fixed Rate would be 10%.
• waivedRate: this is the YTD weekly rolling average (in percentage) of the Waived Rate for Security/License/Quality/Other vulnerabilities combined, for all Low/Moderate/Severe/Critical threat levels combined for that particular app. waivedRate is calculated as waivedCounts / openCountsAtTimePeriodEnd (for the previous week) in percentage. For example if you waived 5 Security Critical vulnerabilities in week 2 and at the end of week 1 you had left 50 open, the Waived Rate would be 10%.
• dealtRate: this is the YTD weekly rolling average (in percentage) of the Dealt-with Rate for Security/License/Quality/Other vulnerabilities combined, for all Low/Moderate/Severe/Critical threat levels combined for that particular app. DealtRate is calculated as (fixedCounts + waivedCounts) / openCountsAtTimePeriodEnd (for the previous week) in percentage. For example if you fixed 5 and waived 15 Security Critical vulnerabilities in week 2 and at the end of week 1 you had left 100 open, the Dealt-with Rate would be 20% for Security Critical vulnerabilities. *FixRate: this is the overall combined Fix rate over all the weeks in scope. *WaiveRate: this is the overall combined Waive rate over all the weeks in scope. *DealtRate: this is the overall combined Dealt rate over all the weeks in scope.
• FixPercent: this is the unitary percentage (0.5 = 50%) of all dealt-with vulnerabilities that were fixed for that particular app.
• WaiPercent : this is the unitary percentage (0.5 = 50%) of all dealt-with vulnerabilities that were waived for that particular app. Please note that FixPercent + WaiPercent = 1
• evaluationCount: this is the number of evaluations or scans that were performed on that particular app. avg provides the overall average number of scans over the weeks in scope and rng provides the isolated scans/week.

The following metrics are dictionaries and inside them, they have the avg (average value) and rng (range, or isolated values per week of data) parameters. Some of them go into more detail, with a selection of TOTAL, SECURITY, LICENSE, QUALITY and OTHER violation types and LOW, MODERATE, SEVERE and CRITICAL threat levels.

• mttrLowThreat: this is the Mean Time To Resolution (measured in days instead of milliseconds) for Low threat level violations for that particular app. avg provides the overall average of the weeks in scope and rng provides the isolated MTTR values.
• mttrModerateThreat: this is the Mean Time To Resolution (measured in days instead of milliseconds) for Moderate threat level violations for that particular app. avg provides the overall average of the weeks in scope and rng provides the isolated MTTR values.
• mttrSevereThreat: this is the Mean Time To Resolution (measured in days instead of milliseconds) for Severe threat level violations for that particular app. avg provides the overall average of the weeks in scope and rng provides the isolated MTTR values.
• mttrCriticalThreat: this is the Mean Time To Resolution (measured in days instead of milliseconds) for Critical threat level violations for that particular app. avg provides the overall average of the weeks in scope and rng provides the isolated MTTR values.
• discoveredCounts: this is the number of discovered vulnerabilities of a particular type (TOTAL, SECURITY, LICENSE, QUALITY, OTHER), of a particular threat level (TOTAL, LOW, MODERATE, SEVERE, CRITICAL) for that particular app. avg provides the overall average number of vulnerabilities of that type and threat level and rng provides the isolated number per week.
• fixedCounts: this is the number of fixed vulnerabilities of a particular type (TOTAL, SECURITY, LICENSE, QUALITY, OTHER), of a particular threat level (TOTAL, LOW, MODERATE, SEVERE, CRITICAL) for that particular app. avg provides the overall average number of vulnerabilities of that type and threat level and rng provides the isolated number per week.
• waivedCounts: this is the number of waived vulnerabilities of a particular type (TOTAL, SECURITY, LICENSE, QUALITY, OTHER), of a particular threat level (TOTAL, LOW, MODERATE, SEVERE, CRITICAL) for that particular app. avg provides the overall average number of vulnerabilities of that type and threat level and rng provides the isolated number per week.
• openCountsAtTimePeriodEnd: this is the number of open vulnerabilities of a particular type (TOTAL, SECURITY, LICENSE, QUALITY, OTHER), of a particular threat level (TOTAL, LOW, MODERATE, SEVERE, CRITICAL) for that particular app. avg provides the overall average number of vulnerabilities of that type and threat level and rng provides the isolated number per week. Open counts accumulate from previous time periods (weeks/months) and constitute the technical debt backlog to fix/remediate. For example, if you discovered 10 Security Critical violations each week for 3 weeks (total of 30 violations) and you fixed and/or waived a total of 10 Security Critical violations at the end of those 3 weeks, the openCountAtTimePeriodEndSecurityCritical counter would show 20 (Security Critical open violations).

Understanding licences

The structure is identical to summary with data being exclusive to licence violations.

Understanding security

The structure is identical to summary with data being exclusive to security violations.

Building the Docker Image

docker build -f Dockerfile -t iq_success_metrics:latest .

To keep the docker container running after its main process finishes, instead run the container with this command:

docker run -dit -p 5000:5000 iq-success-metrics:latest

The above command should print a container id, similar to bc1d9d006052b433761a0b03453e0f5c069bcb3ebb2af0a6d9ccbaa3278ddf83. You can attach to the running container using that id with a command like this:

docker attach bc1d9d006052b433761a0b03453e0f5c069bcb3ebb2af0a6d9ccbaa3278ddf83

NOTE: optionally specify docker build --build-arg ALT_DOCKER_REGISTRY=host.docker.internal:19443 --build-arg ALT_PYPI_REGISTRY=http://host.docker.internal:8083/nexus/repository/pypi-python.org-proxy/simple -t iq-success-metrics:latest . to download images from a location other than docker hub

Contributing

If you as well want to speed up the pace of software development by working on this project, jump on in! Before you start work, create a new issue, or comment on an existing issue, to let others know you are!

The Fine Print

It is worth noting that this is NOT SUPPORTED by Sonatype, and is a contribution of ours to the open source community (read: you!)

Don't worry, using this community item does not "void your warranty". In a worst case scenario, you may be asked by the Sonatype Support team to remove the community item in order to determine the root cause of any issues.

Remember:

• Use this contribution at the risk tolerance that you have
• Do NOT file Sonatype support tickets related to iq-success-metrics
• DO file issues here on GitHub, so that the community can pitch in

Phew, that was easier than I thought. Last but not least of all:

Have fun creating and using this plugin and the Nexus platform, we are glad to have you here!

Getting help

Looking to contribute to our code but need some help? There's a few ways to get information:

• Chat with us on Gitter