Initial (internal) validation on a small sample of CVEs

[copernico]

• Each partner to pick 5 random "statements" from: https://github.com/SAP/project-kb/blob/vulnerability-data/statements/
  • SAP: 2020
  • SLAB: 2019
  • FEA: 2018
• Examples of how to report on the "experience": Validate Prospector on existing vulnerability data #207

[chicxurug]

CVE-2018-11777

python client/cli/main.py CVE-2018-11777 --use-nvd --repository https://github.com/apache/hive --report html

More than 1000 candidates, not useful.

Specifying a version interval (2.3.3:2.3.4, based on the advisory text):

python client/cli/main.py CVE-2018-11777 --use-nvd --repository https://github.com/apache/hive --report html --version-interval 2.3.3:2.3.4

Gives only 6 candidates all matching REF_JIRA_ISSUE, which is not particularly useful. Found the right commit based on the term "authorizer" appearing both in the advisory record and the commit message.

[copernico]

have you tried to add --advisory-keywords=authorizer?

[copernico]

It is interesting to observe that only 2 commits of the 6 candidates have the keyword "authorizer" that appears in the paths modified by the commits at hand. That criterion would have helped a lot (but we do not have it working right now).

[chicxurug]

Indeed, adding this keyword helps.

[amilankovich-slab]

CVE-2019-0191

python client/cli/main.py CVE-2019-0191 --use-nvd --repository https://github.com/apache/karaf --report html
Found 1550 candidates

If I restricted the versions according to the vulnerability-data statement:
python client/cli/main.py CVE-2019-0191 --use-nvd --repository https://github.com/apache/karaf --report html --version-interval 4.1:4.2
Found 0 candidates

The version string should be spelled out in 3 digits:
python client/cli/main.py CVE-2019-0191 --use-nvd --repository https://github.com/apache/karaf --report html --version-interval 4.1.0:4.2.10
Found 777 candidates
Saving to backend completed (status code: 422)

Searching for keyword “..” would give good results (5 after using the TOKENS_IN_COMMIT_MSG filter):
python client/cli/main.py CVE-2019-0191 --use-nvd --repository https://github.com/apache/karaf --report html --version-interval 4.1.0:4.2.10 --advisory-keywords=..

In all of the cases of the commit messages (e36a7a66fa08eb5eb253b2b0cec262ffbdef072) in the vulnerability-data statement is not even listed by prospector as candidate.

[copernico]

Have you tried with: python client/cli/main.py CVE-2019-0191 --use-nvd --repository https://github.com/apache/karaf --report html --version-interval 4.2.2:4.2.3

(note: the interval in your command seems too broad, or otherwise wrong)

[amilankovich-slab]

That was the problem indeed. With the command above, I found 47 candidates with 13 that matches prospector rules. Among those, it was easy to spot the right one (c978390a1c8e54ff5525357f2667e588e4fe7dd2), which had the parent commit (fef9a618f11a670dc040d903a4b0f9bbc9f3e9c) mentioned in the vulnerability-data statement.

[amilankovich-slab]

CVE-2019-0193
python client/cli/main.py CVE-2019-0193 --use-nvd --repository https://github.com/apache/lucene-solr --report html
Found 11827 candidates

python client/cli/main.py CVE-2019-0193 --use-nvd --repository https://github.com/apache/lucene-solr --report html --version-interval 6.6.5:6.6.6
Found 6 candidates
Fixing commit according to vulnerability-data statement was not among them.

[chicxurug]

I think every version is vulnerable before 8.2.0, so the version filter is invalid. If I add 8.1.0:8.2.0 and --advisory-keyword=dataConfig I find a fixing commit SOLR-13158: DIH: Add System property toggle for use of dataConfig param (cherry picked from commit 325824cd391c8e71f36f17d687f52344e50e9715) -> the cherry picked hash is the fix in the statement.yaml. Nonetheless, I don't get this commit as a candidate directly either...

[copernico]

Why not 8.1.1:8.2.0 instead?

[amilankovich-slab]

That restricts the number of candidates to 267, but the right commit is hard to spot, only 1 rule matched (the same as other 266 candidates): CH_REL_PATH.

[copernico]

This would work:
python client/cli/main.py CVE-2019-0193 --use-nvd --repository https://github.com/apache/lucene-solr --report html --version-interval 8.1.1:8.2.0 --advisory-keywords=DataImportHandler,dataConfigParam,DIH

[amilankovich-slab]

CVE-2019-9827
python client/cli/main.py CVE-2019-9827 --use-nvd --repository https://github.com/hawtio/hawtio --report html
Found 230 candidates
Filtering with CH_REL_PATH narrowed it down to 12 commits, among which I found the rigth one according to vulnerability-data statement.

[chicxurug]

CVE-2018-1304

python client/cli/main.py CVE-2018-1304 --use-nvd --repository https://github.com/apache/tomcat --report html

As expected gives way too candidates.

Playing around with versions did not yield to results immediately (there are 7.x, 8.x and 9.x vulnerable versions). Finally, the following worked:

python client/cli/main.py CVE-2018-1304 --use-nvd --repository https://github.com/apache/tomcat --report html --version-interval 9.0.0:9.0.5

Even though it gave me 300 candidates I spotted the commit with the term URL pattern of "", which was obviuosly the right one. No specific rule helped (maybe the security related keyword in the commit msg a bit, due to the term security constraint). Following the previous line of refinement, I applied additional keywords from the advisory text:

python client/cli/main.py CVE-2018-1304 --use-nvd --repository https://github.com/apache/tomcat --report html --version-interval 9.0.0:9.0.5 --advisory-keywords="URL pattern","context root","security constraint"

The TOKENS_IN_DIFF rule narrowed down the results to 4 (TOKENS_IN_COMMIT_MSG matched only the right commit). Another possible lead might have been the bugzille issue URL that was mentioned in the commit message (https://bz.apache.org/bugzilla/show_bug.cgi?id=62067). This URL also appeared in the content of one of the external references but that's a bit complicated already...
Textual similarity of the advisory text and the commit message would definitely help in this case (but that is something SAP is already pursuing if I am correct).

[chicxurug]

Detailed evaluation path:
Try1: python client/cli/main.py CVE-2018-1304 --use-nvd --repository https://github.com/apache/tomcat --report html --version-interval 7.0.84:7.0.85 -> 18 candidates but the fix is not there (however, the advisory text suggests that 7.0.84 is the last vulnerable version)
Try2: python client/cli/main.py CVE-2018-1304 --use-nvd --repository https://github.com/apache/tomcat --report html --version-interval 7.0.83:7.0.85 -> 47 candidates (double-checking that 7.0.84 is inclusive or not), no results
Try3: python client/cli/main.py CVE-2018-1304 --use-nvd --repository https://github.com/apache/tomcat --report html --version-interval 8.0.49:8.0.50 -> 2064 candidates, no such tags in the git repo!
Try4: python client/cli/main.py CVE-2018-1304 --use-nvd --repository https://github.com/apache/tomcat --report html --version-interval 9.0.4:9.0.5 -> 53 candidates, only 3 matches SEC_KEYWORD_IN_COMMIT_MSG -> I spotted the commit with the term URL pattern of "", which was obviuosly the right one.

[amilankovich-slab]

CVE-2019-20445
python client/cli/main.py CVE-2019-20445 --use-nvd --repository https://github.com/netty/netty --report html
Found 2874 candidates

python client/cli/main.py CVE-2019-20445 --use-nvd --repository https://github.com/netty/netty --report html --version-interval 4.1.43:4.1.44
Found 82 candidates
Applying filter VULN_MENTIONED_IN_LINKED_ISSUE results in a good fix on another branch, but not the one mentioned in vulnerability-data statement.

[amilankovich-slab]

CVE-2019-10241
python client/cli/main.py CVE-2019-10241 --use-nvd --repository https://github.com/eclipse/jetty.project --report html
Found 5321 candidates

Vulnerable: 9.2.26 and older, 9.3.25 and older, and 9.4.15 and older
python client/cli/main.py CVE-2019-10241 --use-nvd --repository https://github.com/eclipse/jetty.project --report html --version-interval 9.2.26:9.2.27
Found 0 candidates

python client/cli/main.py CVE-2019-10241 --use-nvd --repository https://github.com/eclipse/jetty.project --report html --version-interval 9.3.25:9.3.26
Found 0 candidates

python client/cli/main.py CVE-2019-10241 --use-nvd --repository https://github.com/eclipse/jetty.project --report html --version-interval 9.4.15:9.4.16
Found 0 candidates

python client/cli/main.py CVE-2019-10241 --use-nvd --repository https://github.com/eclipse/jetty.project --report html --version-interval 9.2.26:9.4.16
Found 0 candidates

The problem was that this repo uses version strings as: jetty-9.2.26.v20180806, jetty-9.4.16.v20190411.

python client/cli/main.py CVE-2019-10241 --use-nvd --repository https://github.com/eclipse/jetty.project --report html --version-interval jetty-9.4.15.v20190215:jetty-9.4.16.v20190411
Found 136 candidates
Filtering VULN_MENTIONED_IN_LINKED_ISSUE gave 5 results, among which the 2 right fixes were found.

[copernico]

Please note that we also have --tag-interval=..., but what you did in the last attempt is equivalent.

[chicxurug]

CVE-2018-14637

With the routine I already gathered so far, I already started with this:

python client/cli/main.py CVE-2018-14637 --use-nvd --repository https://github.com/keycloak/keycloak --report html --version-interval 4.5.0.Final:4.6.0.Final --advisory-keywords SAML,endpoint,assertion

77 candidates, but only two matching four different rules REF_JIRA_ISSUE and TOKEN_IN_*. From the commit message and changed files/content of keycloak/keycloak@0fe0b87 it was clear that it is the fix commit.

[chicxurug]

CVE-2018-17193

python client/cli/main.py CVE-2018-17193 --use-nvd --repository https://github.com/apache/nifi --report html --version-interval 1.7.1:1.8.0

Gives 184 candidates but only one matches the CH_REL_PATH rule (the correct commit), which recognizes that message-page.jsp file has changed in this commit (that is also quoted in the advisory text). The term X-ProxyContextPath would have been also helpful as an advisory term. The advisory text was very specific and detailed, so this was a nice issue to catch.

[chicxurug]

CVE-2018-8018

python client/cli/main.py CVE-2018-8018 --use-nvd --repository https://github.com/apache/ignite --report html --version-interval 2.5.0:2.6.0

Gives 13 candidates (versions can be extracted from the advisory text). Easy to spot the right commit saying "Client marshalling improvements" (apache/ignite@82a7b82). Nonetheless, if I apply the obvious advisory keyword of GridClientJdkMarshaller from the advisory description, I get only one single candidate, the right one (matching TOKEN_IN_DIFF and TOKEN_IN_MODIFIED_PATHS rules).