fix(ci): NVD data and test issues #1737

dcaravel · 2024-12-10T02:36:38Z

Description

Fixes multiple CI issues

Removes CVSSv3 metadata from various test cases for Microsoft images that no longer have data avail in the NVD legacy feeds.
Adjusts sanity test logic to skip validating CVSSv3 fields when not provided by test case.
- Also fail the test when/if the CVSSv3 scores return
Increase test timeout to avoid panic: test timed out after 20m0s (such as this one).
~~Removes CVE-2021-26291 from the ose-jenkins image test case, it is no longer appearing in scan results (ref)~~
- Added it back, results flapped, see below

Flapping results for CVE-2021-26291

CVE-2021-26291 Missing

$ rctl image scan -f --image=quay.io/rhacs-eng/qa:ose-jenkins 2>/dev/null | jq -r '.scan.components[] | select(.name == "jenkins-2-plugins").vulns[].cve' 
RHSA-2022:6531
RHSA-2022:7865
RHSA-2022:9098
RHSA-2023:0560
RHSA-2023:0697
RHSA-2023:1655
RHSA-2023:1866
RHSA-2023:3362
RHSA-2023:3625

CVE-2021-26291 Back

Example run

$ rctl image scan -f --image=quay.io/rhacs-eng/qa:ose-jenkins 2>/dev/null | jq -r '.scan.components[] | select(.name == "jenkins-2-plugins").vulns[].cve' 
CVE-2019-1003011
CVE-2019-1003012
CVE-2019-1003013
CVE-2019-1003014
CVE-2019-1010241
CVE-2021-0341
CVE-2021-22573
CVE-2021-26291
CVE-2021-27568
CVE-2021-37533
CVE-2021-4178
CVE-2021-46877
CVE-2022-20613
CVE-2022-20614
CVE-2022-20615
CVE-2022-20616
CVE-2022-20618
CVE-2022-20619
CVE-2022-20621
CVE-2022-23106
CVE-2022-25647
CVE-2022-27195
CVE-2022-27196
CVE-2022-29048
CVE-2022-33980
CVE-2022-38663
CVE-2022-40149
CVE-2022-40150
CVE-2022-41854
CVE-2022-42003
CVE-2022-42004
CVE-2022-43410
CVE-2022-45693
CVE-2023-1436
CVE-2023-2976
CVE-2023-32261
CVE-2023-32262
CVE-2023-3315
CVE-2023-35116
CVE-2023-35142
CVE-2023-35146
CVE-2023-35147
CVE-2023-35148
CVE-2023-35149
CVE-2023-37946
CVE-2023-37947
CVE-2023-40336
CVE-2023-40337
CVE-2023-40338
CVE-2023-40339
CVE-2023-40341
RHSA-2022:6531
RHSA-2022:7865
RHSA-2022:9098
RHSA-2023:0560
RHSA-2023:0697
RHSA-2023:1655
RHSA-2023:1866
RHSA-2023:3362
RHSA-2023:3625

Testing

CI + below logs from CI showing CVSSv3 comparison skips.

=== RUN   TestImageSanity/mcr.microsoft.com/dotnet/core/runtime:3.1.2
=== RUN   TestImageSanity/mcr.microsoft.com/dotnet/core/runtime:3.1.2/microsoft.netcore.app/3.1.2
    sanity_test.go:40: WARN: No CVSSv3 data provided for "CVE-2021-1721", skipping CVSSv3 field validation.
    sanity_test.go:40: WARN: No CVSSv3 data provided for "CVE-2021-1723", skipping CVSSv3 field validation.
    sanity_test.go:40: WARN: No CVSSv3 data provided for "CVE-2021-24112", skipping CVSSv3 field validation.
    sanity_test.go:40: WARN: No CVSSv3 data provided for "CVE-2021-26701", skipping CVSSv3 field validation.
    sanity_test.go:40: WARN: No CVSSv3 data provided for "CVE-2021-31204", skipping CVSSv3 field validation.
=== RUN   TestImageSanity/mcr.microsoft.com/dotnet/core/sdk:3.1.100@sha256:091126a93870729f4438ee7ed682ed98639a89acebed40409af90f84302c48dd
=== RUN   TestImageSanity/mcr.microsoft.com/dotnet/core/sdk:3.1.100@sha256:091126a93870729f4438ee7ed682ed98639a89acebed40409af90f84302c48dd/microsoft.aspnetcore.app/3.1.0
    sanity_test.go:40: WARN: No CVSSv3 data provided for "CVE-2020-1045", skipping CVSSv3 field validation.
    sanity_test.go:40: WARN: No CVSSv3 data provided for "CVE-2021-1723", skipping CVSSv3 field validation.
=== RUN   TestImageSanity/mcr.microsoft.com/dotnet/core/sdk:3.1.100@sha256:091126a93870729f4438ee7ed682ed98639a89acebed40409af90f84302c48dd/microsoft.netcore.app/3.1.0
    sanity_test.go:40: WARN: No CVSSv3 data provided for "CVE-2021-1721", skipping CVSSv3 field validation.
    sanity_test.go:40: WARN: No CVSSv3 data provided for "CVE-2021-1723", skipping CVSSv3 field validation.
    sanity_test.go:40: WARN: No CVSSv3 data provided for "CVE-2021-24112", skipping CVSSv3 field validation.
    sanity_test.go:40: WARN: No CVSSv3 data provided for "CVE-2021-26701", skipping CVSSv3 field validation.
    sanity_test.go:40: WARN: No CVSSv3 data provided for "CVE-2021-31204", skipping CVSSv3 field validation.

Also, using a random test case, I removed the CVSSv3 fields from one of the vulns and verified the test fails when CVSSv3 data is in the feeds (this proves that when the removed NVD data returns, the tests should fail)

=== RUN   TestImageSanity
=== RUN   TestImageSanity/quay.io/rhacs-eng/qa:apache-server-scannerci
=== RUN   TestImageSanity/quay.io/rhacs-eng/qa:apache-server-scannerci/cron/3.0pl1-124ubuntu2
    /Users/dcaravel/dev/stackrox/scanner/e2etests/sanity_test.go:40:
        	Error Trace:	/Users/dcaravel/dev/stackrox/scanner/e2etests/sanity_test.go:40
        	            				/Users/dcaravel/dev/stackrox/scanner/e2etests/sanity_test.go:129
        	Error:      	Should be false
        	Test:       	TestImageSanity/quay.io/rhacs-eng/qa:apache-server-scannerci/cron/3.0pl1-124ubuntu2
        	Messages:   	Test case for "CVE-2019-9706" is missing CVSSv3 scores, please add the scores to the test case. Scores from vuln data: map[ExploitabilityScore:1.8 ImpactScore:3.6 Score:5.5 Vectors:CVSS:3.1/AV:L/AC:L/PR:L/UI:N/S:U/C:N/I:N/A:H]
    /Users/dcaravel/dev/stackrox/scanner/e2etests/sanity_test.go:51: WARN: No CVSSv3 data provided for "CVE-2019-9706", skipping CVSSv3 field validation.
--- FAIL: TestImageSanity (2.51s)
    --- FAIL: TestImageSanity/quay.io/rhacs-eng/qa:apache-server-scannerci (2.51s)
        --- FAIL: TestImageSanity/quay.io/rhacs-eng/qa:apache-server-scannerci/cron/3.0pl1-124ubuntu2 (0.00s)

Of note: Test case for "CVE-2019-9706" is missing CVSSv3 scores, please add the scores to the test case. Scores from vuln data: map[ExploitabilityScore:1.8 ImpactScore:3.6 Score:5.5 Vectors:CVSS:3.1/AV:L/AC:L/PR:L/UI:N/S:U/C:N/I:N/A:H]

dcaravel · 2024-12-11T01:31:13Z

/test e2e-tests

dcaravel · 2024-12-11T01:39:37Z

Last failure appears to be a flake:

=== RUN   TestStackroxVulnImages/quay.io/rhacs-eng/qa:appdynamics
Handling connection for 8443
    utils.go:49: 
        	Error Trace:	/go/src/github.com/stackrox/scanner/e2etests/utils.go:49
        	            				/go/src/github.com/stackrox/scanner/e2etests/utils.go:65
        	            				/go/src/github.com/stackrox/scanner/e2etests/vuln_test.go:39
        	            				/go/src/github.com/stackrox/scanner/e2etests/vuln_test.go:265
        	Error:      	Received unexpected error:
        	            	rpc error: code = Unknown desc = could not advance in the tar archive: archive/tar: invalid tar header
        	Test:       	TestStackroxVulnImages/quay.io/rhacs-eng/qa:appdynamics

e2etests/sanity_test.go

e2etests/testcase_test.go

e2etests/sanity_test.go

e2etests/testcase_test.go

openshift-ci bot added the do-not-merge/work-in-progress label Dec 10, 2024

adjusts test cases and logic to handle removed NVD data

73b0613

dcaravel force-pushed the dc/ci-fix branch from d25245a to 73b0613 Compare December 10, 2024 21:32

dcaravel changed the title ~~[WIP DO NOT MERGE] chore(ci): Fix CI due to NVD feed changes~~ chore(ci): Fix CI due to NVD feed changes Dec 10, 2024

openshift-ci bot removed the do-not-merge/work-in-progress label Dec 10, 2024

dcaravel changed the title ~~chore(ci): Fix CI due to NVD feed changes~~ chore(ci): Fix CI due to NVD legacy feed changes Dec 10, 2024

dcaravel changed the title ~~chore(ci): Fix CI due to NVD legacy feed changes~~ fix(ci): NVD legacy feed changes broke CI Dec 10, 2024

dcaravel added 2 commits December 10, 2024 16:22

increase test timeouts

e7d0542

fix jenkins test case

85cdbb0

dcaravel requested review from a team, jvdm, BradLugo, daynewlee and RTann December 11, 2024 02:16

dcaravel mentioned this pull request Dec 11, 2024

ROX-26604: Add layer lineage to RHEL v2 data model #1720

Open

dcaravel changed the title ~~fix(ci): NVD legacy feed changes broke CI~~ fix(ci): Multiple data and test issues Dec 11, 2024

jvdm reviewed Dec 11, 2024

View reviewed changes

e2etests/sanity_test.go Outdated Show resolved Hide resolved

dcaravel requested a review from jvdm December 11, 2024 22:35

RTann reviewed Dec 11, 2024

View reviewed changes

e2etests/testcase_test.go Show resolved Hide resolved

e2etests/testcase_test.go Outdated Show resolved Hide resolved

e2etests/sanity_test.go Outdated Show resolved Hide resolved

RTann mentioned this pull request Dec 13, 2024

chore: add manual dotnet entries #1731

Closed

dcaravel added 2 commits December 13, 2024 18:30

fail test if v3 data exists but not tested

408224c

put flakey test back

a2e94b7

dcaravel requested a review from RTann December 15, 2024 01:03

dcaravel changed the title ~~fix(ci): Multiple data and test issues~~ fix(ci): NVD data and test issues Dec 15, 2024

This was referenced Dec 15, 2024

ci: replace docker with quay #1730

Open

yearly cert bump #1748

Merged

RTann approved these changes Dec 18, 2024

View reviewed changes

e2etests/testcase_test.go Show resolved Hide resolved

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(ci): NVD data and test issues #1737

fix(ci): NVD data and test issues #1737

dcaravel commented Dec 10, 2024 •

edited

Loading

dcaravel commented Dec 11, 2024

dcaravel commented Dec 11, 2024 •

edited

Loading

fix(ci): NVD data and test issues #1737

Are you sure you want to change the base?

fix(ci): NVD data and test issues #1737

Conversation

dcaravel commented Dec 10, 2024 • edited Loading

Description

Flapping results for CVE-2021-26291

Testing

dcaravel commented Dec 11, 2024

dcaravel commented Dec 11, 2024 • edited Loading

dcaravel commented Dec 10, 2024 •

edited

Loading

dcaravel commented Dec 11, 2024 •

edited

Loading