Include latest v4 vulnerability

[daynewlee]

Generate scanner-vuln-update including v4 vulnerability of latest version
Currently I can only verify scanner v2 work with new scanner-vuln-updates.zip, more V4 related tests will be done in ROX-21981

Test

1. Make sure the zip is containing scanner v4 stuff:

ls -lh scanner-vuln-updates.zip 
-rw-r--r-- 1 yili staff 272M Feb 14 15:04 scanner-vuln-updates.zip

unzip -l scanner-vuln-updates.zip
Archive:  scanner-vuln-updates.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
140154404  02-14-2024 11:41   scanner-defs.zip
   248088  02-14-2024 11:41   k8s-istio.zip
144211510  02-14-2024 11:41   scanner-v4-defs-4.3.zip
---------                     -------
284614002                     3 files

2. Upload to Central:

yli3-mac:stackrox yili$ curl --insecure -u admin:[PWD] -F "[email protected]" https://localhost:8000/api/extensions/scannerdefinitions -v
*   Trying [::1]:8000...
* connect to ::1 port 8000 failed: Connection refused
*   Trying 127.0.0.1:8000...
* Connected to localhost (127.0.0.1) port 8000
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Request CERT (13):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Certificate (11):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES128-GCM-SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: OU=CENTRAL_SERVICE; CN=CENTRAL_SERVICE: Central; serialNumber=16561514245731368659
*  start date: Feb 14 20:08:00 2024 GMT
*  expire date: Feb 13 21:08:00 2025 GMT
*  issuer: CN=StackRox Certificate Authority; serialNumber=1018846594209314016
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* using HTTP/2
* Server auth using Basic with user 'admin'
* [HTTP/2] [1] OPENED stream for https://localhost:8000/api/extensions/scannerdefinitions
* [HTTP/2] [1] [:method: POST]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: localhost:8000]
* [HTTP/2] [1] [:path: /api/extensions/scannerdefinitions]
* [HTTP/2] [1] [authorization: Basic YWRtaW46SzNEWGJmZ3NlNVR0UzQ3c01ibG5PZTBqVQ==]
* [HTTP/2] [1] [user-agent: curl/8.4.0]
* [HTTP/2] [1] [accept: */*]
* [HTTP/2] [1] [content-length: 284614740]
* [HTTP/2] [1] [content-type: multipart/form-data; boundary=------------------------GlBMII0WAr7HTBb57GfFwh]
> POST /api/extensions/scannerdefinitions HTTP/2
> Host: localhost:8000
> Authorization: Basic YWRtaW46SzNEWGJmZ3NlNVR0UzQ3c01ibG5PZTBqVQ==
> User-Agent: curl/8.4.0
> Accept: */*
> Content-Length: 284614740
> Content-Type: multipart/form-data; boundary=------------------------GlBMII0WAr7HTBb57GfFwh
> 
* We are completely uploaded and fine
< HTTP/2 200 
< vary: Accept-Encoding
< content-type: text/plain; charset=utf-8
< content-length: 57
< date: Wed, 14 Feb 2024 21:19:38 GMT
< 
* Connection #0 to host localhost left intact
Successfully stored the offline vulnerability definitions

[dcaravel]

Need to be careful here to not 'break' the current latest bundle if this job runs and there is a bug.

I'd expect the output of this job to include updates to all the versioned v4 offline bundles and the latest offline bundle, would expect the following to be uploaded:

scanner-support-public/offline/v1/scanner-vuln-updates.zip
scanner-support-public/offline/v1/4.4/scanner-vulns-4.4.zip

As of this initial release would expect scanner-vuln-updates.zipto be updated and to be created/updated 4.4/scanner-vulns-4.4.zip and both contain the same data (will diverge in the future as more versioned bundles are created)

[daynewlee]

Need to be careful here to not 'break' the current latest bundle if this job runs and there is a bug.

I'd expect the output of this job to include updates to all the versioned v4 offline bundles and latest offline bundles, would expect the following to be uploaded:

scanner-support-public/offline/v1/scanner-vuln-updates.zip

scanner-support-public/offline/v1/4.4/scanner-vulns-4.4.zip

As of this initial release would expect scanner-vuln-updates.zip and 4.4/scanner-vulns-4.4.zip to contain the same data

I haven't decided how to generate scanner-vuln-[version].zip because I am hesitating to use loop to iterate all scanner-defs-[version].zip in this script at this moment as you said that might introduce bugs to this workflow. Since scanner-vuln-update.zip only contains latest v4 defs, so it makes sense to change this bash function. I'll make another PR for the v4 offline bundles.

[dcaravel]

I haven't decided how to generate scanner-vuln-[version].zip because I am hesitating to use loop to iterate all scanner-defs-[version].zip in this script at this moment as you said that might introduce bugs to this workflow. Since scanner-vuln-update.zip only contains latest v4 defs, so it makes sense to change this bash function. I'll make another PR for the v4 offline bundles.

OK, we may be introducing more variance between latest and the versioned bundles by separating these workflows (since the versioned bundles need the Scanner V2 data produced by this job). If the jobs are separated it will be possible that (for example) scanner-defs-4.4.zip added to scanner-vuln-updates.zip will be different from scanner-defs-4.4.zip in 4.4/scanner-vulns-4.4.zip depending on when jobs complete. One may have newer/older data then the other, but be expected to be the same.

FWIW, there are ways to have functions / commands in bash fail without breaking the whole script (if you change your mind).

[daynewlee]

I haven't decided how to generate scanner-vuln-[version].zip because I am hesitating to use loop to iterate all scanner-defs-[version].zip in this script at this moment as you said that might introduce bugs to this workflow. Since scanner-vuln-update.zip only contains latest v4 defs, so it makes sense to change this bash function. I'll make another PR for the v4 offline bundles.

OK, we may be introducing more variance between latest and the versioned bundles by separating these workflows (since the versioned bundles need the Scanner V2 data produced by this job). If the jobs are separated it will be possible that (for example) scanner-defs-4.4.zip added to scanner-vuln-updates.zip will be different from scanner-defs-4.4.zip in 4.4/scanner-vulns-4.4.zip depending on when jobs complete. One may have newer/older data then the other, but be expected to be the same.

FWIW, there are ways to have functions / commands in bash fail without breaking the whole script (if you change your mind).

Yeah, that's true. I will take that into consideration but possibly build on top of this PR. Do you think this PR/change make sense?
Actually I will probably add generate and upload scanner-vuln-[version].zip at very end of this script in case it breaks previous steps. #1403

[dcaravel]

One more nit.

LGTM, approving because believe it will achieve the desired outcome, however because it will modify scanner-vuln-updates.zip which is a 'live' file, HOLD OFF on the merge until others approve and are aware that the change is being made.

Also recommend before merging to test uploading this 'combo' file to older supported versions of Central and ensure nothing breaks (I don't expect things to break, but let's be sure - ie: because we're adding another 300MiB it's possible the default timeouts may need to be adjusted, and we need to be extra sure that older Centrals will ignore the extra V4 files in the bundle)

[jvdm]

Can you provide your testing steps?

1. Scanner V4 works.
2. Scanner V2 works.

[daynewlee]

Can you provide your testing steps?

1. Scanner V4 works.

2. Scanner V2 works.

That makes sense. I will add steps for testing new scanner-vuln-update.zip working with Scanner V2 but probably I can't test the bundles working with Scanner V4 since stackrox/stackrox#9409 and ROX-21981 are not done yet. @jvdm

[RTann]

@daynewlee can I download a sample ZIP to look at?

[daynewlee]

@daynewlee can I download a sample ZIP to look at?

I believe https://storage.googleapis.com/scanner-support-public/offline/v1/scanner-vuln-updates.zip is overwritten by CI now.
But v4 vulns are available
https://storage.googleapis.com/scanner-support-public/offline/v1/4.2/scanner-vulns-4.2.zip
https://storage.googleapis.com/scanner-support-public/offline/v1/4.3/scanner-vulns-4.3.zip
@RTann