ROX-21387: batch insert vulns

[RTann]

$ go test -run=^$ -bench=^BenchmarkLoadOSVulnsFromDump$ -benchmem ./pkg/vulndump/
goos: darwin
goarch: amd64
pkg: github.com/stackrox/scanner/pkg/vulndump
cpu: Intel(R) Core(TM) i5-1038NG7 CPU @ 2.00GHz
BenchmarkLoadOSVulnsFromDump-8   	       1	4358242218 ns/op	1314951824 B/o 8285803 allocs/op
--- BENCH: BenchmarkLoadOSVulnsFromDump-8
    loader_test.go:26: Loaded 134043 vulns
PASS
ok  	github.com/stackrox/scanner/pkg/vulndump	5.990s



$ go test -run=^$ -bench=^BenchmarkOSLoader$ -benchmem ./pkg/vulndump/
goos: darwin
goarch: amd64
pkg: github.com/stackrox/scanner/pkg/vulndump
cpu: Intel(R) Core(TM) i5-1038NG7 CPU @ 2.00GHz
BenchmarkOSLoader-8   	       1	4075758627 ns/op	702698568 B/op	 8285718 allocs/op
--- BENCH: BenchmarkOSLoader-8
    batch_loader_test.go:57: Loaded 134043 vulns
PASS
ok  	github.com/stackrox/scanner/pkg/vulndump	5.450s

The batched implementation uses a little over half of the memory/op than the original implementation.

I also ran an experiment where I ran the current implementation and the implementation from this PR. In each separate cluster, I ran StackRox in offline-mode, and I uploaded the latest "offline" vuln dump to Central. From there I found the current implementation stayed at around 3.5G for about 20 minutes, while this PR's implementation hovered around 700MB for a few minutes before going back down to about 500MB, which is its steady-state. This is a rather significant memory reduction.

[ghost]

Images are ready for the commit at 748bc9e.

To use the images, use the tag 2.31.x-64-g748bc9ee93.

[ludydoo]

Amazing! 🥇

[RTann]

Current Scanner loading all vulns (offline-mode):

This implementation loading all vulns (offline-mode):

You'll find the current Scanner requires about 3.5G while this implementation only requires ~700MB.

[dcaravel]

LGTM, wondering if its worth running some queries against the DB just to double check that the # of vulns loaded before/after this fix are the same, and perhaps also query the table for total size / bytes consumed - if they are the same, ship it!

[RTann]

I added a benchmark test for the offline dumps, too:

$ go test -run=^$ -bench=^BenchmarkLoadOSVulnsFromDump -benchmem ./pkg/vulndump/
goos: darwin
goarch: amd64
pkg: github.com/stackrox/scanner/pkg/vulndump
cpu: Intel(R) Core(TM) i5-1038NG7 CPU @ 2.00GHz
BenchmarkLoadOSVulnsFromDump-8           	       1	4300235287 ns/op	1315103992 B/op	 8293226 allocs/op
--- BENCH: BenchmarkLoadOSVulnsFromDump-8
    loader_test.go:38: Loaded 134212 vulns
BenchmarkLoadOSVulnsFromDump_Offline-8   	       1	13470219395 ns/op	4659625192 B/op	26622319 allocs/op
--- BENCH: BenchmarkLoadOSVulnsFromDump_Offline-8
    loader_test.go:38: Loaded 415338 vulns
PASS
ok  	github.com/stackrox/scanner/pkg/vulndump	22.328s


$ go test -run=^$ -bench=^BenchmarkOSLoader -benchmem ./pkg/vulndump/
goos: darwin
goarch: amd64
pkg: github.com/stackrox/scanner/pkg/vulndump
cpu: Intel(R) Core(TM) i5-1038NG7 CPU @ 2.00GHz
BenchmarkOSLoader-8           	       1	3906161039 ns/op	702851656 B/op	 8293140 allocs/op
--- BENCH: BenchmarkOSLoader-8
    batch_loader_test.go:109: Loaded 134212 vulns
BenchmarkOSLoader_Offline-8   	       1	12385573250 ns/op	2255503144 B/op	26622223 allocs/op
--- BENCH: BenchmarkOSLoader_Offline-8
    batch_loader_test.go:109: Loaded 415338 vulns
PASS
ok  	github.com/stackrox/scanner/pkg/vulndump	20.754s

This also prints out the number of vulnerabilities we read from the JSON file, and they both match. I will merge with CI

[openshift-ci]

@RTann: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-tests	748bc9e	link	false	/test e2e-tests

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.