Collect code coverage after running every test case

[Spedoske]

Here is demo of collecting code coverage at runtime. To collect coverage, the following code should add to the operator. We can also make the code as a module and use anonymous import to run the code. The coverage can be collected after every test case.

package main

// See https://go.dev/testing/coverage/
// Build with -cover
// go 1.20+ required

// $ mkdir coverage
// $ go run -cover main.go
// $ curl localhost:8888
// $ curl localhost:8888/dump/coverage
// $ go tool covdata textfmt -o foo -i=coverage
// $ go tool covdata percent -i=coverage
// $ curl localhost:8888 --cookie "bar=baz"
// $ curl localhost:8888/dump/coverage
// $ go tool covdata textfmt -o bar -i=coverage
// $ go tool covdata percent -i=coverage
import (
	"net/http"
	"runtime/coverage"
)

const CoverageDir = "./coverage"

func dumpCoverage(http.ResponseWriter, *http.Request) {
	err := coverage.WriteCountersDir(CoverageDir)
	if err != nil {
		panic(err)
	}
}

func handle(rw http.ResponseWriter, r *http.Request) {
	if len(r.Cookies()) == 0 {
		_, err := rw.Write([]byte("Foo"))
		if err != nil {
			panic(err)
		}
		return
	}
	_, err := rw.Write([]byte("Bar"))
	if err != nil {
		panic(err)
	}
}

func main() {
	err := coverage.WriteMetaDir(CoverageDir)
	if err != nil {
		panic(err)
	}
	http.HandleFunc("/", handle)
	http.HandleFunc("/dump/coverage", dumpCoverage)
	err = http.ListenAndServe(":8888", nil)
	if err != nil {
		panic(err)
	}
}

Coverage-guided testing can be done though this, but it should be hard. As we must wait for the states of the cluster converges. Therefore, the procedure of generating new tests must be very precise, so that we can discover new paths and improve code coverage using these new tests effectively.

[tylergu]

We actually have already done something like this before, but to collect code coverage at the end of entire testing.
We ended up having to rewrite the main functions for each operator to collect code coverage for e2e tests.
For example, collecting CassOp's codecov: https://github.com/xlab-uiuc/acto/blob/main/data/cass-operator/config_cov.json
It turned out that our coverage is not as good as existing operators' tests, I think it is mainly because the codecov is harder to achieve for e2e test versus unittest.

I think it is a good idea to use code coverage for input generation.
One of the challenges is that e2e tests are too expensive to try a lot of inputs.

One idea I had was to first generates a lot of inputs guided by coverage running integration test. Once we have the list of inputs which can have large code coverage, then we run e2e tests to test their correctness.

[tianyin]

Great post @Spedoske !

I love the idea of collecting code coverage, but code coverage per test case is too fine-grained. If there is a way to aggregate to code coverage per test campaign, it will be more useful.

[tianyin]

It turned out that our coverage is not as good as existing operators' tests, I think it is mainly because the codecov is harder to achieve for e2e test versus unittest.

So, that's a reason we didn't put it in the draft, but I think adding the code coverage support is very useful for us.

[tianyin]

Coverage-guided testing can be done though this, but it should be hard. As we must wait for the states of the cluster converges. Therefore, the procedure of generating new tests must be very precise, so that we can discover new paths and improve code coverage using these new tests effectively.

One idea I had was to first generates a lot of inputs guided by coverage running integration test. Once we have the list of inputs which can have large code coverage, then we run e2e tests to test their correctness.

We will need a deeper discussion on how to leverage some idea(s) from fuzzing to do smarter things.

[tianyin]

@Spedoske have you worked on this task? Just check.

[Spedoske]

@Spedoske have you worked on this task? Just check.

No. I am currently working on other tasks. If we want to push the project further, we can have a discussion first.

[tianyin]

What tasks are you focusing now?

@tylergu and I want to prioritize this task. By I need to know what you are working on to be able to "prioritize."

[tylergu]

One example of making the operator to output coverage is written here: tylergu/zookeeper-operator@3d3a1e2

While the changes needed for each operator are different, I am going to document the highlevel steps at https://github.com/xlab-uiuc/acto/blob/main/docs/resource.md#measure-code-coverage-for-acto

[tianyin]

Thanks @tylergu ! This'll be super useful.

[Spedoske]

Go 1.20 ("runtime/coverage" package) enable us to collect real-time coverage, which means we can collect coverage after each step in one single trial.

Here is an example.
https://gitlab.isp.moe/Spedoske/cluster-operator/-/commit/0f8e33cb615c894b030504f8fe0b16902d3afd05

[tianyin]

WOW

[tianyin]

This looks very cool because it seems there is more standard procedures to collect coverage?

[tylergu]

Nice work! So you are going to turn the operator an http server to respond the coverage requests I guess?

[tylergu]

This looks very cool because it seems there is more standard procedures to collect coverage?

@tianyin , the difference this runtime/coverage makes is that, we can collect the coverage information while the operator is still running. Using my old method the coverage is only written to the disk when the operator is terminated.

We still need to go through all the hacks of changing operators' source code and deployment script though.

[tianyin]

@tylergu got it -- thank you for the clarification.

btw I don't really get why we call them "hacks" (which indicates that it was an ad hoc process). To me, collecting code coverage should be a standard practice, despite that different projects are different.

~t