Skip to content

Commit

Permalink
per-test-execution: add html report (#3776)
Browse files Browse the repository at this point in the history 
* per-test-execution: add html report

Refactoring wrt functions included.

Signed-off-by: Daniel Hiller <[email protected]>

* per-test-execution: parallelize test lane data retrieval

Signed-off-by: Daniel Hiller <[email protected]>

* per-test-execution,report: add link to latest failed

Adds a link per each test that will navigate to the latest run where the
test had failed.

Signed-off-by: Daniel Hiller <[email protected]>

* per-test-execution: introduce sorting by highest rate

Sort output by highest overall test failure rate from top to bottom.

Signed-off-by: Daniel Hiller <[email protected]>

* per-test-execution: change to write to directory

All files now have persistent names (test lane name, index.html), only
the target directory name is generated.

Signed-off-by: Daniel Hiller <[email protected]>

* per-test-execution: add flag output-directory

Since we need to have a clear destination we are adding the flakg
`output-directory`, in order to use it inside a prow job.

Signed-off-by: Daniel Hiller <[email protected]>

* per-test-results: create periodic job

Signed-off-by: Daniel Hiller <[email protected]>

* per-test-execution: fix bazel build

Signed-off-by: Daniel Hiller <[email protected]>

* per-test-execution: add link to generated report

Adds a link to the `reports.md` report overview page, so that we can
easily navigate to the generated report.

Signed-off-by: Daniel Hiller <[email protected]>

* per-test-execution: address review comment

Signed-off-by: Daniel Hiller <[email protected]>

* per-test-execution: improve readability

Signed-off-by: Daniel Hiller <[email protected]>

---------

Signed-off-by: Daniel Hiller <[email protected]>
  • Loading branch information
@dhiller
dhiller authored Dec 3, 2024
1 parent 18cfc49 commit 948ca35
Show file tree
Hide file tree
Showing 7 changed files with 470 additions and 118 deletions.
21 changes: 11 additions & 10 deletions docs/reports.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,16 +4,17 @@ Collection of links to search portals, statistics, and reports that are periodic

## kubevirt/kubevirt

| Link | Docs |
|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------|
| [testgrid](https://testgrid.k8s.io/kubevirt) | TestGrid holds stats for our periodic and presubmit e2e jobs |
| [ci-health](https://kubevirt.io/ci-health/#kubevirtkubevirt) | Overall health of KubeVirt CI, stats for merge queue, days to merge etc. |
| [merge queue (GitHub query)](https://github.com/kubevirt/kubevirt/pulls?q=sort%3Aupdated-desc+is%3Apr+is%3Aopen+label%3Aapproved+label%3Algtm+-label%3Ado-not-merge%2Fhold+-label%3Ado-not-merge%2Fwork-in-progress+-label%3Ado-not-merge%2Frelease-note-label-needed+-label%3Aneeds-rebase) | GitHub query showing PRs that are actually fit for merge, i.e. they have the required set of labels |
| [flakefinder reports](https://storage.googleapis.com/kubevirt-prow/reports/flakefinder/kubevirt/kubevirt/index.html) | Report over test failures that have occurred on merged PRs |
| [flake stats](https://storage.googleapis.com/kubevirt-prow/reports/flakefinder/kubevirt/kubevirt/flake-stats-14days.html) | Stats to show which flakes have hurt the most, and where |
| [presubmits](https://storage.googleapis.com/kubevirt-prow/reports/e2ejobs/kubevirt/kubevirt/presubmits.html) | List of e2e presubmit jobs, their ENV var settings, and whether they gate the merge or not |
| [periodics](https://storage.googleapis.com/kubevirt-prow/reports/e2ejobs/kubevirt/kubevirt/periodics.html) | List of e2e periodic jobs, their ENV var settings, and whether they gate the merge or not |
| [quarantined tests](https://storage.googleapis.com/kubevirt-prow/reports/quarantined-tests/kubevirt/kubevirt/index.html) | List of quarantined tests, i.e. tests that have the QUARANTINED label |
| Link | Docs |
|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------|
| [testgrid](https://testgrid.k8s.io/kubevirt) | TestGrid holds stats for our periodic and presubmit e2e jobs |
| [ci-health](https://kubevirt.io/ci-health/#kubevirtkubevirt) | Overall health of KubeVirt CI, stats for merge queue, days to merge etc. |
| [merge queue (GitHub query)](https://github.com/kubevirt/kubevirt/pulls?q=sort%3Aupdated-desc+is%3Apr+is%3Aopen+label%3Aapproved+label%3Algtm+-label%3Ado-not-merge%2Fhold+-label%3Ado-not-merge%2Fwork-in-progress+-label%3Ado-not-merge%2Frelease-note-label-needed+-label%3Aneeds-rebase) | GitHub query showing PRs that are actually fit for merge, i.e. they have the required set of labels |
| [flakefinder reports](https://storage.googleapis.com/kubevirt-prow/reports/flakefinder/kubevirt/kubevirt/index.html) | Report over test failures that have occurred on merged PRs |
| [flake stats](https://storage.googleapis.com/kubevirt-prow/reports/flakefinder/kubevirt/kubevirt/flake-stats-14days.html) | Stats to show which flakes have hurt the most, and where |
| [presubmits](https://storage.googleapis.com/kubevirt-prow/reports/e2ejobs/kubevirt/kubevirt/presubmits.html) | List of e2e presubmit jobs, their ENV var settings, and whether they gate the merge or not |
| [periodics](https://storage.googleapis.com/kubevirt-prow/reports/e2ejobs/kubevirt/kubevirt/periodics.html) | List of e2e periodic jobs, their ENV var settings, and whether they gate the merge or not |
| [quarantined tests](https://storage.googleapis.com/kubevirt-prow/reports/quarantined-tests/kubevirt/kubevirt/index.html) | List of quarantined tests, i.e. tests that have the QUARANTINED label |
| [per-test-execution (6 months)](https://storage.googleapis.com/kubevirt-prow/reports/per-test-results/kubevirt/kubevirt/last-six-months/index.html) | top-x report of failing tests per lane over the last 6 months |

See also [org debugging resources](./debugging-resources/org-debugging-resources.md)

Expand Down
32 changes: 31 additions & 1 deletion github/ci/prow-deploy/files/jobs/kubevirt/kubevirt/kubevirt-periodics.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,37 @@ periodics:
- -ce
env:
- name: GIMME_GO_VERSION
value: "1.19"
value: "1.21.6"
image: quay.io/kubevirtci/golang:v20241014-80f340c
name: ""
resources: {}
- annotations:
testgrid-create-test-group: "false"
cluster: kubevirt-prow-control-plane
cron: 45 3 * * *
decorate: true
extra_refs:
- base_ref: main
org: kubevirt
repo: project-infra
labels:
preset-gcs-credentials: "true"
name: periodic-push-per-test-execution-results-six-months
spec:
containers:
- args:
- |
report_dir="/tmp/per-test-results/last-six-months"
mkdir -p "${report_dir}"
go run ./robots/cmd/per-test-execution --months 6 --output-directory "${report_dir}"
gsutil cp "${report_dir}/*" "gs://kubevirt-prow/reports/per-test-results/kubevirt/kubevirt/last-six-months/"
command:
- /usr/local/bin/runner.sh
- /bin/bash
- -ce
env:
- name: GIMME_GO_VERSION
value: "1.21.6"
image: quay.io/kubevirtci/golang:v20241014-80f340c
name: ""
resources: {}
Expand Down
5 changes: 4 additions & 1 deletion robots/cmd/per-test-execution/BUILD.bazel
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,10 @@ load("@io_bazel_rules_go//go:def.bzl", "go_binary", "go_library", "go_test")
go_library(
name = "go_default_library",
srcs = ["main.go"],
embedsrcs = ["config.yaml"],
embedsrcs = [
"config.yaml",
"per-test-execution-top-x.gohtml",
],
importpath = "kubevirt.io/project-infra/robots/cmd/per-test-execution",
visibility = ["//visibility:private"],
deps = [
Expand Down
34 changes: 34 additions & 0 deletions robots/cmd/per-test-execution/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# per-test-execution

Creates a set of CSV files in the format

| test-name | number-of-executions | number-of-failures | {build-url-1} | ... | {build-url-m} |
|-----------------|--------------------------|------------------------|---------------|-----|---------------|
| {{test-name-1}} | {{number-of-executions}} | {{number-of-failures}} | {p,f,s} | ... | {p,f,s} |
| ... | ... | ... | ... | ... | ... |
| {{test-name-n}} | {{number-of-executions}} | {{number-of-failures}} | {p,f,s} | ... | {p,f,s} |

where `n` is the number of tests and `m` is the number of builds.

Data is fetched from the build artifacts of the prow jobs, namely the `junit.xml` files that contain the test result data.

In detail the process works as follows:
* Per a target directory depending on the job we want to look at, we look at it's content directories, which resemble a job run each
* each job run we check whether it's inside the interval we want, and then extract the junit results
* we then aggregate the results and write them into a csv file

## Pre-requisites

env `GOOGLE_APPLICATION_CREDENTIALS` set to point to a GCS credentials file - see GCloud credentials docs [1] .

## Examples

```bash
go run ./robots/cmd/per-test-execution --days 7
```

```bash
go run ./robots/cmd/per-test-execution --months 6
```

[1]: https://cloud.google.com/docs/authentication/application-default-credentials
Loading

0 comments on commit 948ca35

Please sign in to comment.