Failpoint raftAfterSave=sleep(1s) is flaking in robustness test

[serathius]

Which Github Action / Prow Jobs are flaking?

https://testgrid.k8s.io/sig-etcd-robustness#ci-etcd-robustness-amd64

Which tests are flaking?

TestRobustnessExploratoryKubernetesHighTrafficClusterOfSize3

Github Action / Prow Job link

https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-etcd-robustness-main-amd64/1806029033377370112

Reason for failure (if possible)

    logger.go:146: 2024-06-26T19:45:53.423Z	INFO	goFailpoint deactivate failed	{"failpoint": "raftAfterSave=sleep(1s)", "error": "Delete \"http://127.0.0.1:12381/raftAfterSave\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
    logger.go:146: 2024-06-26T19:45:53.423Z	ERROR	Failed to trigger failpoint	{"failpoint": "raftAfterSave=sleep(1s)", "error": "goFailpoint raftAfterSave=sleep(1s) deactivate failed, err: Delete \"http://127.0.0.1:12381/raftAfterSave\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
    main_test.go:122: failed triggering failpoint, err: goFailpoint raftAfterSave=sleep(1s) deactivate failed, err: Delete "http://127.0.0.1:12381/raftAfterSave": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Anything else we need to know?

No response

[MadhavJivrajani]

validate.go:34: Broken validation assumptions: non empty database at start or first write didn't succeed, required by model implementation

@serathius I also see this in the logs ^

From

etcd/tests/robustness/validate/validate.go

Line 62 in 0f24953

func checkValidationAssumptions(reports []report.ClientReport, persistedRequests []model.EtcdRequest) error {

[MadhavJivrajani]

For the failpoint flake, we have the option of increasing the client timeout for that particular failpoint, but I'm still not entirely sure why its timing out to begin with.

[siyuanfoundation]

/cc @henrybear327

[serathius]

/assign @henrybear327

[k8s-ci-robot]

@serathius: GitHub didn't allow me to assign the following users: henrybear327.

Note that only etcd-io members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign @henrybear327

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[henrybear327]

/assign @henrybear327

[serathius]

@henrybear327 when you will be able to send the PR? Just the timeout increase should be 1 line change.

[henrybear327]

@henrybear327 when you will be able to send the PR? Just the timeout increase should be 1 line change.

Done just now!

I will still continue my investigation on the gofail library to see if any issues are coming from the gofail v0.2.0 changes that I made.

[henrybear327]

I think we can close this issue, right @serathius? IIRC #18397 fixed the issue.

[serathius]

Right, thanks @henrybear327