Fix flaky TestMultipleClientsWithMixedLabelsAndExpectFailure
#599
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
/find-flaky-tests Run: https://github.com/grafana/dskit/actions/runs/11237399709 |
julienduchesne
force-pushed
the
julienduchesne/flaky-test-TestMultipleClientsWithMixedLabelsAndExpectFailure
branch
from
3b58d5f
to
f76ff6b
Compare
julienduchesne
commented
Oct 8, 2024
Comment on lines
-11
to
+12
| fail-fast: false | ||
| strategy: | ||
| fail-fast: false |
There was a problem hiding this comment.
I'll get it right sometime... I managed to start a run anyways since "workflow dispatch" seems to use the workflow config from the branch, while a workflow triggered from issue comments takes the workflow config from main (while checking out the branch to run tests)
This failed in a bunch of runs in #598 When it fails, it has 3 updates, but they're all from the same member. I believe what we want to check is that the cluster never gets joined by members with labels
julienduchesne
force-pushed
the
julienduchesne/flaky-test-TestMultipleClientsWithMixedLabelsAndExpectFailure
branch
from
f76ff6b
to
7464c1e
Compare
seizethedave
approved these changes
Oct 8, 2024
julienduchesne
deleted the
julienduchesne/flaky-test-TestMultipleClientsWithMixedLabelsAndExpectFailure
branch
julienduchesne
added a commit
that referenced
this pull request
Oct 9, 2024
#599 went in a wrong direction, so I revert it here Instead, to fix the `TestMultipleClientsWithMixedLabelsAndExpectFailure` test, I add a couple more labeled members, that way, even if we reach 3 updates (which happened some times), we'll never get to 5
julienduchesne
added a commit
that referenced
this pull request
Oct 9, 2024
#599 went in a wrong direction, so I revert it here: it makes all `TestMultipleClients` tests flaky since not all members are necessarily joined after the updates Instead, to fix the `TestMultipleClientsWithMixedLabelsAndExpectFailure` test, I add a couple more labeled members, that way, even if we reach 3 updates (which happened some times), we'll never get to 5 with a single member Also, try to fix the `TestTLSServerWithLocalhostCertWithClientCertificateEnforcementUsingClientCA1` test by closing GRPC connections. Maybe this (golang/go#36700) is the issue?
julienduchesne
added a commit
that referenced
this pull request
Oct 9, 2024
#599 went in a wrong direction, so I revert it here: it makes all `TestMultipleClients` tests flaky since not all members are necessarily joined after the updates Instead, to fix the `TestMultipleClientsWithMixedLabelsAndExpectFailure` test, I add a couple more labeled members, that way, even if we reach 3 updates (which happened some times), we'll never get to 5 with a single member Also, try to fix the `TestTLSServerWithLocalhostCertWithClientCertificateEnforcementUsingClientCA1` test by closing GRPC connections. Maybe this (golang/go#36700) is the issue?
julienduchesne
added a commit
that referenced
this pull request
Oct 10, 2024
* Really fix flaky `TestMultipleClients` tests #599 went in a wrong direction, so I revert it here: it makes all `TestMultipleClients` tests flaky since not all members are necessarily joined after the updates Instead, to fix the `TestMultipleClientsWithMixedLabelsAndExpectFailure` test, I add a couple more labeled members, that way, even if we reach 3 updates (which happened some times), we'll never get to 5 with a single member Also, try to fix the `TestTLSServerWithLocalhostCertWithClientCertificateEnforcementUsingClientCA1` test by closing GRPC connections. Maybe this (golang/go#36700) is the issue? * Set `MaxIdleConnsPerHost` * New attempt to fix this test. Could it be aborted connections? * Retry RSTs
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This failed in a bunch of runs in #598
When it fails, it has 3 updates, but they're all from the same member. I believe what we want to check is that the cluster never gets joined by members with labels