kvserver: add rac2 v1 integration tests #130728

In `ProcessPiggybackedAdmittedAtLeaderRaftMuLocked`, we swap the pending updates map with another one before releasing the mutex, in order to minimize the time holding a mutex. Subsequently, the method would then check if the updates map was empty by calling `len()`, performing a read. When the updates map was not empty, the read was against a map which was now only referenced in this function, because a new map was swapped in under the lock. When the updates map was empty, the map isn't swapped out and is referenced by the receiver struct instance, reading a ref without locking. Return whether the updates map was empty, to avoid racing when reading the length of the shared ref empty map. Part of: cockroachdb#130187 Release note: None

Pass in `ReplicasStateInfo` via `RaftEvent` to the `RangeController`. This resolves an issue where the Raft status was raced on due to not holding a replica mu while reading. Part of: cockroachdb#130187 Release note: None

Pass the original token deduction into the `OverrideTokenDeduction` testing knob function. This change enables rounding or modifying the original value, which is useful for flow control integration tests which wish to issue variable sized writes, without the associated flakiness. Resolves: cockroachdb#130187 Release note: None

Introduce a new field on `StoreTestingKnobs`: `RaftReportUnreachableBypass`. `RaftReportUnreachableBypass` is called when a replica reports that another replica is unreachable. If the bypass function is non-nil and returns true, the report is ignored and `ReportUnreachable` is not called on the raft group for that replica. Resolves: cockroachdb#130187 Release note: None

The `Processor` calls `isLeaderUsingV2ProcLocked` to determine which store work queue admit method to call, opting for the v1 method if `isLeaderUsingV2ProcLocked` returns false. Update `isLeaderUsingV2ProcLocked` to correctly return false when the local replica is the leader and running v1, having previously seen a leader running v2. Part of: cockroachdb#130431 Release note: None

The existing flow control integration tests, in `flow_control_integration_test.go`, provide a substantial suite of tests to verify flow control integration behavior. Refactor the `flowControlTestHelper` and associated use in tests, to enable sharing the helper between v1 (existing) and not-yet implemented v2 flow control integration tests. Part of: cockroachdb#130187 Release note: None

Introduce several tests in `flow_control_integration_test.go`, mirroring the existing tests but applied to the replication flow control v2 machinery. The tests largely follow an identical pattern to the existing v1 tests, swapping in rac2 metrics and vtables. The following tests are added: ``` TestFlowControlBasicV2 TestFlowControlRangeSplitMergeV2 TestFlowControlBlockedAdmissionV2 TestFlowControlAdmissionPostSplitMergeV2 TestFlowControlCrashedNodeV2 TestFlowControlRaftSnapshotV2 TestFlowControlRaftMembershipV2 TestFlowControlRaftMembershipRemoveSelfV2 TestFlowControlClassPrioritizationV2 TestFlowControlQuiescedRangeV2 TestFlowControlUnquiescedRangeV2 TestFlowControlTransferLeaseV2 TestFlowControlLeaderNotLeaseholderV2 TestFlowControlGranterAdmitOneByOneV2 ``` These tests all have at least two variants: ``` V2EnabledWhenLeaderV1Encoding V2EnabledWhenLeaderV2Encoding ``` When `V2EnabledWhenLeaderV1Encoding` is run, the tests use a different testdata file, which has a `_v1_encoding` suffix. A separate file is necessary because when the protocol enablement level is `V2EnabledWhenLeaderV1Encoding`, all entries which are subject to admission control are encoded as `raftpb.LowPri`, regardless of their original priority, as we don't want to pay the cost to deserialize the raft admission meta. The v1 encoding variants retain the same comments as the v2 encoding, however any comments referring to regular tokens should be interpreted as elastic tokens instead, due to the above. Two v1 tests are not ported over to v2: ``` TestFlowControlRaftTransportBreak TestFlowControlRaftTransportCulled ``` These omitted tests behave identically to `TestFlowControlCrashedNodeV2` as rac2 is less tightly coupled to the raft transport, instead operating on replication states (e.g., `StateProbe`, `StateReplicate`). Resolves: cockroachdb#130187 Release note: None

Add `TestFlowControlV1ToV2Transition`, which ratchets up the enabled version of replication flow control v2: ``` v1 protocol with v1 encoding => v2 protocol with v1 encoding => v2 protocol with v2 encoding ``` The test is structured to issue writes and wait for returned tokens whenever the protocol transitions from v1 to v2, or a leader changes. More specifically, the test takes the following steps: ``` (1) Start n1, n2, n3 with v1 protocol and v1 encoding. (2) Upgrade n1 to v2 protocol with v1 encoding. (3) Transfer the range lease to n2. (4) Upgrade n2 to v2 protocol with v1 encoding. (5) Upgrade n3 to v2 protocol with v1 encoding. (6) Upgrade n1 to v2 protocol with v2 encoding. (7) Transfer the range lease to n1. (8) Upgrade n2,n3 to v2 protocol with v2 encoding. (9) Transfer the range lease to n3. ``` Between each step, we issue writes, (un)block admission and observe the flow control metrics and vtables. Resolves: cockroachdb#130431 Resolves: cockroachdb#129276 Release note: None

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kvserver: add rac2 v1 integration tests #130728

kvserver: add rac2 v1 integration tests #130728

Commits on Sep 25, 2024