clientv3test: add comments for clientv3test

[shaoqin2]

Add tests for clientv3test.TestUnresolvableOrderViolation and clientv3test.TestEndpointSwitchResolvesViolation

Improve TestEndpointSwitchResolvesViolation with a partition resolved return no error case

Addresses #16865

[fuweid]

NOTE: The orderingKv.Get doesn't have backoff try. It might make the test case flaky since the CI VM is unstable and the 1 second might be not enough for member[2] to get latest data.

Sorry, I don't have good idea to ensure the data is synced from leader. Maybe we can remove clientv3.WithSerializable() option to force to use linear read.

[shaoqin2]

I think when creating this orderingKv object, we passed the retry function NewOrderViolationSwitchEndpointClosure

	orderingKv := ordering.NewKV(cli.KV, ordering.NewOrderViolationSwitchEndpointClosure(cli))

when the Get fails, we will enter the retry based on

		err = kv.orderViolationFunc(op, r, prevRev)
		if err != nil {
			return nil, err
		}

Am I understanding this correctly? The only thing missing might be an actual backoff function, as it is I think it immediately retries, not sure if that's what you meant.

[fuweid]

Sorry for unclear comment.

The orderingKv did retry immediately.

However, the Github Action CI VM is unstable. When you recover member[2] from network partition and sleep 1s, it doesn't guarantee the member[2] can sync with leader during ordering's retry.

The Line 87 _, err = orderingKv.Get(ctx, "foo", clientv3.WithSerializable()) still might return error.

[shaoqin2]

Ah that make sense. The idea is with default linearizable it will be forced to talk to the other members to ensure the most recent data is returned when Get is called right?

Removed with WithSerializable option from the call. Let me know if looks good

[fuweid]

LGTM, Thanks!

ping @serathius @ahrtr to trigger the CI~

[jmhbnz]

LGTM - Thanks @shaoqin2