Cop worker got panicked when tidb_replica_read is closest-adaptive

[zyguan]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

The minimal reproduce step is unknown. However, there is potential data race on Request.MatchStoreLabels when tidb_replica_read = closest-adaptive:

• the labels may get appended on BuildCopIterator by ClosestReplicaReadAdjuster
• the labels also get read on handleTaskOnce by copIteratorWorker, and then used in candidate selecting.
• when there are multiple kv ranges for an index worker, each kvReq actually share the same &builder.Request here, then data race may occur (eg. first req start cop worker and read the labels while the following reqs append the labels).

Here is the demo test for the above case:

2. What did you expect to see? (Required)

No error reported.

3. What did you see instead (Required)

[2025/03/18 08:13:00.816 +00:00] [ERROR] [coprocessor.go:1113] ["copIteratorWork meet panic"] [r="runtime error: invalid memory address or nil pointer dereference"] ["stack trace"="github.com/pingcap/tidb/pkg/store/copr.(*copIteratorWorker).handleTask.func1\n\t/workspace/source/tidb/pkg/store/copr/coprocessor.go:1115\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:914\nruntime.panicmem\n\t/usr/local/go/src/runtime/panic.go:261\nruntime.sigpanic\n\t/usr/local/go/src/runtime/signal_unix.go:861\ngithub.com/tikv/client-go/v2/internal/locate.(*Store).IsLabelsMatch\n\t/root/go/pkg/mod/github.com/tikv/client-go/[email protected]/internal/locate/region_cache.go:2780\ngithub.com/tikv/client-go/v2/internal/locate.(*accessFollower).isCandidate\n\t/root/go/pkg/mod/github.com/tikv/client-go/[email protected]/internal/locate/region_request.go:1020\ngithub.com/tikv/client-go/v2/internal/locate.(*accessFollower).next\n\t/root/go/pkg/mod/github.com/tikv/client-go/[email protected]/internal/locate/region_request.go:906\ngithub.com/tikv/client-go/v2/internal/locate.(*replicaSelector).next\n\t/root/go/pkg/mod/github.com/tikv/client-go/[email protected]/internal/locate/region_request.go:1178\ngithub.com/tikv/client-go/v2/internal/locate.(*RegionRequestSender).getRPCContext\n\t/root/go/pkg/mod/github.com/tikv/client-go/[email protected]/internal/locate/region_request.go:1478\ngithub.com/tikv/client-go/v2/internal/locate.(*RegionRequestSender).SendReqCtx\n\t/root/go/pkg/mod/github.com/tikv/client-go/[email protected]/internal/locate/region_request.go:1608\ngithub.com/tikv/client-go/v2/txnkv/txnsnapshot.(*ClientHelper).SendReqCtx\n\t/root/go/pkg/mod/github.com/tikv/client-go/[email protected]/txnkv/txnsnapshot/client_helper.go:149\ngithub.com/pingcap/tidb/pkg/store/copr.(*copIteratorWorker).handleTaskOnce\n\t/workspace/source/tidb/pkg/store/copr/coprocessor.go:1234\ngithub.com/pingcap/tidb/pkg/store/copr.(*copIteratorWorker).handleTask\n\t/workspace/source/tidb/pkg/store/copr/coprocessor.go:1126\ngithub.com/pingcap/tidb/pkg/store/copr.(*copIteratorWorker).run\n\t/workspace/source/tidb/pkg/store/copr/coprocessor.go:799"]

github.com/pingcap/tidb/pkg/store/copr.(*copIteratorWorker).handleTask.func1
        /workspace/source/tidb/pkg/store/copr/coprocessor.go:1115
runtime.gopanic
        /usr/local/go/src/runtime/panic.go:914
runtime.panicmem
        /usr/local/go/src/runtime/panic.go:261
runtime.sigpanic
        /usr/local/go/src/runtime/signal_unix.go:861
github.com/tikv/client-go/v2/internal/locate.(*Store).IsLabelsMatch
        /root/go/pkg/mod/github.com/tikv/client-go/[email protected]/internal/locate/region_cache.go:2780
github.com/tikv/client-go/v2/internal/locate.(*accessFollower).isCandidate
        /root/go/pkg/mod/github.com/tikv/client-go/[email protected]/internal/locate/region_request.go:1020
github.com/tikv/client-go/v2/internal/locate.(*accessFollower).next
        /root/go/pkg/mod/github.com/tikv/client-go/[email protected]/internal/locate/region_request.go:906
github.com/tikv/client-go/v2/internal/locate.(*replicaSelector).next
        /root/go/pkg/mod/github.com/tikv/client-go/[email protected]/internal/locate/region_request.go:1178
github.com/tikv/client-go/v2/internal/locate.(*RegionRequestSender).getRPCContext
        /root/go/pkg/mod/github.com/tikv/client-go/[email protected]/internal/locate/region_request.go:1478
github.com/tikv/client-go/v2/internal/locate.(*RegionRequestSender).SendReqCtx
        /root/go/pkg/mod/github.com/tikv/client-go/[email protected]/internal/locate/region_request.go:1608
github.com/tikv/client-go/v2/txnkv/txnsnapshot.(*ClientHelper).SendReqCtx
        /root/go/pkg/mod/github.com/tikv/client-go/[email protected]/txnkv/txnsnapshot/client_helper.go:149
github.com/pingcap/tidb/pkg/store/copr.(*copIteratorWorker).handleTaskOnce
        /workspace/source/tidb/pkg/store/copr/coprocessor.go:1234
github.com/pingcap/tidb/pkg/store/copr.(*copIteratorWorker).handleTask
        /workspace/source/tidb/pkg/store/copr/coprocessor.go:1126
github.com/pingcap/tidb/pkg/store/copr.(*copIteratorWorker).run
        /workspace/source/tidb/pkg/store/copr/coprocessor.go:799

4. What is your TiDB version? (Required)

v7.5.6