Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cdc: kv.clientSuite.TestPrewriteNotMatchError is not stable #199

Open
pingyu opened this issue Aug 9, 2022 · 1 comment
Open

cdc: kv.clientSuite.TestPrewriteNotMatchError is not stable #199

pingyu opened this issue Aug 9, 2022 · 1 comment
Assignees
Labels
status/stale type/bug Something isn't working

Comments

@pingyu
Copy link
Collaborator

pingyu commented Aug 9, 2022

Bug Report

1. Describe the bug

2. Minimal reproduce step (Required)

Run make test in Github action (CI).

3. What did you see instead (Required)

Failed. See https://github.com/tikv/migration/runs/7741166681?check_suite_focus=true.

=== FAIL: cdc/kv Test (unknown)
Error: 8/09 07:37:51.194 +00:00] [ERROR] [client.go:1041] ["tikv reported the request cluster ID mismatch error, which is not expected"] [tikvCurrentClusterID=0] [requestClusterID=1] [stack="github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).handleError\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:1041\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func4\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:574\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
Error: 8/09 07:37:52.042 +00:00] [ERROR] [region_worker.go:686] ["The CommitTs must be greater than the resolvedTs"] ["Event Type"=COMMIT] [CommitTs=98] [resolvedTs=100] [regionID=3] [stack="github.com/tikv/migration/cdc/cdc/kv.(*regionWorker).handleEventEntry\n\t/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:686\ngithub.com/tikv/migration/cdc/cdc/kv.(*regionWorker).processEvent\n\t/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:386\ngithub.com/tikv/migration/cdc/cdc/kv.(*regionWorker).eventHandler\n\t/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:478\ngithub.com/tikv/migration/cdc/cdc/kv.(*regionWorker).run.func3\n\t/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:601\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
Error: 8/09 07:37:53.772 +00:00] [ERROR] [client.go:1036] ["tikv reported compatibility error, which is not expected"] [rpcCtx="region ID: 3, meta: id:3 region_epoch:<> peers:<id:4 store_id:1 > , peer: id:4 store_id:1 , addr: 127.0.0.1:43357, idx: 0, reqStoreType: TiKvOnly, runStoreType: tikv"] [error="required_version:\"v4.0.7\" "] [stack="github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).handleError\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:1036\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func4\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:574\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
Error: 8/09 07:37:59.937 +00:00] [ERROR] [client.go:1032] ["tikv reported duplicated request to the same region, which is not expected"] [regionID=3] [stack="github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).handleError\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:1032\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func4\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:574\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
Error: 8/09 07:38:10.277 +00:00] [ERROR] [client.go:365] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.0.10 is not supported, the minimal compatible version is 6.2.0-alpha"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.0.10 is not supported, the minimal compatible version is 6.2.0-alpha\ngithub.com/pingcap/errors.AddStack\n\t/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/normalize.go:164\ngithub.com/tikv/migration/cdc/pkg/version.CheckStoreVersion\n\t/home/runner/work/migration/migration/cdc/pkg/version/check.go:177\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:363\ngithub.com/tikv/migration/cdc/pkg/retry.run\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:54\ngithub.com/tikv/migration/cdc/pkg/retry.Do\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:32\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:351\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:747\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:515\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/asm_amd64.s:1571"] [storeID=1] [stack="github.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:365\ngithub.com/tikv/migration/cdc/pkg/retry.run\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:54\ngithub.com/tikv/migration/cdc/pkg/retry.Do\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:32\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:351\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:747\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:515\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
Error: 8/09 07:38:10.386 +00:00] [ERROR] [client.go:365] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.1.0 is not supported, the minimal compatible version is 6.2.0-alpha"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.1.0 is not supported, the minimal compatible version is 6.2.0-alpha\ngithub.com/pingcap/errors.AddStack\n\t/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/normalize.go:164\ngithub.com/tikv/migration/cdc/pkg/version.CheckStoreVersion\n\t/home/runner/work/migration/migration/cdc/pkg/version/check.go:177\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:363\ngithub.com/tikv/migration/cdc/pkg/retry.run\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:54\ngithub.com/tikv/migration/cdc/pkg/retry.Do\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:32\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:351\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:747\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:515\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/asm_amd64.s:1571"] [storeID=1] [stack="github.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:365\ngithub.com/tikv/migration/cdc/pkg/retry.run\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:54\ngithub.com/tikv/migration/cdc/pkg/retry.Do\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:32\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:351\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:747\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:515\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
Error: 8/09 07:38:10.488 +00:00] [ERROR] [client.go:365] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 2.1.10 is not supported, the minimal compatible version is 6.2.0-alpha"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 2.1.10 is not supported, the minimal compatible version is 6.2.0-alpha\ngithub.com/pingcap/errors.AddStack\n\t/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/normalize.go:164\ngithub.com/tikv/migration/cdc/pkg/version.CheckStoreVersion\n\t/home/runner/work/migration/migration/cdc/pkg/version/check.go:177\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:363\ngithub.com/tikv/migration/cdc/pkg/retry.run\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:54\ngithub.com/tikv/migration/cdc/pkg/retry.Do\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:32\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:351\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:747\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:515\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/asm_amd64.s:1571"] [storeID=1] [stack="github.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:365\ngithub.com/tikv/migration/cdc/pkg/retry.run\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:54\ngithub.com/tikv/migration/cdc/pkg/retry.Do\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:32\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:351\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:747\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:515\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
Error: 8/09 07:38:10.589 +00:00] [ERROR] [client.go:365] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.0.10 is not supported, the minimal compatible version is 6.2.0-alpha"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.0.10 is not supported, the minimal compatible version is 6.2.0-alpha\ngithub.com/pingcap/errors.AddStack\n\t/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/normalize.go:164\ngithub.com/tikv/migration/cdc/pkg/version.CheckStoreVersion\n\t/home/runner/work/migration/migration/cdc/pkg/version/check.go:177\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:363\ngithub.com/tikv/migration/cdc/pkg/retry.run\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:54\ngithub.com/tikv/migration/cdc/pkg/retry.Do\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:32\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:351\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:747\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:515\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/asm_amd64.s:1571"] [storeID=1] [stack="github.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:365\ngithub.com/tikv/migration/cdc/pkg/retry.run\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:54\ngithub.com/tikv/migration/cdc/pkg/retry.Do\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:32\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:351\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:747\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:515\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
Error: 8/09 07:38:11.454 +00:00] [ERROR] [client_test.go:1710] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestIncompatibleTiKV.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:1710\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
Error: 8/09 07:39:18.617 +00:00] [ERROR] [client_test.go:2932] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).testKVClientForceReconnect.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:2932\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
Error: 8/09 07:39:19.513 +00:00] [ERROR] [client_test.go:2989] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).testKVClientForceReconnect.func4\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:2989\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
Error: 8/09 07:39:23.544 +00:00] [ERROR] [client_test.go:3416] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestPrewriteNotMatchError.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:3416\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
Error: 8/09 07:39:23.545 +00:00] [ERROR] [client_test.go:3416] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestPrewriteNotMatchError.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:3416\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
panic: close of closed channel

goroutine 3169 [running]:
github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestPrewriteNotMatchError.func1.1()
	/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:3409 +0x5f
github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestPrewriteNotMatchError.func1({0x6f2b870, 0xc000b59ac0})
	/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:3417 +0x34d
github.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1()
	/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253 +0x62
created by github.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed
	/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:252 +0x13b

DONE 225 tests, 1 failure in 519.662s
make: *** [Makefile:119: unit_test_in_verify_ci] Error 1
Error: Process completed with exit code 2.

4. What did you expect to see? (Required)

Succeed.

5. What is your migration tool and TiKV version? (Required)

  • TiKV:
  • TiKV CDC:
  • TiKV BR:
  • TiKV Online Bulk Load:
@pingyu pingyu added the type/bug Something isn't working label Aug 9, 2022
@github-actions
Copy link

github-actions bot commented Sep 9, 2022

This issue is stale because it has been open 30 days with no activity.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
status/stale type/bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants