Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature]: Sort results of getTablets for consistency in output/readability #17771

Merged
merged 2 commits into from
Feb 13, 2025

Conversation

lmorduch
Copy link
Contributor

@lmorduch lmorduch commented Feb 12, 2025

Description

This sorts the results of GetTablets (which for my current use case, will sort the output of ListShardTablets) and allow it to be consistently grokkable due to it's deterministic ordering.

To test this change, I first modified the TestGetTablets unit test under server_test.go from calling testutil.AssertSameTablets(t, tt.expected, resp.Tablets), which first sorts the expected and actual, to utils.MustMatch(t, tt.expected, resp.Tablets), the call made under the hood after sorting in AssertSameTablets. I got the following running go test with just that test change:

--- FAIL: TestGetTablets (0.02s)
    --- FAIL: TestGetTablets/cells_filter (0.00s)
panic: interface conversion: interface {} is []string, not string [recovered]
	panic: interface conversion: interface {} is []string, not string

goroutine 467 [running]:
testing.tRunner.func1.2({0x1062f4160, 0x1400131a810})
	/Users/lmorduchowicz/go/pkg/mod/golang.org/[email protected]/src/testing/testing.go:1632 +0x1bc
testing.tRunner.func1()
	/Users/lmorduchowicz/go/pkg/mod/golang.org/[email protected]/src/testing/testing.go:1635 +0x334
panic({0x1062f4160?, 0x1400131a810?})
	/Users/lmorduchowicz/go/pkg/mod/golang.org/[email protected]/src/runtime/panic.go:785 +0x124
github.com/stretchr/testify/assert.messageFromMsgAndArgs({0x1400110be28?, 0x10650d520?, 0x2?})
	/Users/lmorduchowicz/go/pkg/mod/github.com/stretchr/[email protected]/assert/assertions.go:298 +0x100
github.com/stretchr/testify/assert.Fail({0x10654ce80, 0x14000ec01a0}, {0x105bda086, 0x13}, {0x1400110be28, 0x2, 0x2})
	/Users/lmorduchowicz/go/pkg/mod/github.com/stretchr/[email protected]/assert/assertions.go:364 +0x1ac
github.com/stretchr/testify/assert.FailNow({0x10654ce80, 0x14000ec01a0}, {0x105bda086, 0x13}, {0x1400110be28, 0x2, 0x2})
	/Users/lmorduchowicz/go/pkg/mod/github.com/stretchr/[email protected]/assert/assertions.go:331 +0x8c
github.com/stretchr/testify/require.FailNow({0x106557f50, 0x14000ec01a0}, {0x105bda086, 0x13}, {0x1400110be28, 0x2, 0x2})
	/Users/lmorduchowicz/go/pkg/mod/github.com/stretchr/[email protected]/require/require.go:516 +0xac
vitess.io/vitess/go/test/utils.MustMatchFn.func3(0x14000ec01a0, {0x10621a520, 0x140013106a8}, {0x10621a520, 0x140013106c0}, {0x0, 0x0, 0x0})
	/Users/lmorduchowicz/go/src/vitess2/vitess/go/test/utils/diff.go:71 +0xf4
vitess.io/vitess/go/vt/vtctl/grpcvtctldserver.TestGetTablets.func1(0x14000ec01a0)
	/Users/lmorduchowicz/go/src/vitess2/vitess/go/vt/vtctl/grpcvtctldserver/server_test.go:8280 +0x368
testing.tRunner(0x14000ec01a0, 0x14000eae090)
	/Users/lmorduchowicz/go/pkg/mod/golang.org/[email protected]/src/testing/testing.go:1690 +0xe4
created by testing.(*T).Run in goroutine 208
	/Users/lmorduchowicz/go/pkg/mod/golang.org/[email protected]/src/testing/testing.go:1743 +0x314
exit status 2
FAIL	vitess.io/vitess/go/vt/vtctl/grpcvtctldserver	1.047s

After implementing the actual code changes in server.go, I get:



--- PASS: TestGetTablets (0.01s)
    --- PASS: TestGetTablets/no_tablets (0.00s)
    --- PASS: TestGetTablets/cells_filter_with_single_error_is_fatal_in_strict_mode (0.00s)
    --- PASS: TestGetTablets/tablet_alias_filter_with_none_found (0.00s)
    --- PASS: TestGetTablets/in_nonstrict_mode_if_all_cells_fail_the_request_fails (0.00s)
    --- PASS: TestGetTablets/tablet_type_filter (0.00s)
    --- PASS: TestGetTablets/tablet_alias_filtering (0.00s)
    --- PASS: TestGetTablets/keyspace_and_shard_filter_-_error (0.00s)
    --- PASS: TestGetTablets/cells_filter (0.00s)
    --- PASS: TestGetTablets/stale_primary (0.00s)
    --- PASS: TestGetTablets/cells_filter_with_single_error_is_nonfatal (0.00s)
    --- PASS: TestGetTablets/keyspace,_shard,_and_tablet_type_filter (0.00s)
    --- PASS: TestGetTablets/keyspace_and_shard_filter_-_stale_primary (0.00s)
    --- PASS: TestGetTablets/keyspace_filter (0.00s)
    --- PASS: TestGetTablets/keyspace_and_shard_filter (0.00s)
    --- PASS: TestGetTablets/multiple_cells_with_one_timing_out_and_strict_true (2.00s)
    --- PASS: TestGetTablets/multiple_cells_with_one_timing_out_and_strict_false (2.00s)

Related Issue(s)

This is the related issue: #17744

Checklist

  • "Backport to:" labels have been added if this change should be back-ported to release branches
  • If this change is to be back-ported to previous releases, a justification is included in the PR description
  • Tests were added or are not required
  • Did the new or modified tests pass consistently locally and on CI?
  • Documentation was added or is not required

Deployment Notes

I don't believe this has any special properties with regards to deployments

Copy link
Contributor

vitess-bot bot commented Feb 12, 2025

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

  • Ensure that the Pull Request has a descriptive title.
  • Ensure there is a link to an issue (except for internal cleanup and flaky test fixes), new features should have an RFC that documents use cases and test cases.

Tests

  • Bug fixes should have at least one unit or end-to-end test, enhancement and new features should have a sufficient number of tests.

Documentation

  • Apply the release notes (needs details) label if users need to know about this change.
  • New features should be documented.
  • There should be some code comments as to why things are implemented the way they are.
  • There should be a comment at the top of each new or modified test to explain what the test does.

New flags

  • Is this flag really necessary?
  • Flag names must be clear and intuitive, use dashes (-), and have a clear help text.

If a workflow is added or modified:

  • Each item in Jobs should be named in order to mark it as required.
  • If the workflow needs to be marked as required, the maintainer team must be notified.

Backward compatibility

  • Protobuf changes should be wire-compatible.
  • Changes to _vt tables and RPCs need to be backward compatible.
  • RPC changes should be compatible with vitess-operator
  • If a flag is removed, then it should also be removed from vitess-operator and arewefastyet, if used there.
  • vtctl command output order should be stable and awk-able.

@vitess-bot vitess-bot bot added NeedsBackportReason If backport labels have been applied to a PR, a justification is required NeedsDescriptionUpdate The description is not clear or comprehensive enough, and needs work NeedsIssue A linked issue is missing for this Pull Request NeedsWebsiteDocsUpdate What it says labels Feb 12, 2025
@github-actions github-actions bot added this to the v22.0.0 milestone Feb 12, 2025
@lmorduch
Copy link
Contributor Author

Woops, looks like my commit signing was off. Will fix.

@mattlord mattlord added Type: Enhancement Logical improvement (somewhere between a bug and feature) Component: Cluster management and removed NeedsDescriptionUpdate The description is not clear or comprehensive enough, and needs work NeedsWebsiteDocsUpdate What it says NeedsIssue A linked issue is missing for this Pull Request NeedsBackportReason If backport labels have been applied to a PR, a justification is required labels Feb 12, 2025
Copy link

codecov bot commented Feb 12, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 67.95%. Comparing base (22cf8b6) to head (783eb3a).
Report is 7 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #17771      +/-   ##
==========================================
- Coverage   67.95%   67.95%   -0.01%     
==========================================
  Files        1585     1585              
  Lines      255162   255160       -2     
==========================================
- Hits       173408   173390      -18     
- Misses      81754    81770      +16     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

@mattlord mattlord left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks, @lmorduch ! I thought that we had already done this but it appears that the reality was that I was always thinking we should do this (whenever I noticed output changes in successive GetTablets commands) but I never actually did. 🙂

Do you mind doing one more thing? I think that we should get rid of the AssertSameTablets() function and use utils.MustMatch() instead in the other call sites as well.

❯ git grep AssertSameTablets | cat
go/vt/vtctl/grpcvtctldserver/server_test.go:                            testutil.AssertSameTablets(t, tt.expectedRemainingTablets, resp.Tablets)
go/vt/vtctl/grpcvtctldserver/server_test.go:                    testutil.AssertSameTablets(t, tt.expected, resp.Tablets)
go/vt/vtctl/grpcvtctldserver/server_test.go:                                    testutil.AssertSameTablets(t, tt.expectedTopo, resp.Tablets)
go/vt/vtctl/grpcvtctldserver/testutil/proto_compare.go:func AssertSameTablets(t *testing.T, expected, actual []*topodatapb.Tablet) {

…ed for getTablets now

Signed-off-by: Lucas Morduchowicz <[email protected]>
@lmorduch
Copy link
Contributor Author

LGTM. Thanks, @lmorduch ! I thought that we had already done this but it appears that the reality was that I was always thinking we should do this (whenever I noticed output changes in successive GetTablets commands) but I never actually did. 🙂

Do you mind doing one more thing? I think that we should get rid of the AssertSameTablets() function and use utils.MustMatch() instead in the other call sites as well.

❯ git grep AssertSameTablets | cat
go/vt/vtctl/grpcvtctldserver/server_test.go:                            testutil.AssertSameTablets(t, tt.expectedRemainingTablets, resp.Tablets)
go/vt/vtctl/grpcvtctldserver/server_test.go:                    testutil.AssertSameTablets(t, tt.expected, resp.Tablets)
go/vt/vtctl/grpcvtctldserver/server_test.go:                                    testutil.AssertSameTablets(t, tt.expectedTopo, resp.Tablets)
go/vt/vtctl/grpcvtctldserver/testutil/proto_compare.go:func AssertSameTablets(t *testing.T, expected, actual []*topodatapb.Tablet) {

Thanks a ton for the review and the feedback! My bad re the test function, I should have checked the other calls to assertSameTablets- they all are checking results from getTablets. Updated to remove the helper function and just call utils.MustMatch(). Reran tests in vitess/go/vt/vtctl/grpcvtctldserver with go test -v and got:

--- PASS: TestGetTablets (0.02s)

Full output below.

W0212 13:30:17.867804   10863 log.go:39] Failed to read in config : Config File "vtconfig" Not Found in "[/Users/lmorduchowicz/go/src/vitess/go/vt/vtctl/grpcvtctldserver]". This is optional, and can be ignored if you are not using config files. For a detailed explanation, see https://github.com/vitessio/vitess/blob/main/doc/viper/viper.md#config-files.
W0212 13:30:17.869686   10863 log.go:39] Failed to read in config : Config File "vtconfig" Not Found in "[/Users/lmorduchowicz/go/src/vitess/go/vt/vtctl/grpcvtctldserver]". This is optional, and can be ignored if you are not using config files. For a detailed explanation, see https://github.com/vitessio/vitess/blob/main/doc/viper/viper.md#config-files.
=== RUN   TestRowToSchemaMigration
=== PAUSE TestRowToSchemaMigration
=== RUN   TestValueToVTTime
=== PAUSE TestValueToVTTime
=== RUN   TestValueToVTDuration
=== PAUSE TestValueToVTDuration
=== RUN   TestAlterSchemaMigrationQuery
=== PAUSE TestAlterSchemaMigrationQuery
=== RUN   TestEmergencyReparentShardSlow
=== PAUSE TestEmergencyReparentShardSlow
=== RUN   TestPlannedReparentShardSlow
=== PAUSE TestPlannedReparentShardSlow
=== RUN   TestSleepTablet
=== PAUSE TestSleepTablet
=== RUN   TestPanicHandler
=== PAUSE TestPanicHandler
=== RUN   TestAddCellInfo
=== PAUSE TestAddCellInfo
=== RUN   TestAddCellsAlias
=== PAUSE TestAddCellsAlias
=== RUN   TestApplyRoutingRules
=== PAUSE TestApplyRoutingRules
=== RUN   TestApplyVSchema
=== PAUSE TestApplyVSchema
=== RUN   TestBackup
=== RUN   TestBackup/ok
W0212 13:30:17.875067   10863 test_tmclient.go:441] testutil.TabletManagerClient.Backup faked with no event interval for zone1-0000000100, defaulting to 10ms
W0212 13:30:17.875101   10863 test_tmclient.go:446] testutil.TabletManagerClient.Backup faked with no event jitter for zone1-0000000100, defaulting to 1ms
=== RUN   TestBackup/cannot_backup_primary
=== RUN   TestBackup/allow-primary
W0212 13:30:17.908844   10863 test_tmclient.go:441] testutil.TabletManagerClient.Backup faked with no event interval for zone1-0000000100, defaulting to 10ms
W0212 13:30:17.908863   10863 test_tmclient.go:446] testutil.TabletManagerClient.Backup faked with no event jitter for zone1-0000000100, defaulting to 1ms
=== RUN   TestBackup/no_tablet
E0212 13:30:17.942179   10863 tablet.go:186] unable to connect to tablet "cell:\"zone1\" uid:404": node doesn't exist: tablets/zone1-0000000404/Tablet
=== RUN   TestBackup/midstream_error
W0212 13:30:17.943145   10863 test_tmclient.go:446] testutil.TabletManagerClient.Backup faked with no event jitter for zone1-0000000100, defaulting to 1ms
--- PASS: TestBackup (0.09s)
    --- PASS: TestBackup/ok (0.03s)
    --- PASS: TestBackup/cannot_backup_primary (0.00s)
    --- PASS: TestBackup/allow-primary (0.03s)
    --- PASS: TestBackup/no_tablet (0.00s)
    --- PASS: TestBackup/midstream_error (0.02s)
=== RUN   TestBackupShard
=== RUN   TestBackupShard/ok
W0212 13:30:17.966249   10863 test_tmclient.go:441] testutil.TabletManagerClient.Backup faked with no event interval for zone1-0000000100, defaulting to 10ms
W0212 13:30:17.966321   10863 test_tmclient.go:446] testutil.TabletManagerClient.Backup faked with no event jitter for zone1-0000000100, defaulting to 1ms
=== RUN   TestBackupShard/cannot_backup_primary
=== RUN   TestBackupShard/allow-primary
W0212 13:30:18.000810   10863 test_tmclient.go:441] testutil.TabletManagerClient.Backup faked with no event interval for zone1-0000000100, defaulting to 10ms
W0212 13:30:18.000838   10863 test_tmclient.go:446] testutil.TabletManagerClient.Backup faked with no event jitter for zone1-0000000100, defaulting to 1ms
=== RUN   TestBackupShard/incremental-from-pos
W0212 13:30:18.034978   10863 test_tmclient.go:441] testutil.TabletManagerClient.Backup faked with no event interval for zone1-0000000100, defaulting to 10ms
W0212 13:30:18.035054   10863 test_tmclient.go:446] testutil.TabletManagerClient.Backup faked with no event jitter for zone1-0000000100, defaulting to 1ms
=== RUN   TestBackupShard/no_available_tablet
--- PASS: TestBackupShard (0.10s)
    --- PASS: TestBackupShard/ok (0.03s)
    --- PASS: TestBackupShard/cannot_backup_primary (0.00s)
    --- PASS: TestBackupShard/allow-primary (0.03s)
    --- PASS: TestBackupShard/incremental-from-pos (0.03s)
    --- PASS: TestBackupShard/no_available_tablet (0.00s)
=== RUN   TestCancelSchemaMigration
=== PAUSE TestCancelSchemaMigration
=== RUN   TestChangeTabletTags
=== PAUSE TestChangeTabletTags
=== RUN   TestChangeTabletType
=== PAUSE TestChangeTabletType
=== RUN   TestCleanupSchemaMigration
=== PAUSE TestCleanupSchemaMigration
=== RUN   TestForceCutOverSchemaMigration
=== PAUSE TestForceCutOverSchemaMigration
=== RUN   TestCompleteSchemaMigration
=== PAUSE TestCompleteSchemaMigration
=== RUN   TestCreateKeyspace
=== PAUSE TestCreateKeyspace
=== RUN   TestCreateShard
=== PAUSE TestCreateShard
=== RUN   TestDeleteCellInfo
=== PAUSE TestDeleteCellInfo
=== RUN   TestDeleteCellsAlias
=== PAUSE TestDeleteCellsAlias
=== RUN   TestDeleteKeyspace
=== PAUSE TestDeleteKeyspace
=== RUN   TestDeleteShards
=== PAUSE TestDeleteShards
=== RUN   TestDeleteSrvKeyspace
=== PAUSE TestDeleteSrvKeyspace
=== RUN   TestDeleteTablets
=== PAUSE TestDeleteTablets
=== RUN   TestEmergencyReparentShard
=== PAUSE TestEmergencyReparentShard
=== RUN   TestExecuteFetchAsApp
=== PAUSE TestExecuteFetchAsApp
=== RUN   TestExecuteFetchAsDBA
=== PAUSE TestExecuteFetchAsDBA
=== RUN   TestExecuteMultiFetchAsDBA
=== PAUSE TestExecuteMultiFetchAsDBA
=== RUN   TestExecuteHook
=== PAUSE TestExecuteHook
=== RUN   TestGetUnresolvedTransactions
=== RUN   TestGetUnresolvedTransactions/unresolved_transaction_on_both_shards
=== RUN   TestGetUnresolvedTransactions/unresolved_transaction_on_one_shard
=== RUN   TestGetUnresolvedTransactions/keyspace_not_found
--- PASS: TestGetUnresolvedTransactions (0.00s)
    --- PASS: TestGetUnresolvedTransactions/unresolved_transaction_on_both_shards (0.00s)
    --- PASS: TestGetUnresolvedTransactions/unresolved_transaction_on_one_shard (0.00s)
    --- PASS: TestGetUnresolvedTransactions/keyspace_not_found (0.00s)
=== RUN   TestConcludeTransaction
=== RUN   TestConcludeTransaction/invalid_dtid
=== RUN   TestConcludeTransaction/invalid_transaction_id
=== RUN   TestConcludeTransaction/only_dtid
=== RUN   TestConcludeTransaction/only_dtid_-_empty_metadata
=== RUN   TestConcludeTransaction/only_dtid_-_fail
=== RUN   TestConcludeTransaction/with_participant
=== RUN   TestConcludeTransaction/call_error
--- PASS: TestConcludeTransaction (0.00s)
    --- PASS: TestConcludeTransaction/invalid_dtid (0.00s)
    --- PASS: TestConcludeTransaction/invalid_transaction_id (0.00s)
    --- PASS: TestConcludeTransaction/only_dtid (0.00s)
    --- PASS: TestConcludeTransaction/only_dtid_-_empty_metadata (0.00s)
    --- PASS: TestConcludeTransaction/only_dtid_-_fail (0.00s)
    --- PASS: TestConcludeTransaction/with_participant (0.00s)
    --- PASS: TestConcludeTransaction/call_error (0.00s)
=== RUN   TestGetTransactionInfo
=== RUN   TestGetTransactionInfo/invalid_dtid
=== RUN   TestGetTransactionInfo/invalid_transaction_id
=== RUN   TestGetTransactionInfo/Success
=== RUN   TestGetTransactionInfo/Success_-_empty_metadata
--- PASS: TestGetTransactionInfo (0.00s)
    --- PASS: TestGetTransactionInfo/invalid_dtid (0.00s)
    --- PASS: TestGetTransactionInfo/invalid_transaction_id (0.00s)
    --- PASS: TestGetTransactionInfo/Success (0.00s)
    --- PASS: TestGetTransactionInfo/Success_-_empty_metadata (0.00s)
=== RUN   TestFindAllShardsInKeyspace
=== PAUSE TestFindAllShardsInKeyspace
=== RUN   TestGetBackups
=== RUN   TestGetBackups/no_backupstorage
=== RUN   TestGetBackups/listbackups_error
=== RUN   TestGetBackups/parsing_times_and_aliases
=== RUN   TestGetBackups/limiting
--- PASS: TestGetBackups (0.00s)
    --- PASS: TestGetBackups/no_backupstorage (0.00s)
    --- PASS: TestGetBackups/listbackups_error (0.00s)
    --- PASS: TestGetBackups/parsing_times_and_aliases (0.00s)
    --- PASS: TestGetBackups/limiting (0.00s)
=== RUN   TestGetKeyspace
=== PAUSE TestGetKeyspace
=== RUN   TestGetCellInfoNames
=== PAUSE TestGetCellInfoNames
=== RUN   TestGetCellInfo
=== PAUSE TestGetCellInfo
=== RUN   TestGetCellsAliases
=== PAUSE TestGetCellsAliases
=== RUN   TestGetFullStatus
=== PAUSE TestGetFullStatus
=== RUN   TestGetKeyspaces
=== PAUSE TestGetKeyspaces
=== RUN   TestGetPermissions
=== PAUSE TestGetPermissions
=== RUN   TestGetRoutingRules
=== PAUSE TestGetRoutingRules
=== RUN   TestGetSchema
=== RUN   TestGetSchema/normal_path
=== RUN   TestGetSchema/table_names_only
=== RUN   TestGetSchema/table_sizes_only
=== RUN   TestGetSchema/table_names_take_precedence_over_table_sizes
=== RUN   TestGetSchema/no_tablet
E0212 13:30:18.076709   10863 tablet.go:175] unable to get connection for cell "notfound": node doesn't exist: cells/notfound/CellInfo
=== RUN   TestGetSchema/no_schema
--- PASS: TestGetSchema (0.00s)
    --- PASS: TestGetSchema/normal_path (0.00s)
    --- PASS: TestGetSchema/table_names_only (0.00s)
    --- PASS: TestGetSchema/table_sizes_only (0.00s)
    --- PASS: TestGetSchema/table_names_take_precedence_over_table_sizes (0.00s)
    --- PASS: TestGetSchema/no_tablet (0.00s)
    --- PASS: TestGetSchema/no_schema (0.00s)
=== RUN   TestGetSchemaMigrations
=== PAUSE TestGetSchemaMigrations
=== RUN   TestGetShard
=== PAUSE TestGetShard
=== RUN   TestGetShardReplication
=== PAUSE TestGetShardReplication
=== RUN   TestGetSrvKeyspaceNames
=== PAUSE TestGetSrvKeyspaceNames
=== RUN   TestGetSrvKeyspaces
=== PAUSE TestGetSrvKeyspaces
=== RUN   TestGetSrvVSchema
=== PAUSE TestGetSrvVSchema
=== RUN   TestGetSrvVSchemas
=== PAUSE TestGetSrvVSchemas
=== RUN   TestGetTablet
=== PAUSE TestGetTablet
=== RUN   TestGetTablets
=== PAUSE TestGetTablets
=== RUN   TestGetTopologyPath
=== PAUSE TestGetTopologyPath
=== RUN   TestGetVSchema
=== PAUSE TestGetVSchema
=== RUN   TestLaunchSchemaMigration
=== PAUSE TestLaunchSchemaMigration
=== RUN   TestPingTablet
=== PAUSE TestPingTablet
=== RUN   TestPlannedReparentShard
=== PAUSE TestPlannedReparentShard
=== RUN   TestRebuildKeyspaceGraph
=== PAUSE TestRebuildKeyspaceGraph
=== RUN   TestRebuildVSchemaGraph
=== PAUSE TestRebuildVSchemaGraph
=== RUN   TestRefreshState
=== PAUSE TestRefreshState
=== RUN   TestRefreshStateByShard
=== PAUSE TestRefreshStateByShard
=== RUN   TestReloadSchema
=== PAUSE TestReloadSchema
=== RUN   TestReloadSchemaKeyspace
=== PAUSE TestReloadSchemaKeyspace
=== RUN   TestReloadSchemaShard
=== PAUSE TestReloadSchemaShard
=== RUN   TestRemoveBackup
=== RUN   TestRemoveBackup/ok
=== RUN   TestRemoveBackup/no_bucket_found
=== RUN   TestRemoveBackup/no_backup_found_in_bucket
--- PASS: TestRemoveBackup (0.00s)
    --- PASS: TestRemoveBackup/ok (0.00s)
    --- PASS: TestRemoveBackup/no_bucket_found (0.00s)
    --- PASS: TestRemoveBackup/no_backup_found_in_bucket (0.00s)
=== RUN   TestRemoveKeyspaceCell
=== PAUSE TestRemoveKeyspaceCell
=== RUN   TestRemoveShardCell
=== PAUSE TestRemoveShardCell
=== RUN   TestReparentTablet
=== PAUSE TestReparentTablet
=== RUN   TestRestoreFromBackup
=== RUN   TestRestoreFromBackup/ok
W0212 13:30:18.082887   10863 test_tmclient.go:1164] testutil.TabletManagerClient.RestoreFromBackup faked with no event interval for zone1-0000000100, defaulting to 10ms
W0212 13:30:18.082909   10863 test_tmclient.go:1169] testutil.TabletManagerClient.RestoreFromBackup faked with no event jitter for zone1-0000000100, defaulting to 1ms
=== RUN   TestRestoreFromBackup/no_such_tablet
E0212 13:30:18.116536   10863 tablet.go:175] unable to get connection for cell "zone404": node doesn't exist: cells/zone404/CellInfo
--- PASS: TestRestoreFromBackup (0.03s)
    --- PASS: TestRestoreFromBackup/ok (0.03s)
    --- PASS: TestRestoreFromBackup/no_such_tablet (0.00s)
=== RUN   TestRetrySchemaMigration
=== PAUSE TestRetrySchemaMigration
=== RUN   TestRunHealthCheck
=== PAUSE TestRunHealthCheck
=== RUN   TestSetKeyspaceDurabilityPolicy
=== PAUSE TestSetKeyspaceDurabilityPolicy
=== RUN   TestSetShardIsPrimaryServing
=== PAUSE TestSetShardIsPrimaryServing
=== RUN   TestSetShardTabletControl
=== PAUSE TestSetShardTabletControl
=== RUN   TestSetWritable
=== PAUSE TestSetWritable
=== RUN   TestShardReplicationAdd
=== PAUSE TestShardReplicationAdd
=== RUN   TestShardReplicationPositions
=== PAUSE TestShardReplicationPositions
=== RUN   TestShardReplicationRemove
=== PAUSE TestShardReplicationRemove
=== RUN   TestSourceShardAdd
=== PAUSE TestSourceShardAdd
=== RUN   TestSourceShardDelete
=== PAUSE TestSourceShardDelete
=== RUN   TestStartReplication
=== PAUSE TestStartReplication
=== RUN   TestStopReplication
=== PAUSE TestStopReplication
=== RUN   TestTabletExternallyReparented
=== PAUSE TestTabletExternallyReparented
=== RUN   TestUpdateCellInfo
=== PAUSE TestUpdateCellInfo
=== RUN   TestUpdateCellsAlias
=== PAUSE TestUpdateCellsAlias
=== RUN   TestValidate
=== PAUSE TestValidate
=== RUN   TestValidateSchemaKeyspace
=== RUN   TestValidateSchemaKeyspace/valid_schemas
=== RUN   TestValidateSchemaKeyspace/different_schemas
=== RUN   TestValidateSchemaKeyspace/skip-no-primary:_no_primary
--- PASS: TestValidateSchemaKeyspace (0.00s)
    --- PASS: TestValidateSchemaKeyspace/valid_schemas (0.00s)
    --- PASS: TestValidateSchemaKeyspace/different_schemas (0.00s)
    --- PASS: TestValidateSchemaKeyspace/skip-no-primary:_no_primary (0.00s)
=== RUN   TestValidateVersionKeyspace
=== RUN   TestValidateVersionKeyspace/valid_versions
=== RUN   TestValidateVersionKeyspace/different_versions
--- PASS: TestValidateVersionKeyspace (0.00s)
    --- PASS: TestValidateVersionKeyspace/valid_versions (0.00s)
    --- PASS: TestValidateVersionKeyspace/different_versions (0.00s)
=== RUN   TestValidateVersionShard
=== PAUSE TestValidateVersionShard
=== RUN   TestValidateShard
=== PAUSE TestValidateShard
=== CONT  TestShardReplicationPositions
=== CONT  TestReloadSchemaShard
=== CONT  TestReloadSchemaKeyspace
=== CONT  TestRowToSchemaMigration
=== RUN   TestShardReplicationPositions/success
=== RUN   TestReloadSchemaShard/ok
=== PAUSE TestReloadSchemaShard/ok
=== RUN   TestReloadSchemaShard/shard_not_found
=== PAUSE TestReloadSchemaShard/shard_not_found
=== RUN   TestReloadSchemaShard/include_primary,_with_failure
=== PAUSE TestReloadSchemaShard/include_primary,_with_failure
=== CONT  TestReloadSchema
=== RUN   TestReloadSchema/ok
=== RUN   TestRowToSchemaMigration/#00
=== RUN   TestReloadSchema/tablet_not_found
E0212 13:30:18.127348   10863 tablet.go:186] unable to connect to tablet "cell:\"zone1\" uid:404": node doesn't exist: tablets/zone1-0000000404/Tablet
=== RUN   TestReloadSchemaKeyspace/ok
=== PAUSE TestReloadSchemaKeyspace/ok
=== RUN   TestReloadSchemaKeyspace/keyspace_not_found
=== PAUSE TestReloadSchemaKeyspace/keyspace_not_found
=== CONT  TestSetWritable
=== RUN   TestSetWritable/writable_ok
=== RUN   TestShardReplicationPositions/timeouts_are_nonfatal
=== RUN   TestRowToSchemaMigration/eta_seconds_defaults_to_-1
=== PAUSE TestSetWritable/writable_ok
=== RUN   TestSetWritable/writable_fail
=== PAUSE TestSetWritable/writable_fail
=== RUN   TestSetWritable/read_only_ok
=== PAUSE TestSetWritable/read_only_ok
=== RUN   TestSetWritable/read_only_fail
=== PAUSE TestSetWritable/read_only_fail
=== CONT  TestSetShardIsPrimaryServing
=== CONT  TestSetKeyspaceDurabilityPolicy
=== CONT  TestRunHealthCheck
=== CONT  TestRetrySchemaMigration
=== CONT  TestReparentTablet
=== CONT  TestRemoveShardCell
=== CONT  TestRemoveKeyspaceCell
=== CONT  TestShardReplicationAdd
=== CONT  TestRefreshStateByShard
=== RUN   TestRefreshStateByShard/success
=== CONT  TestSetShardTabletControl
=== RUN   TestRowToSchemaMigration/bad_data
E0212 13:30:18.129293   10863 tablet.go:186] unable to connect to tablet "cell:\"zone1\" uid:404": node doesn't exist: tablets/zone1-0000000404/Tablet
W0212 13:30:18.132528   10863 replication.go:151] Tablet cell:"zone1" uid:404 is in the replication graph, but does not exist, removing it
=== RUN   TestRunHealthCheck/ok
=== PAUSE TestRunHealthCheck/ok
=== RUN   TestRunHealthCheck/no_tablet
=== PAUSE TestRunHealthCheck/no_tablet
=== RUN   TestRunHealthCheck/tmc_call_failed
=== PAUSE TestRunHealthCheck/tmc_call_failed
=== RUN   TestSetWritable/no_such_tablet
=== PAUSE TestSetWritable/no_such_tablet
=== RUN   TestSetShardIsPrimaryServing/ok
=== RUN   TestSetKeyspaceDurabilityPolicy/ok
=== RUN   TestReloadSchema/tmc_failure
=== RUN   TestRemoveShardCell/success
=== RUN   TestRetrySchemaMigration/#00
=== RUN   TestReparentTablet/success
=== RUN   TestRemoveKeyspaceCell/success
=== RUN   TestSetShardTabletControl/ok
=== RUN   TestRefreshStateByShard/cell_filtering
=== CONT  TestRefreshState
--- PASS: TestShardReplicationAdd (0.00s)
=== CONT  TestRebuildVSchemaGraph
=== RUN   TestSetWritable/bad_request
=== PAUSE TestSetKeyspaceDurabilityPolicy/ok
=== PAUSE TestRemoveShardCell/success
=== PAUSE TestReparentTablet/success
=== PAUSE TestRemoveKeyspaceCell/success
--- PASS: TestRowToSchemaMigration (0.02s)
    --- PASS: TestRowToSchemaMigration/#00 (0.00s)
    --- PASS: TestRowToSchemaMigration/eta_seconds_defaults_to_-1 (0.00s)
    --- PASS: TestRowToSchemaMigration/bad_data (0.00s)
=== CONT  TestRebuildKeyspaceGraph
=== PAUSE TestSetWritable/bad_request
=== RUN   TestRebuildVSchemaGraph/success
=== RUN   TestRefreshState/success
=== RUN   TestRebuildKeyspaceGraph/ok
=== PAUSE TestRebuildKeyspaceGraph/ok
=== RUN   TestRebuildKeyspaceGraph/no_such_keyspace
=== PAUSE TestRebuildKeyspaceGraph/no_such_keyspace
=== RUN   TestRemoveShardCell/success/no_tablets
=== RUN   TestRebuildKeyspaceGraph/topo_unavailable
=== PAUSE TestRemoveShardCell/success/no_tablets
=== PAUSE TestRebuildVSchemaGraph/success
=== PAUSE TestRebuildKeyspaceGraph/topo_unavailable
=== RUN   TestRebuildKeyspaceGraph/lock_error
=== PAUSE TestRebuildKeyspaceGraph/lock_error
=== RUN   TestSetKeyspaceDurabilityPolicy/keyspace_not_found
=== RUN   TestSetShardIsPrimaryServing/lock_error
=== PAUSE TestSetKeyspaceDurabilityPolicy/keyspace_not_found
=== RUN   TestSetKeyspaceDurabilityPolicy/fail_to_update_durability_policy
=== PAUSE TestSetKeyspaceDurabilityPolicy/fail_to_update_durability_policy
--- PASS: TestReloadSchema (0.02s)
    --- PASS: TestReloadSchema/ok (0.00s)
    --- PASS: TestReloadSchema/tablet_not_found (0.00s)
    --- PASS: TestReloadSchema/tmc_failure (0.00s)
=== CONT  TestPingTablet
=== CONT  TestLaunchSchemaMigration
=== RUN   TestRetrySchemaMigration/no_shard_primary
=== RUN   TestLaunchSchemaMigration/#00
=== RUN   TestRebuildVSchemaGraph/topo_down
=== RUN   TestSetShardTabletControl/remove_tabletcontrols
=== RUN   TestRemoveShardCell/nonexistent_shard
=== PAUSE TestRemoveShardCell/nonexistent_shard
=== RUN   TestRemoveShardCell/cell_does_not_exist
=== PAUSE TestRemoveShardCell/cell_does_not_exist
=== RUN   TestRemoveShardCell/cell_not_in_serving_list
=== RUN   TestRefreshState/tablet_alias_nil
=== RUN   TestRefreshState/tablet_not_found
=== RUN   TestRefreshStateByShard/partial_result
E0212 13:30:18.147051   10863 tablet.go:186] unable to connect to tablet "cell:\"zone1\" uid:400": node doesn't exist: tablets/zone1-0000000400/Tablet
=== PAUSE TestRebuildVSchemaGraph/topo_down
=== CONT  TestPlannedReparentShard
=== RUN   TestPlannedReparentShard/successful_reparent
=== RUN   TestRefreshState/RefreshState_failed
=== CONT  TestGetTopologyPath
--- PASS: TestSetShardIsPrimaryServing (0.02s)
    --- PASS: TestSetShardIsPrimaryServing/ok (0.00s)
    --- PASS: TestSetShardIsPrimaryServing/lock_error (0.00s)
--- PASS: TestRefreshState (0.00s)
    --- PASS: TestRefreshState/success (0.00s)
    --- PASS: TestRefreshState/tablet_alias_nil (0.00s)
    --- PASS: TestRefreshState/tablet_not_found (0.00s)
    --- PASS: TestRefreshState/RefreshState_failed (0.00s)
=== CONT  TestGetTablet
=== CONT  TestGetVSchema
=== RUN   TestGetVSchema/found
=== PAUSE TestLaunchSchemaMigration/#00
W0212 13:30:18.147676   10863 server.go:3402] RefreshTabletsByShard: failed to refresh zone2-0000000100: assert.AnError general error for testing: RefreshState failed on zone2-100
E0212 13:30:18.147759   10863 schemamanager.go:115] failed to open executor: shard: 80- does not have a primary
E0212 13:30:18.147831   10863 tablet.go:186] unable to connect to tablet "cell:\"cell1\" uid:101": node doesn't exist: tablets/cell1-0000000101/Tablet
=== RUN   TestLaunchSchemaMigration/no_shard_primary
=== PAUSE TestLaunchSchemaMigration/no_shard_primary
=== RUN   TestLaunchSchemaMigration/executeQuery_failure
=== PAUSE TestLaunchSchemaMigration/executeQuery_failure
=== RUN   TestRetrySchemaMigration/executeQuery_failure
=== CONT  TestValidateShard
=== RUN   TestReparentTablet/tablet_is_nil
=== PAUSE TestRemoveShardCell/cell_not_in_serving_list
=== RUN   TestGetTopologyPath/root_path
=== RUN   TestPingTablet/ok
=== CONT  TestGetTablets
=== RUN   TestSetShardTabletControl/disable_queryservice
=== RUN   TestPlannedReparentShard/failed_reparent
--- PASS: TestGetTablet (0.00s)
=== CONT  TestTabletExternallyReparented
=== RUN   TestRemoveKeyspaceCell/success/empty_keyspace
=== PAUSE TestRemoveKeyspaceCell/success/empty_keyspace
=== RUN   TestRemoveKeyspaceCell/keyspace_not_found
=== PAUSE TestRemoveKeyspaceCell/keyspace_not_found
=== RUN   TestRemoveKeyspaceCell/topo_is_down
=== PAUSE TestRemoveKeyspaceCell/topo_is_down
=== RUN   TestRemoveKeyspaceCell/topo_is_locked
=== PAUSE TestGetTopologyPath/root_path
=== RUN   TestGetTopologyPath/invalid_path
=== PAUSE TestReparentTablet/tablet_is_nil
=== RUN   TestReparentTablet/tablet_not_in_topo
=== PAUSE TestGetTopologyPath/invalid_path
=== RUN   TestRefreshStateByShard/missing_keyspace_argument
=== PAUSE TestRemoveKeyspaceCell/topo_is_locked
=== RUN   TestValidateShard/ok
=== RUN   TestTabletExternallyReparented/success
=== RUN   TestPlannedReparentShard/invalid_WaitReplicasTimeout
=== RUN   TestRemoveShardCell/tablets/non-recursive
=== RUN   TestGetTablets/no_tablets
=== RUN   TestGetVSchema/not_found
--- PASS: TestRetrySchemaMigration (0.03s)
    --- PASS: TestRetrySchemaMigration/#00 (0.00s)
    --- PASS: TestRetrySchemaMigration/no_shard_primary (0.00s)
    --- PASS: TestRetrySchemaMigration/executeQuery_failure (0.00s)
=== CONT  TestGetSrvVSchemas
=== RUN   TestPingTablet/tablet_not_found
=== PAUSE TestReparentTablet/tablet_not_in_topo
=== RUN   TestGetTopologyPath/global_path
=== RUN   TestRemoveKeyspaceCell/srvkeyspace_already_deleted
=== PAUSE TestTabletExternallyReparented/success
=== PAUSE TestRemoveShardCell/tablets/non-recursive
=== RUN   TestRemoveShardCell/topo_server_down
=== PAUSE TestRemoveShardCell/topo_server_down
=== RUN   TestRemoveShardCell/cannot_lock_keyspace
=== PAUSE TestRemoveShardCell/cannot_lock_keyspace
E0212 13:30:18.166031   10863 tablet.go:175] unable to get connection for cell "zone2": node doesn't exist: cells/zone2/CellInfo
=== PAUSE TestRemoveKeyspaceCell/srvkeyspace_already_deleted
=== RUN   TestTabletExternallyReparented/tablet_is_nil
=== RUN   TestReparentTablet/shard_not_in_topo
=== PAUSE TestReparentTablet/shard_not_in_topo
=== RUN   TestReparentTablet/shard_has_no_primary
=== PAUSE TestReparentTablet/shard_has_no_primary
=== RUN   TestReparentTablet/shard_primary_not_in_topo
=== PAUSE TestReparentTablet/shard_primary_not_in_topo
=== RUN   TestReparentTablet/shard_primary_is_not_type_PRIMARY
=== PAUSE TestReparentTablet/shard_primary_is_not_type_PRIMARY
=== RUN   TestReparentTablet/shard_primary_is_not_actually_in_shard
=== PAUSE TestReparentTablet/shard_primary_is_not_actually_in_shard
=== PAUSE TestTabletExternallyReparented/tablet_is_nil
=== PAUSE TestGetTopologyPath/global_path
=== CONT  TestValidateVersionShard
=== RUN   TestGetTopologyPath/terminal_data_path
=== PAUSE TestGetTopologyPath/terminal_data_path
=== CONT  TestValidate
=== PAUSE TestGetVSchema/not_found
=== PAUSE TestGetTablets/no_tablets
=== RUN   TestSetShardTabletControl/keyspace_lock_error
=== RUN   TestRefreshStateByShard/missing_shard_argument
=== RUN   TestValidateShard/no_shard
=== RUN   TestPlannedReparentShard/tablet_unreachable
=== RUN   TestPingTablet/ping_rpc_error
=== RUN   TestGetSrvVSchemas/success
=== RUN   TestReparentTablet/requested_tablet_is_shard_primary
=== PAUSE TestReparentTablet/requested_tablet_is_shard_primary
=== RUN   TestReparentTablet/tmc.SetReplicationSource_failure
=== PAUSE TestReparentTablet/tmc.SetReplicationSource_failure
=== RUN   TestReparentTablet/topo_is_down
=== PAUSE TestReparentTablet/topo_is_down
=== RUN   TestValidateShard/no_primary_in_shard
=== RUN   TestValidateVersionShard/valid_versions
=== RUN   TestGetTablets/keyspace_and_shard_filter
=== CONT  TestUpdateCellsAlias
=== PAUSE TestGetTablets/keyspace_and_shard_filter
=== RUN   TestTabletExternallyReparented/topo_is_down
=== PAUSE TestTabletExternallyReparented/topo_is_down
=== CONT  TestGetSrvVSchema
=== RUN   TestUpdateCellsAlias/remove_one_cell
=== PAUSE TestUpdateCellsAlias/remove_one_cell
=== RUN   TestUpdateCellsAlias/add_one_cell
=== PAUSE TestUpdateCellsAlias/add_one_cell
=== RUN   TestUpdateCellsAlias/alias_does_not_exist
=== PAUSE TestUpdateCellsAlias/alias_does_not_exist
=== RUN   TestUpdateCellsAlias/invalid_alias_list
=== PAUSE TestUpdateCellsAlias/invalid_alias_list
=== RUN   TestGetTablets/keyspace_filter
=== PAUSE TestGetTablets/keyspace_filter
=== CONT  TestUpdateCellInfo
=== RUN   TestGetTablets/keyspace_and_shard_filter_-_stale_primary
=== RUN   TestUpdateCellInfo/update
=== PAUSE TestUpdateCellInfo/update
=== RUN   TestUpdateCellInfo/partial_update
=== PAUSE TestUpdateCellInfo/partial_update
--- PASS: TestGetSrvVSchema (0.00s)
=== RUN   TestUpdateCellInfo/no_update
=== PAUSE TestUpdateCellInfo/no_update
=== RUN   TestUpdateCellInfo/cell_not_found
=== PAUSE TestUpdateCellInfo/cell_not_found
=== RUN   TestUpdateCellInfo/cannot_update
=== RUN   TestValidateVersionShard/different_versions
=== PAUSE TestGetTablets/keyspace_and_shard_filter_-_stale_primary
=== RUN   TestGetTablets/stale_primary
=== PAUSE TestGetTablets/stale_primary
=== RUN   TestGetTablets/keyspace_and_shard_filter_-_error
=== PAUSE TestGetTablets/keyspace_and_shard_filter_-_error
--- PASS: TestPingTablet (0.03s)
    --- PASS: TestPingTablet/ok (0.00s)
    --- PASS: TestPingTablet/tablet_not_found (0.00s)
    --- PASS: TestPingTablet/ping_rpc_error (0.00s)
=== RUN   TestGetTablets/cells_filter
=== PAUSE TestGetTablets/cells_filter
=== RUN   TestGetTablets/cells_filter_with_single_error_is_nonfatal
=== PAUSE TestGetTablets/cells_filter_with_single_error_is_nonfatal
=== RUN   TestGetTablets/cells_filter_with_single_error_is_fatal_in_strict_mode
=== PAUSE TestGetTablets/cells_filter_with_single_error_is_fatal_in_strict_mode
=== RUN   TestGetTablets/multiple_cells_with_one_timing_out_and_strict_false
--- PASS: TestValidateVersionShard (0.00s)
    --- PASS: TestValidateVersionShard/valid_versions (0.00s)
    --- PASS: TestValidateVersionShard/different_versions (0.00s)
=== CONT  TestGetSrvKeyspaces
=== RUN   TestGetSrvKeyspaces/success
=== PAUSE TestGetSrvKeyspaces/success
=== RUN   TestGetSrvKeyspaces/filtering_by_cell
=== PAUSE TestGetSrvKeyspaces/filtering_by_cell
=== RUN   TestGetSrvKeyspaces/no_srvkeyspace_for_single_cell
=== PAUSE TestGetSrvKeyspaces/no_srvkeyspace_for_single_cell
=== RUN   TestGetSrvKeyspaces/error_getting_cell_names
=== PAUSE TestGetSrvKeyspaces/error_getting_cell_names
=== PAUSE TestGetTablets/multiple_cells_with_one_timing_out_and_strict_false
=== RUN   TestGetTablets/multiple_cells_with_one_timing_out_and_strict_true
=== PAUSE TestGetTablets/multiple_cells_with_one_timing_out_and_strict_true
=== RUN   TestGetTablets/in_nonstrict_mode_if_all_cells_fail_the_request_fails
=== PAUSE TestGetTablets/in_nonstrict_mode_if_all_cells_fail_the_request_fails
=== RUN   TestGetTablets/tablet_alias_filtering
=== PAUSE TestGetTablets/tablet_alias_filtering
=== RUN   TestGetTablets/tablet_alias_filter_with_none_found
=== PAUSE TestGetTablets/tablet_alias_filter_with_none_found
=== RUN   TestGetTablets/tablet_type_filter
=== PAUSE TestGetTablets/tablet_type_filter
=== RUN   TestGetTablets/keyspace,_shard,_and_tablet_type_filter
=== PAUSE TestGetTablets/keyspace,_shard,_and_tablet_type_filter
=== PAUSE TestUpdateCellInfo/cannot_update
=== CONT  TestStopReplication
=== RUN   TestStopReplication/ok
=== CONT  TestGetShardReplication
=== RUN   TestGetShardReplication/success
=== PAUSE TestGetShardReplication/success
=== RUN   TestGetShardReplication/cell_filtering
=== PAUSE TestGetShardReplication/cell_filtering
=== RUN   TestGetShardReplication/all_cells_topo_down
=== RUN   TestValidateShard/two_primaries_in_shard
=== PAUSE TestStopReplication/ok
=== RUN   TestStopReplication/fail
=== PAUSE TestStopReplication/fail
=== RUN   TestStopReplication/no_such_tablet
=== PAUSE TestStopReplication/no_such_tablet
=== RUN   TestStopReplication/bad_request
=== PAUSE TestStopReplication/bad_request
--- PASS: TestPlannedReparentShard (0.03s)
    --- PASS: TestPlannedReparentShard/successful_reparent (0.00s)
    --- PASS: TestPlannedReparentShard/failed_reparent (0.00s)
    --- PASS: TestPlannedReparentShard/invalid_WaitReplicasTimeout (0.00s)
    --- PASS: TestPlannedReparentShard/tablet_unreachable (0.00s)
=== RUN   TestRefreshStateByShard/shard_not_found
--- PASS: TestRefreshStateByShard (0.05s)
    --- PASS: TestRefreshStateByShard/success (0.00s)
    --- PASS: TestRefreshStateByShard/cell_filtering (0.00s)
    --- PASS: TestRefreshStateByShard/partial_result (0.00s)
    --- PASS: TestRefreshStateByShard/missing_keyspace_argument (0.00s)
    --- PASS: TestRefreshStateByShard/missing_shard_argument (0.00s)
    --- PASS: TestRefreshStateByShard/shard_not_found (0.00s)
=== CONT  TestGetShard
=== RUN   TestGetShard/success
=== PAUSE TestGetShard/success
=== RUN   TestGetShard/shard_not_found
=== PAUSE TestGetShard/shard_not_found
=== RUN   TestGetShard/unavailable_topo_server
=== PAUSE TestGetShard/unavailable_topo_server
=== CONT  TestShardReplicationRemove
=== RUN   TestTabletExternallyReparented/tablet_is_already_primary
=== PAUSE TestTabletExternallyReparented/tablet_is_already_primary
=== RUN   TestTabletExternallyReparented/cannot_change_tablet_type
=== PAUSE TestTabletExternallyReparented/cannot_change_tablet_type
=== RUN   TestGetTopologyPath/terminal_data_path_with_data_as_json
=== PAUSE TestGetTopologyPath/terminal_data_path_with_data_as_json
=== CONT  TestGetSchemaMigrations
=== RUN   TestGetSchemaMigrations/#00
=== PAUSE TestGetSchemaMigrations/#00
=== RUN   TestGetSchemaMigrations/bad_uuid_input
=== PAUSE TestGetSchemaMigrations/bad_uuid_input
=== RUN   TestGetSchemaMigrations/gettablets_failure
=== PAUSE TestGetSchemaMigrations/gettablets_failure
=== RUN   TestGetSchemaMigrations/execute_fetch_failure
=== PAUSE TestGetSchemaMigrations/execute_fetch_failure
=== RUN   TestGetSchemaMigrations/bad_row_data
=== PAUSE TestGetSchemaMigrations/bad_row_data
=== CONT  TestGetSrvKeyspaceNames
=== RUN   TestGetSrvKeyspaceNames/success
=== PAUSE TestGetSrvKeyspaceNames/success
=== RUN   TestGetSrvKeyspaceNames/cell_filtering
=== PAUSE TestGetSrvKeyspaceNames/cell_filtering
=== RUN   TestGetSrvKeyspaceNames/all_cells_topo_down
=== PAUSE TestGetSrvKeyspaceNames/all_cells_topo_down
=== RUN   TestGetSrvKeyspaceNames/cell_filtering_topo_down
=== PAUSE TestGetSrvKeyspaceNames/cell_filtering_topo_down
=== CONT  TestDeleteCellInfo
=== RUN   TestDeleteCellInfo/#00
=== RUN   TestDeleteCellInfo/cell_does_not_exist
--- PASS: TestDeleteCellInfo (0.00s)
    --- PASS: TestDeleteCellInfo/#00 (0.00s)
    --- PASS: TestDeleteCellInfo/cell_does_not_exist (0.00s)
=== CONT  TestCreateKeyspace
=== RUN   TestCreateKeyspace/normal_keyspace
=== PAUSE TestCreateKeyspace/normal_keyspace
=== RUN   TestCreateKeyspace/snapshot_keyspace
=== PAUSE TestCreateKeyspace/snapshot_keyspace
=== RUN   TestCreateKeyspace/snapshot_keyspace_with_no_base_keyspace_specified
=== PAUSE TestCreateKeyspace/snapshot_keyspace_with_no_base_keyspace_specified
=== RUN   TestCreateKeyspace/snapshot_keyspace_with_no_snapshot_time
=== PAUSE TestCreateKeyspace/snapshot_keyspace_with_no_snapshot_time
=== RUN   TestCreateKeyspace/snapshot_keyspace_with_nonexistent_base_keyspace
=== PAUSE TestCreateKeyspace/snapshot_keyspace_with_nonexistent_base_keyspace
=== RUN   TestCreateKeyspace/invalid_keyspace_type
=== PAUSE TestCreateKeyspace/invalid_keyspace_type
=== RUN   TestCreateKeyspace/keyspace_exists/no_force
=== PAUSE TestCreateKeyspace/keyspace_exists/no_force
=== RUN   TestCreateKeyspace/keyspace_exists/force
=== PAUSE TestCreateKeyspace/keyspace_exists/force
=== RUN   TestCreateKeyspace/allow_empty_vschema
=== PAUSE TestCreateKeyspace/allow_empty_vschema
=== RUN   TestCreateKeyspace/keyspace_with_durability_policy_specified
=== CONT  TestCreateShard
=== PAUSE TestCreateKeyspace/keyspace_with_durability_policy_specified
=== CONT  TestSourceShardAdd
=== RUN   TestSourceShardAdd/ok
=== CONT  TestGetKeyspaces
=== NAME  TestValidateShard/two_primaries_in_shard
    util.go:184: shard ks1/- already has a serving primary (zone1-0000000100); replacing with zone1-0000000101 because ForceSetShardPrimary = true
=== PAUSE TestGetShardReplication/all_cells_topo_down
=== RUN   TestGetShardReplication/cell_filtering_topo_down
=== PAUSE TestGetShardReplication/cell_filtering_topo_down
=== PAUSE TestSourceShardAdd/ok
=== RUN   TestSourceShardAdd/uid_already_used
=== PAUSE TestSourceShardAdd/uid_already_used
=== RUN   TestSourceShardAdd/cannot_lock_keyspace
=== PAUSE TestSourceShardAdd/cannot_lock_keyspace
=== CONT  TestForceCutOverSchemaMigration
=== CONT  TestStartReplication
=== CONT  TestGetFullStatus
=== RUN   TestStartReplication/ok
=== PAUSE TestStartReplication/ok
=== RUN   TestStartReplication/fail
=== PAUSE TestStartReplication/fail
=== RUN   TestCreateShard/success
=== PAUSE TestCreateShard/success
=== RUN   TestCreateShard/include_parent
=== RUN   TestGetFullStatus/success
--- PASS: TestShardReplicationRemove (0.00s)
=== CONT  TestGetPermissions
=== PAUSE TestGetSrvVSchemas/success
=== CONT  TestGetRoutingRules
=== CONT  TestCleanupSchemaMigration
=== RUN   TestCleanupSchemaMigration/#00
=== PAUSE TestCleanupSchemaMigration/#00
=== RUN   TestGetRoutingRules/success
=== PAUSE TestGetRoutingRules/success
=== CONT  TestCompleteSchemaMigration
=== RUN   TestCompleteSchemaMigration/#00
=== RUN   TestForceCutOverSchemaMigration/#00
=== PAUSE TestCompleteSchemaMigration/#00
=== RUN   TestCompleteSchemaMigration/no_shard_primary
=== PAUSE TestCompleteSchemaMigration/no_shard_primary
=== PAUSE TestForceCutOverSchemaMigration/#00
=== CONT  TestSourceShardDelete
=== RUN   TestSourceShardDelete/ok
=== PAUSE TestGetFullStatus/success
--- PASS: TestValidate (0.01s)
=== PAUSE TestCreateShard/include_parent
=== RUN   TestGetPermissions/ok
=== RUN   TestGetSrvVSchemas/filtering_by_cell
=== PAUSE TestGetSrvVSchemas/filtering_by_cell
=== RUN   TestGetSrvKeyspaces/error_getting_srvkeyspace
=== RUN   TestCleanupSchemaMigration/no_shard_primary
=== PAUSE TestGetSrvKeyspaces/error_getting_srvkeyspace
=== RUN   TestGetRoutingRules/empty_routing_rules
=== PAUSE TestGetRoutingRules/empty_routing_rules
=== RUN   TestStartReplication/no_such_tablet
=== RUN   TestCompleteSchemaMigration/executeQuery_failure
=== PAUSE TestCompleteSchemaMigration/executeQuery_failure
=== RUN   TestForceCutOverSchemaMigration/no_shard_primary
=== PAUSE TestSourceShardDelete/ok
=== RUN   TestSourceShardDelete/no_SourceShard_with_uid
=== PAUSE TestSourceShardDelete/no_SourceShard_with_uid
=== RUN   TestGetFullStatus/tablet_not_found
=== RUN   TestValidateShard/ping_tablets/ok
--- PASS: TestGetKeyspaces (0.00s)
=== RUN   TestCreateShard/keyspace_does_not_exist
=== PAUSE TestCreateShard/keyspace_does_not_exist
=== PAUSE TestGetPermissions/ok
=== RUN   TestGetPermissions/no_tablet
=== RUN   TestGetSrvVSchemas/no_SrvVSchema_for_single_cell
=== PAUSE TestGetSrvVSchemas/no_SrvVSchema_for_single_cell
=== PAUSE TestCleanupSchemaMigration/no_shard_primary
=== RUN   TestCleanupSchemaMigration/executeQuery_failure
=== CONT  TestGetCellsAliases
=== RUN   TestGetRoutingRules/topo_error
=== PAUSE TestGetRoutingRules/topo_error
=== PAUSE TestStartReplication/no_such_tablet
=== RUN   TestStartReplication/bad_request
=== CONT  TestChangeTabletType
=== RUN   TestChangeTabletType/success
=== PAUSE TestChangeTabletType/success
=== RUN   TestChangeTabletType/dry_run
=== PAUSE TestChangeTabletType/dry_run
=== RUN   TestChangeTabletType/tablet_not_found
=== PAUSE TestStartReplication/bad_request
=== PAUSE TestForceCutOverSchemaMigration/no_shard_primary
--- PASS: TestGetCellsAliases (0.00s)
=== CONT  TestChangeTabletTags
=== RUN   TestChangeTabletTags/success
=== PAUSE TestChangeTabletTags/success
=== RUN   TestChangeTabletTags/success_with_existing
=== PAUSE TestChangeTabletTags/success_with_existing
=== RUN   TestChangeTabletTags/success_with_replace
=== PAUSE TestChangeTabletType/tablet_not_found
=== RUN   TestChangeTabletType/primary_promotions_not_allowed
=== PAUSE TestChangeTabletType/primary_promotions_not_allowed
=== PAUSE TestGetFullStatus/tablet_not_found
=== RUN   TestCreateShard/include_parent/keyspace_exists/no_force
=== CONT  TestGetKeyspace
=== PAUSE TestGetPermissions/no_tablet
=== RUN   TestGetPermissions/tmc_call_failed
=== RUN   TestGetSrvVSchemas/topology_error
=== PAUSE TestCreateShard/include_parent/keyspace_exists/no_force
=== PAUSE TestCleanupSchemaMigration/executeQuery_failure
=== CONT  TestGetCellInfo
=== CONT  TestCancelSchemaMigration
--- PASS: TestGetKeyspace (0.00s)
=== CONT  TestGetCellInfoNames
=== RUN   TestSourceShardDelete/cannot_lock_keyspace
=== PAUSE TestSourceShardDelete/cannot_lock_keyspace
=== CONT  TestApplyRoutingRules
=== RUN   TestApplyRoutingRules/success
=== RUN   TestForceCutOverSchemaMigration/executeQuery_failure
=== PAUSE TestChangeTabletTags/success_with_replace
=== PAUSE TestForceCutOverSchemaMigration/executeQuery_failure
=== RUN   TestChangeTabletTags/tablet_not_found
=== RUN   TestValidateShard/ping_tablets/GetReplicas_failed
=== CONT  TestAddCellsAlias
=== RUN   TestChangeTabletType/primary_demotions_not_allowed
=== PAUSE TestChangeTabletType/primary_demotions_not_allowed
=== RUN   TestChangeTabletType/tabletmanager_failure
=== PAUSE TestChangeTabletType/tabletmanager_failure
=== RUN   TestAddCellsAlias/#00
=== CONT  TestExecuteMultiFetchAsDBA
=== PAUSE TestGetSrvVSchemas/topology_error
=== RUN   TestGetSrvVSchemas/cell_doesn't_exist
=== PAUSE TestGetSrvVSchemas/cell_doesn't_exist
=== RUN   TestExecuteMultiFetchAsDBA/ok
=== PAUSE TestExecuteMultiFetchAsDBA/ok
=== RUN   TestGetSrvVSchemas/one_of_many_cells_doesn't_exist
=== RUN   TestCancelSchemaMigration/#00
=== PAUSE TestCancelSchemaMigration/#00
=== CONT  TestFindAllShardsInKeyspace
--- PASS: TestGetCellInfo (0.00s)
--- PASS: TestGetCellInfoNames (0.00s)
=== RUN   TestApplyRoutingRules/rebuild_failed_(bad_cell)
=== CONT  TestApplyVSchema
=== RUN   TestAddCellsAlias/alias_exists
E0212 13:30:18.178412   10863 srv_vschema.go:228] node doesn't exist: cells/zone2/CellInfo: UpdateSrvVSchema(zone2) failed
=== RUN   TestAddCellsAlias/alias_overlaps
=== RUN   TestApplyRoutingRules/rebuild_skipped
--- PASS: TestAddCellsAlias (0.00s)
    --- PASS: TestAddCellsAlias/#00 (0.00s)
    --- PASS: TestAddCellsAlias/alias_exists (0.00s)
    --- PASS: TestAddCellsAlias/alias_overlaps (0.00s)
=== RUN   TestValidateShard/ping_tablets/no_replicas
=== RUN   TestApplyVSchema/normal
W0212 13:30:18.178510   10863 server.go:190] Skipping rebuild of SrvVSchema, will need to run RebuildVSchemaGraph for changes to take effect
--- PASS: TestFindAllShardsInKeyspace (0.00s)
=== CONT  TestExecuteFetchAsDBA
=== RUN   TestExecuteFetchAsDBA/ok
=== PAUSE TestExecuteFetchAsDBA/ok
=== RUN   TestExecuteFetchAsDBA/tablet_not_found
=== RUN   TestApplyRoutingRules/topo_down
=== PAUSE TestGetPermissions/tmc_call_failed
=== RUN   TestCreateShard/include_parent/keyspace_exists/force
=== RUN   TestExecuteMultiFetchAsDBA/tablet_not_found
=== PAUSE TestGetSrvVSchemas/one_of_many_cells_doesn't_exist
=== RUN   TestCancelSchemaMigration/no_shard_primary
=== CONT  TestExecuteHook
=== CONT  TestAddCellInfo
=== PAUSE TestChangeTabletTags/tablet_not_found
=== RUN   TestValidateShard/ping_tablets/orphaned_replica
=== RUN   TestApplyVSchema/skip_rebuild
=== PAUSE TestExecuteFetchAsDBA/tablet_not_found
=== PAUSE TestCreateShard/include_parent/keyspace_exists/force
=== CONT  TestPanicHandler
=== PAUSE TestExecuteMultiFetchAsDBA/tablet_not_found
--- PASS: TestApplyRoutingRules (0.01s)
    --- PASS: TestApplyRoutingRules/success (0.00s)
    --- PASS: TestApplyRoutingRules/rebuild_failed_(bad_cell) (0.00s)
    --- PASS: TestApplyRoutingRules/rebuild_skipped (0.00s)
    --- PASS: TestApplyRoutingRules/topo_down (0.00s)
=== CONT  TestExecuteFetchAsApp
=== PAUSE TestCancelSchemaMigration/no_shard_primary
=== CONT  TestSleepTablet
=== RUN   TestExecuteHook/ok
=== RUN   TestAddCellInfo/#00
=== RUN   TestExecuteFetchAsDBA/query_error
=== PAUSE TestExecuteFetchAsDBA/query_error
=== RUN   TestCreateShard/shard_exists/no_force
=== PAUSE TestCreateShard/shard_exists/no_force
=== RUN   TestCreateShard/shard_exists/force
=== CONT  TestEmergencyReparentShard
=== RUN   TestEmergencyReparentShard/successful_reparent
=== PAUSE TestCreateShard/shard_exists/force
=== RUN   TestCreateShard/topo_is_down
=== PAUSE TestCreateShard/topo_is_down
=== CONT  TestPlannedReparentShardSlow
=== RUN   TestPlannedReparentShardSlow/nil_WaitReplicasTimeout_and_request_takes_29_seconds_is_ok
=== PAUSE TestPlannedReparentShardSlow/nil_WaitReplicasTimeout_and_request_takes_29_seconds_is_ok
=== RUN   TestPlannedReparentShardSlow/nil_WaitReplicasTimeout_and_request_takes_31_seconds_is_error
=== PAUSE TestPlannedReparentShardSlow/nil_WaitReplicasTimeout_and_request_takes_31_seconds_is_error
=== CONT  TestDeleteTablets
=== RUN   TestDeleteTablets/single_replica
=== PAUSE TestDeleteTablets/single_replica
=== RUN   TestDeleteTablets/single_primary/no_AllowPrimary
=== PAUSE TestDeleteTablets/single_primary/no_AllowPrimary
=== RUN   TestDeleteTablets/single_primary/with_AllowPrimary
=== PAUSE TestDeleteTablets/single_primary/with_AllowPrimary
=== RUN   TestDeleteTablets/multiple_tablets
=== PAUSE TestDeleteTablets/multiple_tablets
=== RUN   TestDeleteTablets/stale_primary_record
=== PAUSE TestDeleteTablets/stale_primary_record
=== RUN   TestDeleteTablets/tablet_not_found
=== PAUSE TestDeleteTablets/tablet_not_found
=== RUN   TestDeleteTablets/shard_is_locked
=== PAUSE TestDeleteTablets/shard_is_locked
=== RUN   TestDeleteTablets/another_shard_is_locked
=== PAUSE TestDeleteTablets/another_shard_is_locked
=== RUN   TestDeleteTablets/topo_server_is_down
=== PAUSE TestDeleteTablets/topo_server_is_down
=== CONT  TestEmergencyReparentShardSlow
=== RUN   TestEmergencyReparentShardSlow/nil_WaitReplicasTimeout_and_request_takes_29_seconds_is_ok
=== PAUSE TestEmergencyReparentShardSlow/nil_WaitReplicasTimeout_and_request_takes_29_seconds_is_ok
=== RUN   TestEmergencyReparentShardSlow/nil_WaitReplicasTimeout_and_request_takes_31_seconds_is_error
=== PAUSE TestEmergencyReparentShardSlow/nil_WaitReplicasTimeout_and_request_takes_31_seconds_is_error
=== CONT  TestDeleteSrvKeyspace
=== RUN   TestDeleteSrvKeyspace/success
=== RUN   TestSleepTablet/ok
=== RUN   TestExecuteMultiFetchAsDBA/query_error
=== PAUSE TestExecuteMultiFetchAsDBA/query_error
=== RUN   TestAddCellInfo/cell_already_exists
=== PAUSE TestDeleteSrvKeyspace/success
=== RUN   TestDeleteSrvKeyspace/cell_not_found
=== PAUSE TestDeleteSrvKeyspace/cell_not_found
=== RUN   TestDeleteSrvKeyspace/empty_cell_argument
=== PAUSE TestDeleteSrvKeyspace/empty_cell_argument
=== CONT  TestAlterSchemaMigrationQuery
=== RUN   TestAlterSchemaMigrationQuery/cleanup_4e5dcf80_354b_11eb_82cd_f875a4d24e90
=== PAUSE TestAlterSchemaMigrationQuery/cleanup_4e5dcf80_354b_11eb_82cd_f875a4d24e90
=== RUN   TestAlterSchemaMigrationQuery/cancel_4e5dcf80_354b_11eb_82cd_f875a4d24e90
=== PAUSE TestAlterSchemaMigrationQuery/cancel_4e5dcf80_354b_11eb_82cd_f875a4d24e90
=== RUN   TestAlterSchemaMigrationQuery/cancel_all
=== PAUSE TestAlterSchemaMigrationQuery/cancel_all
=== RUN   TestAlterSchemaMigrationQuery/cancel_ALL
=== PAUSE TestAlterSchemaMigrationQuery/cancel_ALL
=== CONT  TestDeleteKeyspace
=== RUN   TestDeleteKeyspace/success
=== CONT  TestValueToVTDuration
=== RUN   TestValueToVTDuration/#00
=== RUN   TestCancelSchemaMigration/executeQuery_failure
=== PAUSE TestCancelSchemaMigration/executeQuery_failure
--- PASS: TestPanicHandler (0.00s)
=== RUN   TestValidateShard/ping_tablets/Ping_failed
=== CONT  TestValueToVTTime
=== RUN   TestValueToVTTime/#00
=== PAUSE TestValueToVTTime/#00
=== RUN   TestValueToVTTime/empty_string
=== PAUSE TestValueToVTTime/empty_string
=== RUN   TestChangeTabletTags/tabletmanager_failure
=== RUN   TestAddCellInfo/no_cell_root
=== PAUSE TestChangeTabletTags/tabletmanager_failure
=== CONT  TestDeleteShards
=== CONT  TestDeleteCellsAlias
=== RUN   TestDeleteShards/success
=== PAUSE TestDeleteShards/success
=== RUN   TestExecuteFetchAsApp/ok
=== PAUSE TestExecuteFetchAsApp/ok
=== RUN   TestExecuteFetchAsApp/tablet_not_found
=== RUN   TestApplyVSchema/both
=== RUN   TestDeleteCellsAlias/#00
--- PASS: TestAddCellInfo (0.01s)
    --- PASS: TestAddCellInfo/#00 (0.00s)
    --- PASS: TestAddCellInfo/cell_already_exists (0.00s)
    --- PASS: TestAddCellInfo/no_cell_root (0.00s)
=== RUN   TestValueToVTTime/parse_error
=== RUN   TestValueToVTDuration/#01
=== RUN   TestDeleteKeyspace/keyspace_does_not_exist
=== PAUSE TestValueToVTTime/parse_error
=== RUN   TestDeleteShards/shard_not_found
=== RUN   TestDeleteCellsAlias/alias_does_not_exist
=== PAUSE TestDeleteShards/shard_not_found
=== RUN   TestDeleteShards/multiple_shards
=== PAUSE TestDeleteShards/multiple_shards
=== PAUSE TestExecuteFetchAsApp/tablet_not_found
=== RUN   TestExecuteFetchAsApp/query_error
=== PAUSE TestExecuteFetchAsApp/query_error
--- PASS: TestDeleteCellsAlias (0.00s)
    --- PASS: TestDeleteCellsAlias/#00 (0.00s)
    --- PASS: TestDeleteCellsAlias/alias_does_not_exist (0.00s)
=== RUN   TestSleepTablet/default_sleep_duration
=== CONT  TestReloadSchemaKeyspace/ok
=== CONT  TestReloadSchemaShard/ok
=== RUN   TestValueToVTDuration/no_unit_in_value
--- PASS: TestValidateShard (0.04s)
    --- PASS: TestValidateShard/ok (0.00s)
    --- PASS: TestValidateShard/no_shard (0.00s)
    --- PASS: TestValidateShard/no_primary_in_shard (0.00s)
    --- PASS: TestValidateShard/two_primaries_in_shard (0.00s)
    --- PASS: TestValidateShard/ping_tablets/ok (0.00s)
    --- PASS: TestValidateShard/ping_tablets/GetReplicas_failed (0.00s)
    --- PASS: TestValidateShard/ping_tablets/no_replicas (0.00s)
    --- PASS: TestValidateShard/ping_tablets/orphaned_replica (0.00s)
    --- PASS: TestValidateShard/ping_tablets/Ping_failed (0.00s)
=== CONT  TestReloadSchemaKeyspace/keyspace_not_found
=== RUN   TestValueToVTDuration/empty
=== CONT  TestRunHealthCheck/ok
=== RUN   TestValueToVTDuration/bad_input
--- PASS: TestValueToVTDuration (0.00s)
    --- PASS: TestValueToVTDuration/#00 (0.00s)
    --- PASS: TestValueToVTDuration/#01 (0.00s)
    --- PASS: TestValueToVTDuration/no_unit_in_value (0.00s)
    --- PASS: TestValueToVTDuration/empty (0.00s)
    --- PASS: TestValueToVTDuration/bad_input (0.00s)
=== CONT  TestReloadSchemaShard/shard_not_found
=== CONT  TestReloadSchemaShard/include_primary,_with_failure
=== CONT  TestRunHealthCheck/no_tablet
=== RUN   TestDeleteShards/topo_is_down
=== PAUSE TestDeleteShards/topo_is_down
=== RUN   TestExecuteHook/nil_hook_request
=== CONT  TestRunHealthCheck/tmc_call_failed
E0212 13:30:18.194250   10863 tablet.go:186] unable to connect to tablet "cell:\"zone1\" uid:100": node doesn't exist: tablets/zone1-0000000100/Tablet
=== RUN   TestExecuteHook/hook_with_slash
--- PASS: TestRunHealthCheck (0.00s)
    --- PASS: TestRunHealthCheck/ok (0.00s)
    --- PASS: TestRunHealthCheck/no_tablet (0.00s)
    --- PASS: TestRunHealthCheck/tmc_call_failed (0.00s)
=== RUN   TestDeleteShards/shard_is_serving/EvenIfServing=false
=== RUN   TestApplyVSchema/neither
=== CONT  TestRebuildKeyspaceGraph/ok
--- PASS: TestReloadSchemaShard (0.00s)
    --- PASS: TestReloadSchemaShard/shard_not_found (0.00s)
    --- PASS: TestReloadSchemaShard/ok (0.00s)
    --- PASS: TestReloadSchemaShard/include_primary,_with_failure (0.00s)
=== CONT  TestRebuildKeyspaceGraph/topo_unavailable
=== CONT  TestSetWritable/writable_ok
=== RUN   TestExecuteHook/no_such_tablet
=== RUN   TestEmergencyReparentShard/failed_reparent
=== RUN   TestDeleteKeyspace/keyspace_has_shards/Recursive=false
=== CONT  TestRebuildKeyspaceGraph/no_such_keyspace
=== CONT  TestSetWritable/bad_request
=== CONT  TestSetKeyspaceDurabilityPolicy/keyspace_not_found
=== PAUSE TestDeleteShards/shard_is_serving/EvenIfServing=false
E0212 13:30:18.194886   10863 tablet.go:186] unable to connect to tablet "cell:\"zone1\" uid:404": node doesn't exist: tablets/zone1-0000000404/Tablet
=== CONT  TestRebuildKeyspaceGraph/lock_error
=== RUN   TestDeleteShards/shard_is_serving/EvenIfServing=true
=== PAUSE TestDeleteShards/shard_is_serving/EvenIfServing=true
=== CONT  TestSetWritable/read_only_fail
=== CONT  TestSetKeyspaceDurabilityPolicy/ok
=== CONT  TestSetKeyspaceDurabilityPolicy/fail_to_update_durability_policy
=== CONT  TestSetWritable/read_only_ok
E0212 13:30:18.199258   10863 server.go:3984] SetWritable: failed to set writable=false on zone1-0000000100: assert.AnError general error for testing
=== CONT  TestLaunchSchemaMigration/#00
=== CONT  TestSetWritable/no_such_tablet
--- PASS: TestReloadSchemaKeyspace (0.00s)
    --- PASS: TestReloadSchemaKeyspace/keyspace_not_found (0.00s)
    --- PASS: TestReloadSchemaKeyspace/ok (0.00s)
=== CONT  TestSetWritable/writable_fail
E0212 13:30:18.202798   10863 tablet.go:175] unable to get connection for cell "zone2": node doesn't exist: cells/zone2/CellInfo
E0212 13:30:18.202842   10863 server.go:3971] SetWritable: failed to read tablet record for zone2-0000000200: node doesn't exist: cells/zone2/CellInfo
=== CONT  TestLaunchSchemaMigration/no_shard_primary
=== RUN   TestDeleteShards/ShardReplication_in_topo
=== PAUSE TestDeleteShards/ShardReplication_in_topo
=== RUN   TestDeleteShards/shard_has_tablets/Recursive=false
=== PAUSE TestDeleteShards/shard_has_tablets/Recursive=false
E0212 13:30:18.203075   10863 server.go:3984] SetWritable: failed to set writable=true on zone1-0000000100: assert.AnError general error for testing
E0212 13:30:18.203192   10863 schemamanager.go:115] failed to open executor: shard: 80- does not have a primary
=== CONT  TestRebuildVSchemaGraph/success
=== CONT  TestRebuildVSchemaGraph/topo_down
=== CONT  TestLaunchSchemaMigration/executeQuery_failure
=== RUN   TestApplyVSchema/unknown_params
=== RUN   TestEmergencyReparentShard/invalid_WaitReplicasTimeout
=== RUN   TestExecuteHook/tablet_hook_failure
--- PASS: TestRebuildKeyspaceGraph (0.00s)
    --- PASS: TestRebuildKeyspaceGraph/ok (0.00s)
    --- PASS: TestRebuildKeyspaceGraph/topo_unavailable (0.00s)
    --- PASS: TestRebuildKeyspaceGraph/no_such_keyspace (0.00s)
    --- PASS: TestRebuildKeyspaceGraph/lock_error (0.00s)
=== RUN   TestDeleteKeyspace/keyspace_has_shards/Recursive=true
=== CONT  TestRemoveShardCell/success
=== RUN   TestDeleteShards/shard_has_tablets/Recursive=true
=== PAUSE TestDeleteShards/shard_has_tablets/Recursive=true
=== RUN   TestDeleteShards/tablets_in_topo_belonging_to_other_shard
=== PAUSE TestDeleteShards/tablets_in_topo_belonging_to_other_shard
=== RUN   TestDeleteShards/shard_is_locked
=== PAUSE TestDeleteShards/shard_is_locked
=== RUN   TestDeleteShards/shard_is_locked_with_force
=== PAUSE TestDeleteShards/shard_is_locked_with_force
=== CONT  TestRemoveKeyspaceCell/success
E0212 13:30:18.207287   10863 tablet.go:186] unable to connect to tablet "cell:\"zone2\" uid:200": node doesn't exist: tablets/zone2-0000000200/Tablet
=== CONT  TestRemoveShardCell/cannot_lock_keyspace
--- PASS: TestEmergencyReparentShard (0.02s)
    --- PASS: TestEmergencyReparentShard/successful_reparent (0.00s)
    --- PASS: TestEmergencyReparentShard/failed_reparent (0.00s)
    --- PASS: TestEmergencyReparentShard/invalid_WaitReplicasTimeout (0.00s)
--- PASS: TestExecuteHook (0.03s)
    --- PASS: TestExecuteHook/ok (0.00s)
    --- PASS: TestExecuteHook/nil_hook_request (0.00s)
    --- PASS: TestExecuteHook/hook_with_slash (0.00s)
    --- PASS: TestExecuteHook/no_such_tablet (0.00s)
    --- PASS: TestExecuteHook/tablet_hook_failure (0.00s)
=== CONT  TestRemoveKeyspaceCell/topo_is_locked
=== CONT  TestRemoveShardCell/topo_server_down
=== CONT  TestRemoveShardCell/tablets/non-recursive
=== CONT  TestRemoveShardCell/cell_not_in_serving_list
=== CONT  TestRemoveShardCell/cell_does_not_exist
=== CONT  TestRemoveShardCell/nonexistent_shard
=== CONT  TestRemoveShardCell/success/no_tablets
=== CONT  TestRemoveKeyspaceCell/srvkeyspace_already_deleted
--- PASS: TestSetKeyspaceDurabilityPolicy (0.02s)
    --- PASS: TestSetKeyspaceDurabilityPolicy/keyspace_not_found (0.00s)
    --- PASS: TestSetKeyspaceDurabilityPolicy/fail_to_update_durability_policy (0.00s)
    --- PASS: TestSetKeyspaceDurabilityPolicy/ok (0.00s)
=== CONT  TestRemoveKeyspaceCell/topo_is_down
--- PASS: TestRebuildVSchemaGraph (0.00s)
    --- PASS: TestRebuildVSchemaGraph/topo_down (0.00s)
    --- PASS: TestRebuildVSchemaGraph/success (0.00s)
--- PASS: TestLaunchSchemaMigration (0.00s)
    --- PASS: TestLaunchSchemaMigration/#00 (0.00s)
    --- PASS: TestLaunchSchemaMigration/executeQuery_failure (0.00s)
    --- PASS: TestLaunchSchemaMigration/no_shard_primary (0.01s)
=== RUN   TestApplyVSchema/strict_unknown_params
--- PASS: TestSetWritable (0.02s)
    --- PASS: TestSetWritable/writable_ok (0.00s)
    --- PASS: TestSetWritable/bad_request (0.00s)
    --- PASS: TestSetWritable/read_only_fail (0.00s)
    --- PASS: TestSetWritable/read_only_ok (0.00s)
    --- PASS: TestSetWritable/no_such_tablet (0.00s)
    --- PASS: TestSetWritable/writable_fail (0.01s)
=== RUN   TestDeleteKeyspace/topo_error
=== CONT  TestReparentTablet/shard_primary_is_not_type_PRIMARY
=== CONT  TestRemoveKeyspaceCell/success/empty_keyspace
=== RUN   TestDeleteKeyspace/keyspace_is_locked
=== RUN   TestApplyVSchema/dry_run
=== CONT  TestReparentTablet/success
=== CONT  TestReparentTablet/shard_primary_not_in_topo
=== CONT  TestGetVSchema/not_found
=== CONT  TestReparentTablet/shard_has_no_primary
=== CONT  TestReparentTablet/topo_is_down
=== CONT  TestReparentTablet/tmc.SetReplicationSource_failure
=== CONT  TestReparentTablet/requested_tablet_is_shard_primary
=== CONT  TestReparentTablet/shard_primary_is_not_actually_in_shard
=== CONT  TestRemoveKeyspaceCell/keyspace_not_found
--- PASS: TestRemoveShardCell (0.04s)
    --- PASS: TestRemoveShardCell/success (0.01s)
    --- PASS: TestRemoveShardCell/nonexistent_shard (0.00s)
    --- PASS: TestRemoveShardCell/cannot_lock_keyspace (0.00s)
    --- PASS: TestRemoveShardCell/cell_does_not_exist (0.00s)
    --- PASS: TestRemoveShardCell/cell_not_in_serving_list (0.00s)
    --- PASS: TestRemoveShardCell/topo_server_down (0.00s)
    --- PASS: TestRemoveShardCell/tablets/non-recursive (0.00s)
    --- PASS: TestRemoveShardCell/success/no_tablets (0.00s)
--- PASS: TestGetVSchema (0.02s)
    --- PASS: TestGetVSchema/found (0.00s)
    --- PASS: TestGetVSchema/not_found (0.00s)
=== CONT  TestReparentTablet/tablet_not_in_topo
=== CONT  TestReparentTablet/tablet_is_nil
=== RUN   TestDeleteKeyspace/keyspace_is_locked_with_force
=== RUN   TestApplyVSchema/dry_run_with_invalid_params
W0212 13:30:18.222980   10863 server.go:1125] lock for keyspace testkeyspace is already held: failed to lock keyspace testkeyspace for deletion, but force=true, proceeding anyway ...
--- PASS: TestDeleteKeyspace (0.03s)
    --- PASS: TestDeleteKeyspace/success (0.00s)
    --- PASS: TestDeleteKeyspace/keyspace_does_not_exist (0.00s)
    --- PASS: TestDeleteKeyspace/keyspace_has_shards/Recursive=false (0.00s)
    --- PASS: TestDeleteKeyspace/keyspace_has_shards/Recursive=true (0.00s)
    --- PASS: TestDeleteKeyspace/topo_error (0.00s)
    --- PASS: TestDeleteKeyspace/keyspace_is_locked (0.00s)
    --- PASS: TestDeleteKeyspace/keyspace_is_locked_with_force (0.00s)
=== RUN   TestApplyVSchema/dry_run_with_unknown_params
=== RUN   TestApplyVSchema/strict_dry_run_with_unknown_params
=== CONT  TestUpdateCellsAlias/invalid_alias_list
=== CONT  TestUpdateCellsAlias/alias_does_not_exist
E0212 13:30:18.223276   10863 tablet.go:175] unable to get connection for cell "zone1": assert.AnError general error for testing
=== CONT  TestUpdateCellInfo/update
=== CONT  TestGetTablets/tablet_type_filter
=== CONT  TestUpdateCellsAlias/add_one_cell
E0212 13:30:18.223682   10863 tablet.go:186] unable to connect to tablet "cell:\"zone3\" uid:300": node doesn't exist: tablets/zone3-0000000300/Tablet
=== CONT  TestGetTablets/tablet_alias_filter_with_none_found
=== CONT  TestGetTablets/tablet_alias_filtering
E0212 13:30:18.223719   10863 tablet.go:186] unable to connect to tablet "cell:\"zone1\" uid:100": node doesn't exist: tablets/zone1-0000000100/Tablet
=== CONT  TestReparentTablet/shard_not_in_topo
=== CONT  TestGetTablets/multiple_cells_with_one_timing_out_and_strict_false
=== CONT  TestGetTablets/cells_filter_with_single_error_is_fatal_in_strict_mode
E0212 13:30:18.228863   10863 tablet.go:175] unable to get connection for cell "zone1": node doesn't exist: cells/zone1/CellInfo
--- PASS: TestRemoveKeyspaceCell (0.04s)
    --- PASS: TestRemoveKeyspaceCell/success (0.00s)
    --- PASS: TestRemoveKeyspaceCell/topo_is_locked (0.00s)
    --- PASS: TestRemoveKeyspaceCell/srvkeyspace_already_deleted (0.00s)
    --- PASS: TestRemoveKeyspaceCell/topo_is_down (0.00s)
    --- PASS: TestRemoveKeyspaceCell/success/empty_keyspace (0.00s)
    --- PASS: TestRemoveKeyspaceCell/keyspace_not_found (0.00s)
--- PASS: TestApplyVSchema (0.06s)
    --- PASS: TestApplyVSchema/normal (0.00s)
    --- PASS: TestApplyVSchema/skip_rebuild (0.00s)
    --- PASS: TestApplyVSchema/both (0.00s)
    --- PASS: TestApplyVSchema/neither (0.00s)
    --- PASS: TestApplyVSchema/unknown_params (0.00s)
    --- PASS: TestApplyVSchema/strict_unknown_params (0.00s)
    --- PASS: TestApplyVSchema/dry_run (0.00s)
    --- PASS: TestApplyVSchema/dry_run_with_invalid_params (0.00s)
    --- PASS: TestApplyVSchema/dry_run_with_unknown_params (0.00s)
    --- PASS: TestApplyVSchema/strict_dry_run_with_unknown_params (0.00s)
W0212 13:30:18.238199   10863 tablet.go:505] cell:"zone1" uid:101: node doesn't exist: cells/zone1/CellInfo
E0212 13:30:18.228874   10863 tablet.go:175] unable to get connection for cell "zone404": node doesn't exist: cells/zone404/CellInfo
W0212 13:30:18.241502   10863 tablet.go:505] cell:"zone404" uid:404: node doesn't exist: cells/zone404/CellInfo
=== CONT  TestGetTablets/no_tablets
=== CONT  TestUpdateCellsAlias/remove_one_cell
=== CONT  TestGetTablets/keyspace_and_shard_filter_-_error
=== CONT  TestGetTablets/keyspace,_shard,_and_tablet_type_filter
=== CONT  TestGetTablets/in_nonstrict_mode_if_all_cells_fail_the_request_fails
=== CONT  TestGetTablets/multiple_cells_with_one_timing_out_and_strict_true
=== CONT  TestGetTablets/cells_filter_with_single_error_is_nonfatal
=== CONT  TestGetTablets/cells_filter
=== CONT  TestGetTablets/stale_primary
=== CONT  TestGetTablets/keyspace_filter
=== CONT  TestGetTablets/keyspace_and_shard_filter
=== CONT  TestGetTablets/keyspace_and_shard_filter_-_stale_primary
=== CONT  TestUpdateCellInfo/cell_not_found
--- PASS: TestReparentTablet (0.04s)
    --- PASS: TestReparentTablet/shard_primary_is_not_type_PRIMARY (0.00s)
    --- PASS: TestReparentTablet/success (0.00s)
    --- PASS: TestReparentTablet/shard_has_no_primary (0.00s)
    --- PASS: TestReparentTablet/requested_tablet_is_shard_primary (0.00s)
    --- PASS: TestReparentTablet/shard_primary_is_not_actually_in_shard (0.00s)
    --- PASS: TestReparentTablet/tablet_is_nil (0.00s)
    --- PASS: TestReparentTablet/tmc.SetReplicationSource_failure (0.00s)
    --- PASS: TestReparentTablet/topo_is_down (0.01s)
    --- PASS: TestReparentTablet/shard_primary_not_in_topo (0.01s)
    --- PASS: TestReparentTablet/tablet_not_in_topo (0.01s)
    --- PASS: TestReparentTablet/shard_not_in_topo (0.00s)
--- PASS: TestUpdateCellsAlias (0.00s)
    --- PASS: TestUpdateCellsAlias/invalid_alias_list (0.00s)
    --- PASS: TestUpdateCellsAlias/alias_does_not_exist (0.00s)
    --- PASS: TestUpdateCellsAlias/add_one_cell (0.00s)
    --- PASS: TestUpdateCellsAlias/remove_one_cell (0.00s)
=== CONT  TestUpdateCellInfo/no_update
=== CONT  TestStopReplication/fail
=== CONT  TestGetTopologyPath/root_path
=== CONT  TestUpdateCellInfo/partial_update
=== CONT  TestGetSchemaMigrations/#00
E0212 13:30:18.251613   10863 server.go:4409] StopReplication: failed to stop replication on zone1-0000000100: assert.AnError general error for testing
=== CONT  TestTabletExternallyReparented/tablet_is_already_primary
=== CONT  TestUpdateCellInfo/cannot_update
--- PASS: TestUpdateCellInfo (0.00s)
    --- PASS: TestUpdateCellInfo/update (0.00s)
    --- PASS: TestUpdateCellInfo/cell_not_found (0.00s)
    --- PASS: TestUpdateCellInfo/no_update (0.00s)
    --- PASS: TestUpdateCellInfo/partial_update (0.00s)
    --- PASS: TestUpdateCellInfo/cannot_update (0.00s)
=== CONT  TestStopReplication/ok
=== CONT  TestTabletExternallyReparented/cannot_change_tablet_type
=== CONT  TestGetShard/success
W0212 13:30:18.252969   10863 server.go:4485] ChangeType(zone2-0000000200, PRIMARY): assert.AnError general error for testing
=== CONT  TestTabletExternallyReparented/topo_is_down
=== CONT  TestTabletExternallyReparented/success
=== CONT  TestStopReplication/bad_request
=== CONT  TestStopReplication/no_such_tablet
=== CONT  TestTabletExternallyReparented/tablet_is_nil
=== CONT  TestGetSrvKeyspaceNames/success
=== CONT  TestGetShard/unavailable_topo_server
=== CONT  TestGetShard/shard_not_found
=== CONT  TestGetSchemaMigrations/gettablets_failure
=== CONT  TestGetTopologyPath/terminal_data_path_with_data_as_json
=== CONT  TestGetTopologyPath/terminal_data_path
E0212 13:30:18.259172   10863 tablet.go:175] unable to get connection for cell "zone2": assert.AnError general error for testing
W0212 13:30:18.262507   10863 server.go:4432] TabletExternallyReparented: failed to read tablet record for zone2-0000000200: assert.AnError general error for testing
E0212 13:30:18.259213   10863 tablet.go:175] unable to get connection for cell "zone2": node doesn't exist: cells/zone2/CellInfo
E0212 13:30:18.262589   10863 server.go:4404] StopReplication: failed to read tablet record for zone2-0000000200: node doesn't exist: cells/zone2/CellInfo
=== CONT  TestGetTopologyPath/global_path
=== CONT  TestCreateKeyspace/normal_keyspace
=== CONT  TestCreateKeyspace/keyspace_with_durability_policy_specified
=== CONT  TestGetTopologyPath/invalid_path
=== CONT  TestCreateKeyspace/allow_empty_vschema
--- PASS: TestGetShard (0.00s)
    --- PASS: TestGetShard/success (0.00s)
    --- PASS: TestGetShard/shard_not_found (0.00s)
    --- PASS: TestGetShard/unavailable_topo_server (0.00s)
=== CONT  TestGetSchemaMigrations/execute_fetch_failure
=== CONT  TestGetSchemaMigrations/bad_row_data
=== CONT  TestGetSrvKeyspaceNames/cell_filtering_topo_down
=== CONT  TestGetSchemaMigrations/bad_uuid_input
=== CONT  TestGetSrvKeyspaceNames/cell_filtering
=== CONT  TestCreateKeyspace/snapshot_keyspace_with_no_snapshot_time
=== CONT  TestCreateKeyspace/snapshot_keyspace_with_no_base_keyspace_specified
=== CONT  TestCreateKeyspace/snapshot_keyspace
=== CONT  TestSourceShardAdd/ok
=== CONT  TestGetSrvKeyspaceNames/all_cells_topo_down
=== CONT  TestGetShardReplication/all_cells_topo_down
=== CONT  TestGetShardReplication/cell_filtering_topo_down
--- PASS: TestStopReplication (0.00s)
    --- PASS: TestStopReplication/fail (0.00s)
    --- PASS: TestStopReplication/ok (0.00s)
    --- PASS: TestStopReplication/bad_request (0.00s)
    --- PASS: TestStopReplication/no_such_tablet (0.00s)
=== CONT  TestGetShardReplication/success
=== CONT  TestGetShardReplication/cell_filtering
=== CONT  TestSourceShardAdd/cannot_lock_keyspace
=== CONT  TestSourceShardAdd/uid_already_used
=== CONT  TestCreateKeyspace/keyspace_exists/force
=== CONT  TestCreateKeyspace/invalid_keyspace_type
=== CONT  TestGetSrvKeyspaces/success
=== CONT  TestCreateKeyspace/snapshot_keyspace_with_nonexistent_base_keyspace
=== CONT  TestCreateKeyspace/keyspace_exists/no_force
=== CONT  TestGetSrvKeyspaces/error_getting_cell_names
=== CONT  TestGetSrvKeyspaces/filtering_by_cell
=== CONT  TestGetSrvKeyspaces/error_getting_srvkeyspace
=== CONT  TestCompleteSchemaMigration/#00
=== CONT  TestCompleteSchemaMigration/executeQuery_failure
=== CONT  TestCompleteSchemaMigration/no_shard_primary
=== CONT  TestGetSrvKeyspaces/no_srvkeyspace_for_single_cell
W0212 13:30:18.273383   10863 server.go:2037] no srvkeyspace for keyspace testkeyspace in cell zone2
=== CONT  TestGetRoutingRules/success
=== CONT  TestGetRoutingRules/topo_error
=== CONT  TestGetRoutingRules/empty_routing_rules
--- PASS: TestTabletExternallyReparented (0.02s)
    --- PASS: TestTabletExternallyReparented/tablet_is_already_primary (0.00s)
    --- PASS: TestTabletExternallyReparented/cannot_change_tablet_type (0.01s)
    --- PASS: TestTabletExternallyReparented/tablet_is_nil (0.00s)
    --- PASS: TestTabletExternallyReparented/success (0.00s)
    --- PASS: TestTabletExternallyReparented/topo_is_down (0.00s)
=== CONT  TestStartReplication/no_such_tablet
=== CONT  TestStartReplication/fail
=== CONT  TestGetFullStatus/tablet_not_found
E0212 13:30:18.274444   10863 tablet.go:175] unable to get connection for cell "zone2": node doesn't exist: cells/zone2/CellInfo
E0212 13:30:18.277507   10863 server.go:4339] StartReplication: failed to read tablet record for zone2-0000000200: node doesn't exist: cells/zone2/CellInfo
=== CONT  TestStartReplication/bad_request
=== CONT  TestCleanupSchemaMigration/no_shard_primary
=== CONT  TestStartReplication/ok
--- PASS: TestGetTopologyPath (0.03s)
    --- PASS: TestGetTopologyPath/root_path (0.00s)
    --- PASS: TestGetTopologyPath/terminal_data_path_with_data_as_json (0.00s)
    --- PASS: TestGetTopologyPath/terminal_data_path (0.00s)
    --- PASS: TestGetTopologyPath/global_path (0.00s)
    --- PASS: TestGetTopologyPath/invalid_path (0.00s)
=== CONT  TestGetFullStatus/success
=== CONT  TestCleanupSchemaMigration/#00
=== CONT  TestCleanupSchemaMigration/executeQuery_failure
E0212 13:30:18.274696   10863 server.go:4380] StartReplication: failed to start replication on zone1-0000000100: assert.AnError general error for testing
=== CONT  TestSourceShardDelete/ok
E0212 13:30:18.277847   10863 tablet.go:186] unable to connect to tablet "cell:\"zone1\" uid:100": node doesn't exist: tablets/zone1-0000000100/Tablet
--- PASS: TestGetShardReplication (0.00s)
    --- PASS: TestGetShardReplication/all_cells_topo_down (0.00s)
    --- PASS: TestGetShardReplication/cell_filtering_topo_down (0.00s)
    --- PASS: TestGetShardReplication/success (0.00s)
    --- PASS: TestGetShardReplication/cell_filtering (0.00s)
E0212 13:30:18.274742   10863 schemamanager.go:115] failed to open executor: shard: 80- does not have a primary
E0212 13:30:18.286254   10863 schemamanager.go:115] failed to open executor: shard: 80- does not have a primary
=== CONT  TestForceCutOverSchemaMigration/no_shard_primary
=== CONT  TestChangeTabletType/success
=== CONT  TestSourceShardDelete/cannot_lock_keyspace
=== CONT  TestForceCutOverSchemaMigration/#00
=== CONT  TestForceCutOverSchemaMigration/executeQuery_failure
=== CONT  TestSourceShardDelete/no_SourceShard_with_uid
E0212 13:30:18.291301   10863 schemamanager.go:115] failed to open executor: shard: 80- does not have a primary
--- PASS: TestGetSchemaMigrations (0.00s)
    --- PASS: TestGetSchemaMigrations/#00 (0.00s)
    --- PASS: TestGetSchemaMigrations/gettablets_failure (0.00s)
    --- PASS: TestGetSchemaMigrations/execute_fetch_failure (0.00s)
    --- PASS: TestGetSchemaMigrations/bad_uuid_input (0.00s)
    --- PASS: TestGetSchemaMigrations/bad_row_data (0.00s)
--- PASS: TestGetSrvKeyspaceNames (0.00s)
    --- PASS: TestGetSrvKeyspaceNames/success (0.00s)
    --- PASS: TestGetSrvKeyspaceNames/cell_filtering_topo_down (0.00s)
    --- PASS: TestGetSrvKeyspaceNames/cell_filtering (0.00s)
    --- PASS: TestGetSrvKeyspaceNames/all_cells_topo_down (0.00s)
=== CONT  TestChangeTabletType/primary_promotions_not_allowed
=== CONT  TestGetPermissions/no_tablet
=== CONT  TestChangeTabletType/dry_run
E0212 13:30:18.294774   10863 tablet.go:186] unable to connect to tablet "cell:\"zone1\" uid:100": node doesn't exist: tablets/zone1-0000000100/Tablet
=== CONT  TestChangeTabletType/tablet_not_found
=== CONT  TestChangeTabletType/tabletmanager_failure
=== CONT  TestChangeTabletType/primary_demotions_not_allowed
=== CONT  TestGetPermissions/ok
=== CONT  TestGetPermissions/tmc_call_failed
=== CONT  TestGetSrvVSchemas/topology_error
--- PASS: TestGetSrvKeyspaces (0.00s)
    --- PASS: TestGetSrvKeyspaces/error_getting_cell_names (0.00s)
    --- PASS: TestGetSrvKeyspaces/error_getting_srvkeyspace (0.00s)
    --- PASS: TestGetSrvKeyspaces/success (0.00s)
    --- PASS: TestGetSrvKeyspaces/no_srvkeyspace_for_single_cell (0.00s)
    --- PASS: TestGetSrvKeyspaces/filtering_by_cell (0.00s)
=== CONT  TestGetSrvVSchemas/filtering_by_cell
E0212 13:30:18.295145   10863 tablet.go:186] unable to connect to tablet "cell:\"zone1\" uid:100": node doesn't exist: tablets/zone1-0000000100/Tablet
--- PASS: TestGetRoutingRules (0.00s)
    --- PASS: TestGetRoutingRules/topo_error (0.00s)
    --- PASS: TestGetRoutingRules/success (0.00s)
    --- PASS: TestGetRoutingRules/empty_routing_rules (0.00s)
=== CONT  TestExecuteFetchAsDBA/tablet_not_found
--- PASS: TestCreateKeyspace (0.00s)
    --- PASS: TestCreateKeyspace/allow_empty_vschema (0.00s)
    --- PASS: TestCreateKeyspace/keyspace_with_durability_policy_specified (0.00s)
    --- PASS: TestCreateKeyspace/normal_keyspace (0.00s)
    --- PASS: TestCreateKeyspace/snapshot_keyspace_with_no_snapshot_time (0.00s)
    --- PASS: TestCreateKeyspace/snapshot_keyspace_with_no_base_keyspace_specified (0.00s)
    --- PASS: TestCreateKeyspace/snapshot_keyspace (0.00s)
    --- PASS: TestCreateKeyspace/invalid_keyspace_type (0.00s)
    --- PASS: TestCreateKeyspace/keyspace_exists/no_force (0.00s)
    --- PASS: TestCreateKeyspace/keyspace_exists/force (0.00s)
    --- PASS: TestCreateKeyspace/snapshot_keyspace_with_nonexistent_base_keyspace (0.00s)
E0212 13:30:18.295248   10863 tablet.go:186] unable to connect to tablet "cell:\"zone1\" uid:404": node doesn't exist: tablets/zone1-0000000404/Tablet
--- PASS: TestSourceShardAdd (0.00s)
    --- PASS: TestSourceShardAdd/ok (0.00s)
    --- PASS: TestSourceShardAdd/cannot_lock_keyspace (0.00s)
    --- PASS: TestSourceShardAdd/uid_already_used (0.00s)
=== CONT  TestGetSrvVSchemas/cell_doesn't_exist
=== CONT  TestGetSrvVSchemas/no_SrvVSchema_for_single_cell
=== CONT  TestGetSrvVSchemas/success
=== CONT  TestExecuteFetchAsDBA/ok
=== CONT  TestExecuteFetchAsDBA/query_error
=== CONT  TestGetSrvVSchemas/one_of_many_cells_doesn't_exist
=== CONT  TestCreateShard/success
=== CONT  TestPlannedReparentShardSlow/nil_WaitReplicasTimeout_and_request_takes_29_seconds_is_ok
W0212 13:30:18.298597   10863 server.go:2211] no SrvVSchema for cell zone3
=== CONT  TestCreateShard/topo_is_down
W0212 13:30:18.298639   10863 server.go:2211] no SrvVSchema for cell zone3
=== CONT  TestCreateShard/include_parent
=== CONT  TestPlannedReparentShardSlow/nil_WaitReplicasTimeout_and_request_takes_31_seconds_is_error
--- PASS: TestStartReplication (0.00s)
    --- PASS: TestStartReplication/no_such_tablet (0.00s)
    --- PASS: TestStartReplication/fail (0.00s)
    --- PASS: TestStartReplication/bad_request (0.00s)
    --- PASS: TestStartReplication/ok (0.00s)
--- PASS: TestGetFullStatus (0.00s)
    --- PASS: TestGetFullStatus/success (0.00s)
    --- PASS: TestGetFullStatus/tablet_not_found (0.01s)
=== CONT  TestCreateShard/shard_exists/force
=== CONT  TestEmergencyReparentShardSlow/nil_WaitReplicasTimeout_and_request_takes_29_seconds_is_ok
=== CONT  TestDeleteTablets/single_replica
=== CONT  TestCreateShard/shard_exists/no_force
=== CONT  TestCreateShard/include_parent/keyspace_exists/force
=== CONT  TestCreateShard/include_parent/keyspace_exists/no_force
=== CONT  TestCreateShard/keyspace_does_not_exist
=== CONT  TestDeleteTablets/multiple_tablets
=== CONT  TestDeleteTablets/topo_server_is_down
--- PASS: TestSourceShardDelete (0.00s)
    --- PASS: TestSourceShardDelete/ok (0.00s)
    --- PASS: TestSourceShardDelete/cannot_lock_keyspace (0.00s)
    --- PASS: TestSourceShardDelete/no_SourceShard_with_uid (0.00s)
--- PASS: TestCompleteSchemaMigration (0.00s)
    --- PASS: TestCompleteSchemaMigration/#00 (0.00s)
    --- PASS: TestCompleteSchemaMigration/executeQuery_failure (0.00s)
    --- PASS: TestCompleteSchemaMigration/no_shard_primary (0.01s)
--- PASS: TestCleanupSchemaMigration (0.00s)
    --- PASS: TestCleanupSchemaMigration/no_shard_primary (0.01s)
    --- PASS: TestCleanupSchemaMigration/executeQuery_failure (0.01s)
    --- PASS: TestCleanupSchemaMigration/#00 (0.01s)
=== CONT  TestDeleteTablets/another_shard_is_locked
=== CONT  TestDeleteTablets/shard_is_locked
E0212 13:30:18.302546   10863 tablet.go:175] unable to get connection for cell "zone1": assert.AnError general error for testing
=== CONT  TestDeleteTablets/tablet_not_found
=== CONT  TestDeleteTablets/stale_primary_record
=== CONT  TestExecuteMultiFetchAsDBA/ok
--- PASS: TestForceCutOverSchemaMigration (0.00s)
    --- PASS: TestForceCutOverSchemaMigration/executeQuery_failure (0.00s)
    --- PASS: TestForceCutOverSchemaMigration/#00 (0.00s)
    --- PASS: TestForceCutOverSchemaMigration/no_shard_primary (0.00s)
--- PASS: TestGetPermissions (0.01s)
    --- PASS: TestGetPermissions/no_tablet (0.00s)
    --- PASS: TestGetPermissions/tmc_call_failed (0.00s)
    --- PASS: TestGetPermissions/ok (0.00s)
=== CONT  TestDeleteTablets/single_primary/with_AllowPrimary
--- PASS: TestExecuteFetchAsDBA (0.01s)
    --- PASS: TestExecuteFetchAsDBA/tablet_not_found (0.00s)
    --- PASS: TestExecuteFetchAsDBA/query_error (0.00s)
    --- PASS: TestExecuteFetchAsDBA/ok (0.00s)
--- PASS: TestGetSrvVSchemas (0.02s)
    --- PASS: TestGetSrvVSchemas/topology_error (0.00s)
    --- PASS: TestGetSrvVSchemas/filtering_by_cell (0.00s)
    --- PASS: TestGetSrvVSchemas/cell_doesn't_exist (0.00s)
    --- PASS: TestGetSrvVSchemas/one_of_many_cells_doesn't_exist (0.00s)
    --- PASS: TestGetSrvVSchemas/no_SrvVSchema_for_single_cell (0.00s)
    --- PASS: TestGetSrvVSchemas/success (0.00s)
E0212 13:30:18.302781   10863 tablet.go:186] unable to connect to tablet "cell:\"zone1\" uid:200": node doesn't exist: tablets/zone1-0000000200/Tablet
--- PASS: TestChangeTabletType (0.00s)
    --- PASS: TestChangeTabletType/success (0.00s)
    --- PASS: TestChangeTabletType/primary_promotions_not_allowed (0.00s)
    --- PASS: TestChangeTabletType/primary_demotions_not_allowed (0.00s)
    --- PASS: TestChangeTabletType/tablet_not_found (0.00s)
    --- PASS: TestChangeTabletType/tabletmanager_failure (0.00s)
    --- PASS: TestChangeTabletType/dry_run (0.00s)
--- PASS: TestCreateShard (0.01s)
    --- PASS: TestCreateShard/success (0.00s)
    --- PASS: TestCreateShard/topo_is_down (0.00s)
    --- PASS: TestCreateShard/include_parent (0.00s)
    --- PASS: TestCreateShard/include_parent/keyspace_exists/no_force (0.00s)
    --- PASS: TestCreateShard/keyspace_does_not_exist (0.00s)
    --- PASS: TestCreateShard/shard_exists/force (0.00s)
    --- PASS: TestCreateShard/include_parent/keyspace_exists/force (0.00s)
    --- PASS: TestCreateShard/shard_exists/no_force (0.00s)
W0212 13:30:18.302865   10863 topo.go:244] Deleting primary zone1-0000000100 from shard testkeyspace/- but primary in Shard object was <nil>
=== CONT  TestDeleteTablets/single_primary/no_AllowPrimary
W0212 13:30:18.302937   10863 topo.go:244] Deleting primary zone1-0000000100 from shard testkeyspace/-80 but primary in Shard object was <nil>
=== CONT  TestDeleteSrvKeyspace/success
=== CONT  TestDeleteSrvKeyspace/empty_cell_argument
W0212 13:30:18.303044   10863 server.go:2211] no SrvVSchema for cell zone2
=== CONT  TestEmergencyReparentShardSlow/nil_WaitReplicasTimeout_and_request_takes_31_seconds_is_error
=== CONT  TestExecuteMultiFetchAsDBA/query_error
=== CONT  TestExecuteMultiFetchAsDBA/tablet_not_found
--- PASS: TestDeleteTablets (0.00s)
    --- PASS: TestDeleteTablets/single_replica (0.00s)
    --- PASS: TestDeleteTablets/multiple_tablets (0.00s)
    --- PASS: TestDeleteTablets/topo_server_is_down (0.00s)
    --- PASS: TestDeleteTablets/tablet_not_found (0.00s)
    --- PASS: TestDeleteTablets/single_primary/with_AllowPrimary (0.00s)
    --- PASS: TestDeleteTablets/shard_is_locked (0.00s)
    --- PASS: TestDeleteTablets/single_primary/no_AllowPrimary (0.00s)
    --- PASS: TestDeleteTablets/another_shard_is_locked (0.00s)
    --- PASS: TestDeleteTablets/stale_primary_record (0.00s)
E0212 13:30:18.306767   10863 tablet.go:186] unable to connect to tablet "cell:\"zone1\" uid:404": node doesn't exist: tablets/zone1-0000000404/Tablet
=== CONT  TestDeleteSrvKeyspace/cell_not_found
--- PASS: TestExecuteMultiFetchAsDBA (0.01s)
    --- PASS: TestExecuteMultiFetchAsDBA/ok (0.00s)
    --- PASS: TestExecuteMultiFetchAsDBA/query_error (0.00s)
    --- PASS: TestExecuteMultiFetchAsDBA/tablet_not_found (0.00s)
=== CONT  TestAlterSchemaMigrationQuery/cancel_4e5dcf80_354b_11eb_82cd_f875a4d24e90
=== CONT  TestAlterSchemaMigrationQuery/cleanup_4e5dcf80_354b_11eb_82cd_f875a4d24e90
=== CONT  TestAlterSchemaMigrationQuery/cancel_ALL
=== CONT  TestCancelSchemaMigration/#00
=== CONT  TestCancelSchemaMigration/no_shard_primary
--- PASS: TestDeleteSrvKeyspace (0.00s)
    --- PASS: TestDeleteSrvKeyspace/empty_cell_argument (0.00s)
    --- PASS: TestDeleteSrvKeyspace/success (0.00s)
    --- PASS: TestDeleteSrvKeyspace/cell_not_found (0.00s)
=== CONT  TestCancelSchemaMigration/executeQuery_failure
=== CONT  TestAlterSchemaMigrationQuery/cancel_all
=== CONT  TestChangeTabletTags/success
E0212 13:30:18.307219   10863 schemamanager.go:115] failed to open executor: shard: 80- does not have a primary
--- PASS: TestAlterSchemaMigrationQuery (0.00s)
    --- PASS: TestAlterSchemaMigrationQuery/cancel_4e5dcf80_354b_11eb_82cd_f875a4d24e90 (0.00s)
    --- PASS: TestAlterSchemaMigrationQuery/cleanup_4e5dcf80_354b_11eb_82cd_f875a4d24e90 (0.00s)
    --- PASS: TestAlterSchemaMigrationQuery/cancel_ALL (0.00s)
    --- PASS: TestAlterSchemaMigrationQuery/cancel_all (0.00s)
=== CONT  TestChangeTabletTags/tablet_not_found
=== CONT  TestChangeTabletTags/tabletmanager_failure
=== CONT  TestChangeTabletTags/success_with_existing
--- PASS: TestCancelSchemaMigration (0.02s)
    --- PASS: TestCancelSchemaMigration/#00 (0.00s)
    --- PASS: TestCancelSchemaMigration/executeQuery_failure (0.00s)
    --- PASS: TestCancelSchemaMigration/no_shard_primary (0.01s)
=== CONT  TestChangeTabletTags/success_with_replace
E0212 13:30:18.312649   10863 tablet.go:186] unable to connect to tablet "cell:\"zone1\" uid:100": node doesn't exist: tablets/zone1-0000000100/Tablet
=== CONT  TestValueToVTTime/#00
=== CONT  TestValueToVTTime/parse_error
=== CONT  TestExecuteFetchAsApp/query_error
=== CONT  TestValueToVTTime/empty_string
=== CONT  TestExecuteFetchAsApp/tablet_not_found
--- PASS: TestValueToVTTime (0.00s)
    --- PASS: TestValueToVTTime/#00 (0.00s)
    --- PASS: TestValueToVTTime/parse_error (0.00s)
    --- PASS: TestValueToVTTime/empty_string (0.00s)
=== CONT  TestDeleteShards/tablets_in_topo_belonging_to_other_shard
=== CONT  TestExecuteFetchAsApp/ok
E0212 13:30:18.312979   10863 tablet.go:186] unable to connect to tablet "cell:\"zone1\" uid:404": node doesn't exist: tablets/zone1-0000000404/Tablet
=== CONT  TestDeleteShards/shard_has_tablets/Recursive=true
--- PASS: TestChangeTabletTags (0.02s)
    --- PASS: TestChangeTabletTags/tabletmanager_failure (0.00s)
    --- PASS: TestChangeTabletTags/success_with_existing (0.00s)
    --- PASS: TestChangeTabletTags/success (0.00s)
    --- PASS: TestChangeTabletTags/tablet_not_found (0.00s)
    --- PASS: TestChangeTabletTags/success_with_replace (0.00s)
=== CONT  TestDeleteShards/ShardReplication_in_topo
=== CONT  TestDeleteShards/shard_has_tablets/Recursive=false
=== CONT  TestDeleteShards/success
--- PASS: TestExecuteFetchAsApp (0.00s)
    --- PASS: TestExecuteFetchAsApp/query_error (0.00s)
    --- PASS: TestExecuteFetchAsApp/tablet_not_found (0.00s)
    --- PASS: TestExecuteFetchAsApp/ok (0.00s)
E0212 13:30:18.313251   10863 tablet.go:186] unable to connect to tablet "cell:\"zone1\" uid:100": node doesn't exist: tablets/zone1-0000000100/Tablet
W0212 13:30:18.316489   10863 tablet.go:505] cell:"zone1" uid:100: node doesn't exist: tablets/zone1-0000000100/Tablet
=== CONT  TestDeleteShards/shard_is_serving/EvenIfServing=true
E0212 13:30:18.316530   10863 tablet.go:186] unable to connect to tablet "cell:\"zone2\" uid:200": node doesn't exist: tablets/zone2-0000000200/Tablet
=== CONT  TestDeleteShards/shard_is_serving/EvenIfServing=false
W0212 13:30:18.316553   10863 tablet.go:505] cell:"zone2" uid:200: node doesn't exist: tablets/zone2-0000000200/Tablet
=== CONT  TestDeleteShards/topo_is_down
E0212 13:30:18.316588   10863 tablet.go:186] unable to connect to tablet "cell:\"zone3\" uid:300": node doesn't exist: tablets/zone3-0000000300/Tablet
W0212 13:30:18.316618   10863 tablet.go:505] cell:"zone3" uid:300: node doesn't exist: tablets/zone3-0000000300/Tablet
=== CONT  TestDeleteShards/multiple_shards
=== CONT  TestDeleteShards/shard_not_found
=== CONT  TestDeleteShards/shard_is_locked_with_force
=== CONT  TestDeleteShards/shard_is_locked
W0212 13:30:18.317073   10863 topo.go:54] lock for shard testkeyspace/- is already held: failed to lock shard testkeyspace/- for deletion, but force=true, proceeding anyway ...
--- PASS: TestDeleteShards (0.01s)
    --- PASS: TestDeleteShards/shard_has_tablets/Recursive=false (0.00s)
    --- PASS: TestDeleteShards/shard_has_tablets/Recursive=true (0.00s)
    --- PASS: TestDeleteShards/tablets_in_topo_belonging_to_other_shard (0.00s)
    --- PASS: TestDeleteShards/success (0.00s)
    --- PASS: TestDeleteShards/topo_is_down (0.00s)
    --- PASS: TestDeleteShards/ShardReplication_in_topo (0.00s)
    --- PASS: TestDeleteShards/shard_is_serving/EvenIfServing=false (0.00s)
    --- PASS: TestDeleteShards/shard_is_serving/EvenIfServing=true (0.00s)
    --- PASS: TestDeleteShards/shard_not_found (0.00s)
    --- PASS: TestDeleteShards/multiple_shards (0.00s)
    --- PASS: TestDeleteShards/shard_is_locked (0.00s)
    --- PASS: TestDeleteShards/shard_is_locked_with_force (0.00s)
W0212 13:30:19.130217   10863 server.go:4133] context deadline exceeded before obtaining replication position from zone1-0000000101: context deadline exceeded
W0212 13:30:19.130202   10863 server.go:4093] context deadline exceeded before obtaining primary position from zone1-0000000100: context deadline exceeded
=== RUN   TestShardReplicationPositions/other_rpc_errors_are_fatal
=== RUN   TestShardReplicationPositions/nonexistent_shard
--- PASS: TestShardReplicationPositions (1.01s)
    --- PASS: TestShardReplicationPositions/success (0.00s)
    --- PASS: TestShardReplicationPositions/timeouts_are_nonfatal (1.00s)
    --- PASS: TestShardReplicationPositions/other_rpc_errors_are_fatal (0.00s)
    --- PASS: TestShardReplicationPositions/nonexistent_shard (0.00s)
--- PASS: TestGetTablets (0.02s)
    --- PASS: TestGetTablets/tablet_type_filter (0.00s)
    --- PASS: TestGetTablets/cells_filter_with_single_error_is_fatal_in_strict_mode (0.00s)
    --- PASS: TestGetTablets/tablet_alias_filter_with_none_found (0.01s)
    --- PASS: TestGetTablets/tablet_alias_filtering (0.01s)
    --- PASS: TestGetTablets/keyspace_and_shard_filter_-_error (0.00s)
    --- PASS: TestGetTablets/no_tablets (0.00s)
    --- PASS: TestGetTablets/cells_filter_with_single_error_is_nonfatal (0.00s)
    --- PASS: TestGetTablets/stale_primary (0.00s)
    --- PASS: TestGetTablets/cells_filter (0.00s)
    --- PASS: TestGetTablets/in_nonstrict_mode_if_all_cells_fail_the_request_fails (0.00s)
    --- PASS: TestGetTablets/keyspace,_shard,_and_tablet_type_filter (0.00s)
    --- PASS: TestGetTablets/keyspace_filter (0.00s)
    --- PASS: TestGetTablets/keyspace_and_shard_filter (0.00s)
    --- PASS: TestGetTablets/keyspace_and_shard_filter_-_stale_primary (0.00s)
    --- PASS: TestGetTablets/multiple_cells_with_one_timing_out_and_strict_false (2.00s)
    --- PASS: TestGetTablets/multiple_cells_with_one_timing_out_and_strict_true (2.00s)
=== RUN   TestSleepTablet/tablet_not_found
E0212 13:30:33.195277   10863 tablet.go:175] unable to get connection for cell "zone2": node doesn't exist: cells/zone2/CellInfo
=== RUN   TestSleepTablet/sleep_rpc_error
--- PASS: TestSleepTablet (15.01s)
    --- PASS: TestSleepTablet/ok (0.00s)
    --- PASS: TestSleepTablet/default_sleep_duration (15.00s)
    --- PASS: TestSleepTablet/tablet_not_found (0.00s)
    --- PASS: TestSleepTablet/sleep_rpc_error (0.00s)
--- PASS: TestPlannedReparentShardSlow (0.00s)
    --- PASS: TestPlannedReparentShardSlow/nil_WaitReplicasTimeout_and_request_takes_29_seconds_is_ok (28.00s)
    --- PASS: TestPlannedReparentShardSlow/nil_WaitReplicasTimeout_and_request_takes_31_seconds_is_error (30.00s)
--- PASS: TestEmergencyReparentShardSlow (0.00s)
    --- PASS: TestEmergencyReparentShardSlow/nil_WaitReplicasTimeout_and_request_takes_29_seconds_is_ok (29.00s)
    --- PASS: TestEmergencyReparentShardSlow/nil_WaitReplicasTimeout_and_request_takes_31_seconds_is_error (30.00s)
--- PASS: TestSetShardTabletControl (45.05s)
    --- PASS: TestSetShardTabletControl/ok (0.00s)
    --- PASS: TestSetShardTabletControl/remove_tabletcontrols (0.00s)
    --- PASS: TestSetShardTabletControl/disable_queryservice (0.00s)
    --- PASS: TestSetShardTabletControl/keyspace_lock_error (45.00s)
PASS
ok  	vitess.io/vitess/go/vt/vtctl/grpcvtctldserver	45.853s

Copy link
Contributor

@mattlord mattlord left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks again, @lmorduch ! Nice work on this

@lmorduch
Copy link
Contributor Author

Thanks again, @lmorduch ! Nice work on this

Thanks for the help and feedback! Really appreciate the assistance.

@mattlord mattlord merged commit 000fbbe into vitessio:main Feb 13, 2025
103 checks passed
@lmorduch lmorduch deleted the sort-get-tablets branch February 13, 2025 20:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Component: Cluster management Type: Enhancement Logical improvement (somewhere between a bug and feature)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants