Confusion around Keyspace APIs

[limbooverlambda]

Bug Report

What did you do?

I am evaluating the Keyspace APIs that are supported by TiKV API v2 (https://github.com/tikv/client-rust). I am running TiUP locally. I created a non-default keyspace "foo" as follows:

curl -H 'Content-Type: application/json' \
      -d '{ "name": "foo"}' \
      -X POST \
      127.0.0.1:19165/pd/api/v2/keyspaces
{
    "id": 1,
    "name": "foo",
    "state": "ENABLED",
    "created_at": 1712076752,
    "state_changed_at": 1712076752
}%

Looking for the keyspaces:

curl 127.0.0.1:19165/pd/api/v2/keyspaces
{
    "keyspaces": [
        {
            "id": 0,
            "name": "DEFAULT",
            "state": "ENABLED",
            "created_at": 1712076651,
            "state_changed_at": 1712076651
        },
        {
            "id": 1,
            "name": "foo",
            "state": "ENABLED",
            "created_at": 1712076752,
            "state_changed_at": 1712076752
        }
    ],
    "

I am using the TiKV rust client and looks like all the operations are working against the "foo" keyspace.

Now, I decide to disable the "foo" keyspace as follow:

 curl -H 'Content-Type: application/json' \
      -d '{ "state": "DISABLED"}' \
      -X PUT \
      127.0.0.1:19165/pd/api/v2/keyspaces/foo/state
{
    "id": 1,
    "name": "foo",
    "state": "DISABLED",
    "created_at": 1712076752,
    "state_changed_at": 1712077470
}%

The keyspace foo is disabled:

curl 127.0.0.1:19165/pd/api/v2/keyspaces
{
    "keyspaces": [
        {
            "id": 0,
            "name": "DEFAULT",
            "state": "ENABLED",
            "created_at": 1712076651,
            "state_changed_at": 1712076651
        },
        {
            "id": 1,
            "name": "foo",
            "state": "DISABLED",
            "created_at": 1712076752,
            "state_changed_at": 1712077470
        }
    ],
    "next_page_token": ""
}%

Now I send the same operations to the disabled Keyspace.

What did you expect to see?

I expect all operations to fail for the disabled Keyspace. Both PD and Storage Node (tikv) should honor the disabled status of the keyspace and not return any results associated with keys prefixed with the keyspace identifier.

What did you see instead?

All operations are returning keys and values from the disabled keyspace.

What version of PD are you using (pd-server -V)?

Using Tiup /.tiup/components/playground/v1.13.1/tiup-playground --mode tikv-slim --pd.port 19165 --kv.config tikv.yaml

[limbooverlambda]

Anyone who can shed some light on this?

[ystaticy]

Hi @limbooverlambda,
Thank you for your issue.
The feature you mentioned has not yet been implemented in client-rust.If you're interested in contributing a PR to it,we would be very welcome!

[limbooverlambda]

Thank @ystaticy . I will be more than happy to contribute. I am however a little unsure about the sequence of events here. Once a Keyspace is marked as disabled, any request from the client can still go through and end up in the TiKV node. If the data is present (the disabled Keyspace has not been GCed), TiKV will still return the data from the disabled keyspaces. In that case client_rust will not not be disabling the keyspace locally. Do we need a periodic polling mechanism for all the keyspaces cached by the client to ensure that they are still valid?