Tarantool topology module

[ligurio]

Introduction

With Tarantool’s vshard module [1], the tuples of a dataset are distributed across multiple nodes, with a Tarantool database server instance on each node. Each instance handles only a subset of the total data, so larger loads can be handled by simply adding more servers. All cluster nodes must share a common topology. vshard static configuration file (see configuration syntax in [2, 3]) describes topology that includes:

• replicaset id's (server groups for each shard)
• bucket's count
• uri's to access Tarantool instances
• shards

Topology schema with possible properties looks like:

vshard --------|----- bucket count
                        |
                        |----- availability zone 1
                       |----- availability zone 2
                       |----- ...
                       |----- availability zone n
                       |
                       |----- shard 1 (replicaset 1)
                       |----- shard 2 (replicaset 2) 
                       |----- ...
                       |----- shard n (replicaset n)
                                   |
                                   |--------- replica 1  (name, uri -> listen + auth, zone)
                                   |
                                   |--------- replica 2 (name, uri -> listen + auth, zone)
                                   |
                                   |--------- replica 3 (name, uri -> listen + auth, zone)
                                   |
                                   |--------- replica n (name, uri -> listen + auth, zone)

Example of topology file used by vshard:

return {
    bucket_count = 16,
    sharding = {
        ['aaaaaaaa-0000-4000-a000-000000000000'] = {
            replicas = {
                ['aaaaaaaa-0000-4000-a000-000000000011'] = {
                    name = 'moscow',
                    master=true,
                    uri="sharding:[email protected]:30011"
                },
                ['aaaaaaaa-0000-4000-a000-000000000012'] = {
                    name='moscow-rep',
                    uri="sharding:[email protected]:30012"
                }
            }
        },
        ['bbbbbbbb-0000-4000-a000-000000000000'] = {
            replicas = {
                ['bbbbbbbb-0000-4000-a000-000000000021'] = {
                    name='spb',
                    master=true,
                    uri="sharding:[email protected]:30021"
                },
                ['bbbbbbbb-0000-4000-a000-000000000022'] = {
                    name='spb-rep',
                    uri="sharding:[email protected]:30022"
                },
            },
        },
    }
}

Currently topology for vshard [2, 3] can be configured manually in a static configuration file.
For Cartridge most of topology can be obtained automatically but vshard roles for instances (router or storage) should be set manually.

Proposal

Create a separate Tarantool module in Lua that will expose API to work with topology. Module should persist topology using Configuration provider (see [9]). Cartridge and vshard should someday switch to start to use Topology provider for work with topology.

Configuration provider API has been proposed in [9]: read(), write(), lock()

There is already a Cartridge and vshard that already use topology using their own API. See [4, 5, 6, 7, 8] for Cartridge and [10] for vshard. It's an open question how Topology provider API should look like. We can create an API that will conform methods that Cartridge and vshard already use to work with topology or we can create API from scratch. For example:

• add an instance with specified properties
  For example adding two instances with same replicaset_id creates replicaset with two nodes.
• remove instance
• set instance property
  Allows to update properties like uid, uri, role, name, replicaset id.

There are also wishes that we should take into account in topology provider:

• use distance term instead of weights for availability zones (see Rename "replica weights" to "distances" vshard#253)
• in future we plan to replace UUIDs by names for replicasets

References

1. vshard https://www.tarantool.io/en/doc/latest/reference/reference_rock/vshard/vshard_index/
2. https://github.com/tarantool/vshard#configuration
3. https://www.tarantool.io/en/doc/latest/reference/reference_rock/vshard/vshard_admin/#configuration
4. https://www.tarantool.io/en/doc/latest/book/cartridge/cartridge_api/modules/cartridge.topology/
5. https://www.tarantool.io/en/doc/latest/book/cartridge/cartridge_api/modules/cartridge.lua-api.get-topology/
6. https://www.tarantool.io/en/doc/latest/book/cartridge/cartridge_api/modules/cartridge.lua-api.edit-topology/
7. https://www.tarantool.io/en/doc/latest/book/cartridge/cartridge_api/modules/cartridge.lua-api.topology/
8. https://www.tarantool.io/en/doc/latest/book/cartridge/cartridge_api/modules/cartridge.topology/
9. module: configuration provider (MVP) module: configuration provider (MVP) #5681
10. https://github.com/tarantool/vshard/blob/master/vshard/replicaset.lua
11. module: cluster topology provider (MVP) #5496

[kostja]

* Sergey Bronnikov <[email protected]> [21/02/05 18:17]:

### Proposal Create a separate Tarantool module in Lua that will expose API to work with topology.

This is called Raft. Configuration changes are Raft core. Vshard topology changes are a custom protocol which is not reliable, this is why Raft was supposed to be added in the first place. Isn't it funny, Raft is declared stable and a core part of Raft is marketed as a separate feature :) This gives a wonderful opportunity to invent a new protocol for configuration changes, qsync 2.0 maybe?

…

-- Konstantin Osipov, Moscow, Russia

[ligurio]

Thanks for comment.

Seems I got your point, but I not agree that we reinvent a wheel.

Yes, Raft knows about configuration changes and it can answer for questions like "who is a current leader?". Sure topology provider will reuse that information. Unfortunately Raft's knowledge limited by a single cluster only while vshard and Cartridge operates with a number of clusters. Besides membership and current leader topology provider should provide other properties like availability zones, buckets, shards etc. I'll add to a proposal that topology provider should reuse information from Raft to make it clear.

[kostja]

This is exactly what's wrong with the current Raft. The original idea was that Raft is used instead of membership to manage entire tarantool cluster, not just one replicaset. One was supposed to create a raft group for the entire cluster and store system spaces in that group. That would make sure entire cluster has a single, consistent schema. Check out old tasks for replication groups. The facilities for replication groups are in the replication protocol.

[Mons]

This task is not about Raft. It's about kinda mongo's config-server.
Clusterwide Raft is a separate challenging task, due to huge cluster size requirements.