Restore so missing tables are created

[smiklosovic]

@SuhailRahman would like to know how to restore into a table which is not there (it was deleted). This thread is for such discussion.

I ll try to summarise what would be necessary to add that feature, I need to think about it too in more depth to have some meaningful solution to this.

[SuhailRahman]

@smiklosovic Yes, I'm interested in knowing how to restore a table.

[smiklosovic]

In order to create a deleted table, you need to do in a nutshel these steps:

1. identify that a table you want to restore into is indeed missing
2. get CQL "create" statement from uploaded manifest related to your restore
3. create that table somewhere along the restoration
4. proceed with the restoration

Point 1) should be done like this:

1. Read what tables you are actually going to restore
2. read what tables there currently are on a live node
3. find out what tables are missing by consolidating 1) and 2)

Point 2) might be done by two ways:
a) reading it via JMX - this can be done by this "interaction" (3)
b) parsing data dir of Cassandra and get it from there - this can be done (4). You do not need to parse this again on your own, I think you will find CassandraData there already populated (will be discussed further).

I think a) is superior solution here because if you read it by b) you might actually get tables which are on disk but they are actually dropped by Cassandra - data might stay there but it does not mean that table is "live".

Point 2) can be satisfied by looking into Manifest (you might download it from a remote location), there is a cql schema as a string.

Point 3) is as follows ...

The groundwork for CQL connection is already done, Esop (hence Icarus) is depeding on a module called "instaclustr-commons" and CQL stuff is in this package (1). You will basically end up with injection of CqlSessionService "somewhere" (this will be elaborated on further).

The CqlSessionService is initialised in CassandraModule. You see that it detects automatically if that code runs in Kubernetes and if it does, it will read secrets and config from Kubernetes and it initialised Cassandra driver - version 4 of driver is able to read its configuration from arbitrary soruce and I am basically reading it from Kubernetes API.

You can read more about this here (2).

This is roughly how I would do it.

Now when it comes to the actual implementation, I believe you might figure out a lot of stuff already on your own but the crucial part is "where to acutally hook this logic".

As you probably know, there are phases in RestorationPhase class. These phases are executed one by one in restoration strategies - we have three

1. HardlinkingRestorationStrategy
2. InPlaceRestorationStrategy
3. ImportingRestorationStrategy

as InPlace strategy can be used only for nodes which are down (not running), this feature will not be possible to implement for that strategy.

Other strategies might be executed in distributed manner - please read how Icarus is treating these global operations in its documentation (it might be actually explained in Esop too). However, in a nutshell, this "create table" statement has to be executed just once for whole cluster - it can not be done in such a way that it would be executed on multiple nodes. Actually, it might be executed on multiple nodes only in case there is "IF NOT EXIST" which seems to be the case but I would ideally avoid this from happening because messing up with a schema from multiple nodes at once seems to be like a bad idea anyway.

The very similar "problem" is solved upon the execution TruncatingPhase. The trick done here is that such phase is executed only at one node if you check (5) , you see that it is executed only by oneClient so only on one node. I would hence create a new phase called SchemaPhase and I would execute it right after InitPhase but only for one client.

In SchemaPhase, if you create a table, this means that schema version of that cluster will change as such - this mean that it might take some time until that version is propagated to all nodes. In order to be sure that all is good, execute ClusterHealthCheckPhase as part of SchemaPhase after tables are created. Please do not execute tables in a quick manner, create a table, give it a time - (few secs) and create another one etc ... and check it is at same version at the end. The changing of schema version might have some other conseqences I do not want to go into right now but we will cross that bridge once we get there.

Please keep in mind that this has to work both for Icarus case as well as for Esop which will be invoked as CLI command however this should be quite easy to do if you follow how other phases are created.

This feature should be able to be turned off - some operators - for example Casskop - does not know how to CQL so do it by introducing a flag into RestorationRequest which will default to false and do this CQL stuff only in case it is true. and bypass it completely otherwise.

Please try to go through the code a bit and you will see it is not so hard after all, hit me if you want to know more, I am very excited you are willing to make this happen!

(1) https://github.com/instaclustr/instaclustr-commons/tree/master/src/main/java/com/instaclustr/cassandra/service
(2) https://github.com/instaclustr/cassandra-operator/wiki/Accounts,-login-and-SSL#how-is-cql-probe-working
(3) https://github.com/instaclustr/esop/blob/master/src/main/java/com/instaclustr/esop/impl/interaction/CassandraTables.java
(4) https://github.com/instaclustr/esop/blob/master/src/main/java/com/instaclustr/esop/impl/CassandraData.java#L43
(5) https://github.com/instaclustr/icarus/blob/master/src/main/java/com/instaclustr/icarus/coordination/IcarusRestoreOperationCoordinator.java#L187

[smiklosovic]

@cscetbon would you be willing to introduce this into Casskop? We might turn this off by default. Ideally you would not know about this feature, it would be all just ridden by one flag, a person interested in this would have to just setup some secrets and configs for CQL but that might be done without Casskop knowing anything ...

[smiklosovic]

@SuhailRahman there is also interesting case as what happens when the keyspace itself does not exist ... we would have to ship cql create statement upon backup too and deal with it similarly with additional checks around same cluster topology in times of a backup because data would not fit the cluster anymore.