Goal: graceful TSN upgrade mechanism

[zolotokrylin]

Right now, to deploy new servers, we're pushing all the data again for substantial changes. So, the new state should be the same as before redeployment. However, if this testnet was a little more public, and some other data provider would start pushing data and testing their own contracts, we could potentially reset it, purging their work.
With other 3rd party data providers, it would be important to establish that this could happen on a strict upgrade schedule so that they know when to reset and prevent experiment losses.
Would also be feasible to set an upgrade pipeline where no data is lost (publish the docker image for tsn, make a backup of docker volumes for posgres + tsn, upgrade it, roll back if necessary) but that is more work that I don't know yet if is a priority

Originally posted by @outerlook in #145 (comment)

[brennanjl]

We're looking really hard at ways we can support this as an official feature / tool. We're still designing the solution, but it will likely work something similar to the Cosmos SDK's Cosmovisor.

[rsoury]

Keep in mind, the typical testnet behaviour by app-chains within the Cosmos ecosystem is to:

1. perform complete resets/purge starting from a particular block.
2. this block is not always the genesis block - but the reset allows for applying software updates
3. communities are usually informed well in advance of any testnet restarts/upgrades

The issue here is that Kwil does not persist history by default.
Pruning historic blocks is (rightly so) the default.
Therefore, we may need a dedicated solution by Kwil for this.

[outerlook]

@zolotokrylin is it time for this goal before

• Goal: onboard Kwil as Node Operator #393

?

Pros:

• if we need to change something in our architecture, our nodes won't be redeployed. We problably just update the docker image of it

Cons:

• Making changes to our contracts / logic becomes more challenging, because we need to coordinate contract drops, etc

Alternative:
We ask other node operators to also reset their instance on our redeployments

[zolotokrylin]

@outerlook, these goals are more important and urgent:

• Goal: onboard Kwil as Node Operator #393
• Goal: run load testing #441
• Goal: Get Usage Metrics From TSN #427