With the successful launch of Neutron on Cosmos Hub, we’ve learned a lot about administering the Hub’s testnet infrastructure, as well as what it means to rally 80+ validators together for many coordinated upgrades and new consumer chain launches. Together we’ve laughed, we’ve learned, and we’ve banged our heads on our desks after seeing the 17th AppHash error in a row.
Currently, testnet rehearsals for launches and upgrades are scheduled on an as-needed basis, with one or two rehearsals a week. Consumer chains represent a growth in complexity for not only The Hub, but also for the Testnets Program. Coupled with the regular cadence of coordinated Gaia upgrades, it’s clear that as more consumer chains onboard onto The Hub, the demand on the Testnets Program will also grow. Adding incentives to testnet participation has come up before, and Hypha is actively working on creative solutions in this space. In parallel, to address the short-term growing demand on validator attention and resources, we’re trialing a new experiment: Testnet Wednesdays.
All coordinated upgrade and launch activities will now take place on the Interchain Security testnet at 14:00 UTC each week. Some weeks, there might not be anything – we’ll let validators know by Monday if the agenda is empty, therefore no rehearsal is needed. In other weeks, there may be a provider chain upgrade and a new chain launch at the same time. We’ll do our best to space things out so that there aren’t more than two events in a single day. As we continue doing launches and upgrades together, we’ll hopefully get better and faster each week.
Emergency release tests for the Hub and for consumer chain teams will still be scheduled, as needed, outside of the normal rehearsal cadence. This will only be invoked in the event of critical security patches that represent a major risk to the Hub.
For chain development teams, this means that we’ll be working more closely with you to make sure that your release cycles line up with testnet rehearsal days. If a binary is not ready with enough lead time to build confidence, the rehearsal will be pushed to the following Wednesday. Committing to a public timeline for release and testing should also create more predictability for validators who want to run their own tests ahead of time.
Here is a tentative schedule of events for each week:
| Day | What's happening |
|---|---|
| Week prior | |
| Monday | By 16:00 UTC Hypha announces scope of the week’s tests |
| Wednesday | 15:00 UTC Rehearsal takes place on testnet |
| Thursday |