ts-redis-cluster integ tests fail on pulumi destroy

[ghost]

ts-redis-cluster, upgrade failed today during pulumi destroy. All the other steps succeeded — we just couldn't destroy the stack. Pulumi reported the following error:

error deleting MemoryDB Cluster (ts-redis-cluster-u-3990398698-redis): InvalidClusterStateFault: Cluster ts-redis-cluster-u-3990398698-redis has status snapshotting which is not valid for deletion.

When I manually tried to delete the stack later, it succeeded. It looks like Pulumi just isn't correctly waiting for the state to change to non-snapshotting.

If this happens again, we should consider adding a between-destroy-retries.sh hook that finds the cluster and waits for its status to be something valid.

I'm not sure exactly what that looks like, because I just took down the cluster (🤦). But next time this happens, we should poke around with aws memorydb describe-clusters and figure out how to script the wait.