need solutions for leader replacement / failover

[jchappelow]

The network will be quite fragile if a leader needs to be replaced i.e. it cannot be brought back with the same private key. We need at least a simple solution for final release IMO.

Deterministically picking a replacement leader from the validator set should be relatively straightforward. For instance, it can be based on last block's app_hash. Deciding when to proceed with replacement is harder.

Alternatively, the new leader can be picked an specified with admin commands in an election process e.g. admin command: validators promote [desired new leader id].

Other challenges?