Speed up state replays

[rkapka]

When executing certain API endpoints, it can take a long time to fetch results because of state replays. As an example, setting --slots-per-archive-point=32 means that Prysm saves a state to the DB every 32 slots. Executing /eth/v1/beacon/states/{state_id}/validators/{validator_id} with state_id equal to the next slot after a saved state's slot requires 31 replays, which takes ~5 seconds on my machine. Below is a CPU profile.

Each of the following takes about 1 second:

• chainForSlot
• ProcessEpoch
• ProcessSlot
• ProcessBlockHeaderNoVerify
• ProcessOperationsNoVerifyAttsSigs

After running the node for 30 minutes, the DB size is roughly 2GB, which means it will reach almost 3TB in a month. This means saving every 32 states is already a lot to ask for from users, so we should strive to respond to requests in a more timely manner e.g. under a second.

We can also explore other means to speed up the API, such as caching.