Incremental Snapshots

Incremental Snapshots have been released, starting with v1.9.0! (Release notes here)

Overview

Incremental Snapshots are a means to speedup validator startup when needing to bootstrap and download snapshots. If a validator already has a full snapshot, it can download a smaller incremental snapshot instead. For more details about the design and implementation, refer to the github issue.

How to Enable

For v1.9

To enable incremental snapshots, add --incremental-snapshots to your validator args.

For v1.10 and later

Incremental snapshots are enabled by default. To disable, use --no-incremental-snapshots.

Configuration

There are a few knobs to can be tweaked. NOTE: This assumes --incremental-snapshots is set!

--full-snapshot-interval-slots

Set how frequently full snapshots are taken. It is recommended to leave this as the default (or, what the known validators are configured with). Must be a multiple of the incremental snapshots interval.

--incremental-snapshot-interval-slots

Set how frequently incremental snapshots are taken. Must be a multiple of the accounts hash interval.

--maximum-full-snapshots-to-retain

Set how many full snapshots to retain. Older snapshots are automatically removed as new snapshots are taken. Note, the oldest full snapshot is never removed.

--maximum-incremental-snapshots-to-retain

Set how many incremental snapshots to retain. Older snapshots are automatically removed as new snapshots are taken. Note, incremental snapshots are only retained for the newest full snapshot; once a new full snapshot is taken, all incremental snapshots based on older full snapshots are removed.

Default Values

https://github.com/solana-labs/solana/blob/7dbde2247d271511905a4b29df4ea631ee690c1d/runtime/src/snapshot_utils.rs#L47-L56

Best Practices

Use the same --incremental-snapshots flag as your known validators

If your known validators have incremental snapshots enabled, it is recommended to do the same. Alternatively, change your known validators to match your incremental snapshots setting.

Use the same full snapshot interval as your known validators

Since downloading snapshots during bootstrap relies on matching snapshots from known validators, it is recommended to use the same full snapshot interval.

For example, if the known validators have a full snapshot at slot 700,000 and an incremental snapshot at slot 789,600, then you would need to already have a full snapshot for slot 700,000 in order to match-and-download the incremental snapshot at 789,600. (Otherwise you may end up downloading the full snapshot as well, which is the same behavior from before incremental snapshots.)

Troubleshooting

Q: Why am I downloading full snapshots all the time?

• A0: Ensure you have v1.9 or later, and have enabled incremental snapshots (see How to Enable)
• A1: Your full snapshot interval is different from the known validators (see Best Practices)
• A2: The known validators have not yet enabled incremental snapshots, so there are none to download

Q: Why am I downloading very old full snapshots?

This can happen when (1) your node has incremental snapshots disabled, and (2) a known validator has incremental snapshots enabled. In this scenario, it is possible that the bootstrap logic will only wait long enough to discover the snapshots from known validator in (2).

Since that known validator as incremental snapshots enabled, its full snapshot interval will be 25,000 slots (assuming defaults) instead of 100 slots, and therefore its highest full snapshot may be from a slot 25,000 slots ago. Your node will then download this very old full snapshot, and then will not download an incremental snapshot.

The fix is to use the same --incremental-snapshots flag as your known validators.