Runbook: backup of the Agoric JavaScript VM State

The Agoric JavaScript VM (SwingSet) active state is stored and committed in a DB (called the swing-store) separate from the cosmos DB. While a partial copy is mirrored in the cosmos DB for consensus and state-sync purposes, during normal operation the JavaScript VM only uses the swing-store. Because these 2 DBs do not have an atomic commit point, some logic exists in the JS side (which commits first) to not re-execute if it detects the latest block again, but only re-emit the state changes instead.

This situation can arise if the validator node crashes between the commit of the swing-store and the commit of the cosmos DB, or if the cosmos state is rolled back.

While this mechanism is designed to handle regular operational situations, some upgrades that perform data migrations may create a situation where this mechanism is unable to recover from a crash of the validator node. While best practices are to backup a node's state before starting an upgrade, it is sometimes impractical given the size of the the combined state.

The following sections provide details on the swing-store and 2 different approaches to perform a backup and restore of the swing-store.

Raw copy of the swing-store

The swing-store is an SQLite3 DB stored under $AG_CHAIN_COSMOS_HOME/data/agoric, where $AG_CHAIN_COSMOS_HOME defaults to $HOME/.agoric. It can simply be backed up and restored using a copy of the swingstore.* files.

cp -a /path/to/.agoric/data/agoric/swingstore.* /path/to/empty/backup/dir/

The stored commit height can be verified using any SQLite3 tool reading the entry with key=host.height in the kvStore table, or

node /path/to/agoric-sdk/packages/SwingSet/misc-tools/db-get.js /path/to/.agoric/data/agoric/ host.height

Import / export tools

There are some internal tools for the JS chain integration which can be used to export and import the JS state contained in the swing-store. While these tools are used to support state-sync and genesis export / start, they can also be used standalone from the command line.

To create an export of the swing-store state:

node /path/to/agoric-sdk/packages/cosmic-swingset/src/export-kernel-db.js \
  --state-dir=/path/to/.agoric/data/agoric/ \
  --export-dir=/path/to/empty/backup/dir \
  --export-data-mode=all --artifact-mode=replay

The artifact-mode can also be operational or archival, however the former will prevent future state-sync exports and require manual intervention in some future upgrade, and the latter may not be successful if the node was ever restored using another mode (e.g. through state-sync which uses the replay artifact-mode)

To recreate the swing-store from a previous export (caution the state-dir target will be emptied):

node /path/to/agoric-sdk/packages/cosmic-swingset/src/import-kernel-db.js \
  --state-dir=/path/to/.agoric/data/agoric/ \
  --export-dir=/path/to/dir/of/previous/backup \
  --export-data-mode=all --artifact-mode=replay

The import artifact-mode needs to be the same or lower than the mode used when creating the export (operational < replay < archival).

An export dir contains an export-manifest.json file which describes the swing-store export, including the height at which it was taken, the artifact-mode, and the filenames used to store each artifact and the "export data" (generic key/value data).