feat(swing-store): budget-limited deletion of snapshot and transcripts

[warner]

Both snapStore.deleteVatSnapshots() and
transcriptStore.deleteVatTranscripts() now take a numeric budget= argument, which will limit the number of snapshots or transcript spans deleted in each call. Both return a { done, cleanups } record so the caller knows when to stop calling.

This enables the slow deletion of large vats (lots of transcript spans or snapshots), a small number of items at a time. Recommended budget is 5, which (given SwingSet's snapInterval=200 default) will cause the deletion of 1000 rows from the transcriptItems table each call, which shouldn't take more than 100ms.

Without this, the kernel's attempt to slowly delete a terminated vat would succeed in slowly draining the kvStore, but would trigger a gigantic SQL transaction at the end, as it deleted every transcript item in the vat's history. The worst-case example I found would be the mainnet chain's v43-walletFactory, which (as of apr-2024) has 8.2M transcript items in 40k spans. A fast machine takes two seconds just to count all the items, and deletion took 22 minutes, with a swingstore.wal file that peaked at 27 GiB. This would cause an enormous chain stall at some surprising point in time weeks or months after the vat was first terminated. In addition, both the transcript spans and the snapshot records are shadowed into IAVL (via export-data) for integrity, and deleting 40k+40k=80k IAVL records in a single block might cause some significant churn too.

The kernel should call transcriptStore.stopUsingTranscript() and snapStore.stopUsingLastSnapshot() as soon as the vat is terminated, to make exports smaller right away (by omitting all transcript/snapshot artifacts for the given vat, even before those DB rows or their export-data records have been deleted).

New swing-store documentation was added.

refs #8928

[cloudflare-workers-and-pages]

Deploying agoric-sdk with    Cloudflare Pages

Latest commit:	c43bf63
Status:	✅  Deploy successful!
Preview URL:	https://00bb9404.agoric-sdk.pages.dev
Branch Preview URL:	https://warner-8980-swingstore-delet.agoric-sdk.pages.dev

View logs

[warner]

Security Considerations

None. Swingstore exports during slow deletion continue to be coherent (export-data records remain until deleted, artifacts stop right away), unit tests demonstrate the importability of these exports. There were no changes to the import-side code, including the hash validation pathways.

Scaling Considerations

This change enables better scaling, by allowing the kernel to delete terminated vats slowly, and avoid overloading validator nodes with tons of DB or IAVL work in a single block.

Documentation Considerations

This PR introduces extensive new swing-store documentation, including coverage of the new deletion-budget feature.

Testing Considerations

The upcoming swingset changes that take advantage of this API should have their own unit tests, which exercise slow deletion.

We'll also need to perform integration/performance tests with real mainnet chain data, and make sure we're satisfied with our choices for deletion rate / budget. We must choose values that delete the old price-feed vats in a timely fashion (weeks or months), and does not overload validators with the constant "background" cleanup work.

Upgrade Considerations

None: the old (unlimited) API continues to work the same as before. It only changes behavior if you provide a new positional budget argument to the deletion calls.

[gibson042]

Just doc changes for this first-round review; next I'll look at the code. But by the way, 👍👍 for making the former so thorough.

[gibson042]

Once again, great testing. No blocking concerns.

[gibson042]

Suggestion: rename "cleanups" for clarity/comprehensibility.

Suggested change

	* deleteVatSnapshots: (vatID: string, budget?: number) => { done: boolean, cleanups: number },
	* deleteVatSnapshots: (vatID: string, budget?: number) => { done: boolean, count: number },

[warner]

I'll look into it.. that'll require matching in the kernel, in the next PR in this series