rebase/cherry-pick commits for upgrade 11 #8159

upgrade-test: isolate vaults-and-beyond env setup

feat(upgrade-test): allow passing tmux flags

instead of using a baked-in constant. Thanks to mfig for the suggestion.

build(cosmos): make Node.js addon tolerate relocation

chore(bin/agd): get expected xsnap version from repoconfig.sh

test: DeliverInbound from un-provisioned account is discarded

build!: remove deprecated `ag-cosmos-helper`

build(deployment): use `bullseye` until `bookworm` bakes a bit

upgrade-test: split targets

fix(cosmos): prevent Golang error wrapping stack frame divergence

Prevent `__has_builtin` compiler differences from causing divergence

upgrade-test: Tagging follows remote repository

upgrade-test: do not overwrite DEST_IMAGE when set

Restore loadgen in deployment test

fix(cosmos): Support building on Linux aarch64

feat: Cosmos upgrade handler calls swingset

remove remnants of `ag-setup-cosmos` symlink

feat(deployment): better `docker build` caching

refactor cosmos init

fix(ci): checkout agoric-sdk in sub folder

fix(cosmos): don't init controller before upgrade

…c-issue feat(deployment): replicate broken state sync in docker upgrade test

Refactor block/snapshot synchronization

…ts_handler.go

…pshotter.go

refactor snapshotter

This creates new files to house `importSwingStore`, `makeSwingStoreExporter`, and the kvStore. All were previously in `swingStore.js`, which is now a shell that gathers together the other components. The module exports have been consolidated into `src/index.js`, and types are now exported by a `types.d.ts`. No behavioral changes.

Previously, the swingstore importer would ignore "historical metadata": records (with hashes) for transcripts and heap snapshots that are not strictly necessary to rebuild workers. This was a mistake: our intention was to always preserve these hashes, so that we might safely (with integrity) repopulate the corresponding data in the future, using artifacts from untrusted sources. This commit rewrites the importer to record *all* metadata records in the first pass, regardless of whether we want historical data or not. All of these records will be stubs: they contain hashes, but are missing the actual bundle or snapshot or transcript items, as if they had been pruned. Then, in the second pass, we populate those stubs using the matching artifacts (or ignore the historical ones, as configured by the `includeHistorical` option). A final `assertComplete` pass insists that all the important (non-historical) records are fully populated. The exporter was updated to omit empty artifacts. New tests were added to assert that metadata is preserved regardless of import mode, and that the `assertComplete` pass really catches everything. Also, we check that an import throws if given a mis-sized artifact, like a transcript span that is missing a few items. A new `docs/swingstore.md` was added to describe the data model, including what it means for records to be pruned, and `docs/data-export.md` was updated. Note: this commit changes the schema of the `snapshots` table (to support temporarily-unpopulated `inUse = 1` snapshot data). To be precise, any swing-store created by this version (either via `initSwingStore` or `importSwingStore`) will get the new schema: pre-existing DBs opened with `openSwingStore` will continue to use the old/strict schema. This is fine for now, but as the comments in snapStore.js explain, we'll need to implement DB schema versioning and upgrade (#8089) before we can safely change any non-`importSwingStore` code to create unpopulated `inUse=1` records. fixes #8025

Since bug #8025 caused swingstore imports to omit historical metadata, there may be deployed instances that are missing data. To fix this, the new `hostStorage.repairMetadata()` method takes an (intact) exporter, and will re-create any metadata records that don't already exist. This function ignores artifacts entirely. The method will throw if the metadata stream has records that already exist in the DB but whose contents do not match. It also throws if the stream creates a new bundle record, since without artifacts that new record cannot possibly be populated, and the data model requires all bundles be present. The method ignores any kvStore records (which are not really metadata). Afterwards, the caller is responsible for calling `hostStorage.commit()` when they are ready. refs #8025

…rade-11

swingstore import should not discard historical metadata, and add repair function

fix(upgrade-test): use correct mainnet-1b image

feat(upgrade-test): plumb and use `UPGRADE_INFO`

Use for swing-store export data Standardize Json handling of export data

Effectively includes historical artifacts in state-sync mitigates the lack of full transcripts for current vat incarnations

Previously, `makeSwingStoreExporter()` took a positional argument named `exportMode`, with values of 'current', 'archival', or 'debug'. This controlled how many artifacts were included in the export, on a best-effort basis (e.g. a DB whose old spans were pruned would emit the same artifacts with either 'current' or 'archival'). `importSwingStore()` took an options bag with both the `makeSwingStore` options (like `keepSnapshots` and `keepTranscripts`), and an import-specific `includeHistorical` boolean, which controlled which artifacts were processed by the import. This was also on a best-effort basis: `includeHistorical: true` on an export dataset that lacked old spans would produce the same (pruned) DB as `false`. This commit changes both APIs to take an options bag with a common `artifactMode` option, with values of `operational`, `replay`, `archival`, or `debug`. The `operational` choice replaces `current` and behaves the same way: just enough data for normal operations. The new `replay` choice 'operational' and 'archival', and selects all transcript spans for the current incarnation of each vat, but omits transcript spans for old incarnations: enough to perform a full vat-replay of the latest incarnation. Note: `makeSwingStoreExporter` was changed from a positional argument to an options bag, and no attempt was made to be compatible with old-style callers. During export, the mode is now strict: if the DB lacks the artifacts requested by the given mode, `makeSwingStoreExporter()` will throw an error, rather than emit fewer artifacts than desired. This means `artifactMode: 'replay'` will fail unless the DB being exported has all those old (current-incarnation) transcript items. And `archival` will fail unless the DB has the old incarnation spans too. The `debug` mode is best-effort, and emits everything available without the additional completeness checks. During import, the mode applies both an import filter and a completeness check. So exporting with `archival` but importing with `operational` will get you a pruned DB, lacking anything historical. Exporting with `operational` and importing with `replay` or `archival` will fail, because the newly-populated DB does not contain any historical artifacts. closes #8105

…-refactor refactor(x/swingset): shared KVEntry and helpers

…factor feat: use x/swingset DB for swing-store shadow copy

fix(swing-store): add 'replay' artifactMode, make export more strict

fix(x/swingset): migration upgrade handler to fix state-sync

Do not assume deamonization

Adding coverage of metrics produced in agoric-upgrade-10 on agoric-upgrade

…store feat(x/swingset): include swing-store in genesis

…ereleases (cherry-picked from commit b1917d6 from trunk, contained in PR #8243) We used to patch this library to reduce the `releaseType` level for our packages that had not yet reached 1.0 . We then switched to an upstream version which performed this reduction itself, except not when doing a "prerelease". Now that we need to do a prerelease, we'd like this reduction behavior in both arms of the conditional, so this commit re-introduces our original patch. refs #8242

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rebase/cherry-pick commits for upgrade 11 #8159

rebase/cherry-pick commits for upgrade 11 #8159

Commits on Aug 16, 2023

Commits on Aug 22, 2023

Commits on Aug 23, 2023