Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cherry-pick upgrade-16 commits #9545

Merged
merged 83 commits into from
Jun 26, 2024
Merged

Conversation

mhofman
Copy link
Member

@mhofman mhofman commented Jun 20, 2024

Rebase todo:

# Branch fix-vow-make-watch-when-more-robust-against-loops-and-hangs-9487-
label base-fix-vow-make-watch-when-more-robust-against-loops-and-hangs-9487-
pick bcecf522d test(vow): add test of more vow upgrade scenarios
pick d7135b25a test: switch vow test to run under xs for metering
pick 99fccca71 test(vow): add test for resolving vow to external promise
pick 6d3f88cb6 test(vow): add test for vow based infinite vat ping pong
pick c78ff0e73 test(vow): check vow consumers for busy loops or hangs
pick 3c63cba02 fix(vow): prevent loops and hangs from watching promises
pick 188c8106d chore(vat-data): remove the deprecated `@agoric/vat-data/vow.js`
pick 44a6d16b9 fix(vow): allow resolving vow to external promise
label fix-vow-make-watch-when-more-robust-against-loops-and-hangs-9487-
reset base-fix-vow-make-watch-when-more-robust-against-loops-and-hangs-9487-
merge -C 4fca04020 fix-vow-make-watch-when-more-robust-against-loops-and-hangs-9487- # fix(vow): make watch/when more robust against loops and hangs (#9487)

# Branch ci-mergify-strip-merge-commit-HTML-comments-9499-
label base-ci-mergify-strip-merge-commit-HTML-comments-9499-
pick 63e21ab30 ci(mergify): strip merge commit HTML comments
label ci-mergify-strip-merge-commit-HTML-comments-9499-
reset base-ci-mergify-strip-merge-commit-HTML-comments-9499-
merge -C 7b93671f7 ci-mergify-strip-merge-commit-HTML-comments-9499- # ci(mergify): strip merge commit HTML comments (#9499)

# Pull request #9506
pick 249a5d429 fix(SwingSet): Undo deviceTools behavioral change from #9153 (#9506)

# Pull request #9507
pick a19a964b3 fix(liveslots): promise watcher to cause unhandled rejection if no handler (#9507)

# Branch feat-make-vat-localchain-resumable-9488-
label base-feat-make-vat-localchain-resumable-9488-
pick 76c17c68d feat: make vat-localchain resumable
pick 40ccba146 fix(vow): correct the typing of `unwrap`
pick 90e062c3a fix(localchain): work around TypeScript mapped tuple bug
pick 3027adf86 fix(network): use new `ERef` and `FarRef`
label feat-make-vat-localchain-resumable-9488-
reset base-feat-make-vat-localchain-resumable-9488-
merge -C 5856dc0a0 feat-make-vat-localchain-resumable-9488- # feat: make vat-localchain resumable (#9488)

# Branch ci-mergify-clarify-queue-conditions-and-merge-conditions-9510-
label base-ci-mergify-clarify-queue-conditions-and-merge-conditions-9510-
pick 7247bd93d ci(mergify): clarify `queue_conditions` and `merge_conditions`
label ci-mergify-clarify-queue-conditions-and-merge-conditions-9510-
reset base-ci-mergify-clarify-queue-conditions-and-merge-conditions-9510-
merge -C 30e56ae34 ci-mergify-clarify-queue-conditions-and-merge-conditions-9510- # ci(mergify): clarify `queue_conditions` and `merge_conditions` (#9510)

# Branch fix-liveslots-cache-delete-does-not-return-a-useful-value-9509-
label base-fix-liveslots-cache-delete-does-not-return-a-useful-value-9509-
pick 42ea8a3f9 fix(liveslots): cache.delete() does not return a useful value
label fix-liveslots-cache-delete-does-not-return-a-useful-value-9509-
reset base-fix-liveslots-cache-delete-does-not-return-a-useful-value-9509-
merge -C a2e54e142 fix-liveslots-cache-delete-does-not-return-a-useful-value-9509- # fix(liveslots): cache.delete() does not return a useful value (#9509)

# Branch chore-swingset-re-enable-test-of-unrecognizable-orphan-cleanup-8694-
label base-chore-swingset-re-enable-test-of-unrecognizable-orphan-cleanup-8694-
pick 9930bd3a8 chore(swingset): re-enable test of unrecognizable orphan cleanup
label chore-swingset-re-enable-test-of-unrecognizable-orphan-cleanup-8694-
reset base-chore-swingset-re-enable-test-of-unrecognizable-orphan-cleanup-8694-
merge -C bc53ef711 chore-swingset-re-enable-test-of-unrecognizable-orphan-cleanup-8694- # chore(swingset): re-enable test of unrecognizable orphan cleanup (#8694)

# Pull request #9508
pick 513adc97b refactor(internal): move async helpers using AggregateError to node (#9508)

# Branch report-bundle-sizing-in-agoric-run-9503-
label base-report-bundle-sizing-in-agoric-run-9503-
pick 68af59cf2 refactor: inline addRunOptions
pick a0115ede1 feat: writeCoreEval returns plan
pick bd0edcb14 feat: stat-bundle and stat-plan scripts
pick 040520264 feat: agoric run --verbose
pick 22b43da25 feat(stat-bundle): show CLI to explode the bundle
label report-bundle-sizing-in-agoric-run-9503-
reset base-report-bundle-sizing-in-agoric-run-9503-
merge -C 7b30169f5 report-bundle-sizing-in-agoric-run-9503- # report bundle sizing in agoric run (#9503)

# Branch ci-test-boot-split-up-test-jobs-via-AVA-recipe-9511-
label base-ci-test-boot-split-up-test-jobs-via-AVA-recipe-9511-
pick 5f3c1d176 test(boot): use a single bundle directory for all tests
pick 50229bda9 ci(all-packages): split tests according to AVA recipe
label ci-test-boot-split-up-test-jobs-via-AVA-recipe-9511-
reset base-ci-test-boot-split-up-test-jobs-via-AVA-recipe-9511-
merge -C 5375019b3 ci-test-boot-split-up-test-jobs-via-AVA-recipe-9511- # ci(test-boot): split up test jobs via AVA recipe (#9511)

# Pull request #9514
pick f908f8918 fix: endow with original unstructured `assert` (#9514)

# Pull request #9535
pick 989aa1934 fix(swingset): log vat termination and upgrade better (#9535)

# Branch types-zoe-setTestJig-param-type-optional-9533-
label base-types-zoe-setTestJig-param-type-optional-9533-
pick 426a3be93 types(zoe): setTestJig param type optional
label types-zoe-setTestJig-param-type-optional-9533-
reset base-types-zoe-setTestJig-param-type-optional-9533-
merge -C bf9f03bb9 types-zoe-setTestJig-param-type-optional-9533- # types(zoe): setTestJig param type optional (#9533)

# Branch adopt-TypeScript-5-5-9476-
label base-adopt-TypeScript-5-5-9476-
pick 381b6a8ee chore(deps): bump Typescript to 5.5 release
label adopt-TypeScript-5-5-9476-
reset base-adopt-TypeScript-5-5-9476-
merge -C 0cfea88fb adopt-TypeScript-5-5-9476- # adopt TypeScript 5.5 (#9476)

# Branch retry-flaky-agoric-cli-integration-test-9550-
label base-retry-flaky-agoric-cli-integration-test-9550-
pick 2a6851001 ci: retry agoric-cli integration test
label retry-flaky-agoric-cli-integration-test-9550-
reset base-retry-flaky-agoric-cli-integration-test-9550-
merge -C c5c52ec69 retry-flaky-agoric-cli-integration-test-9550- # retry flaky agoric-cli integration test (#9550)

# Pull Request #9556
pick 0af876fb0 fix(vow): watcher args instead of context (#9556)

# Pull Request #9561
pick a4f86eb7f fix(vow): handle resolution loops in vows (#9561)

# Branch Restore-a3p-tests-9557-
label base-Restore-a3p-tests-9557-
pick d36382de9 chore(a3p): restore localchain test
pick 5ff628ec4 Revert "test: drop or clean-up old Tests"
pick b5cf8bd51 fix(localchain): `callWhen`s return `PromiseVow`
label Restore-a3p-tests-9557-
reset base-Restore-a3p-tests-9557-
merge -C c65915e60 Restore-a3p-tests-9557- # Restore a3p tests (#9557)

# Pull Request #9559
pick 6073b2bd6 fix(agoric): convey tx opts to `agoric wallet` and subcommands (#9559)

# Branch explicit-heapVowTools-9548-
label base-explicit-heapVowTools-9548-
pick 100de6833 feat!: export heapVowTools
pick 8cb1ee798 refactor: use heapVowTools import
pick 0ac677468 docs: vow vat utils
pick 9128f279a feat: export heapVowE
pick 3b0c8c177 refactor: use heapVowE
pick 9b84bfa0e BREAKING CHANGE: drop V export
pick 6623af555 chore(types): concessions to prepack
label explicit-heapVowTools-9548-
reset base-explicit-heapVowTools-9548-
merge -C 4440ce11c explicit-heapVowTools-9548- # explicit heapVowTools (#9548)

# Pull Request #9564
pick 44926a7a3 fix(bn-patch): fix bad html evasion (#9564)

# Branch Fix-upgrade-behaviors-9526-
label base-Fix-upgrade-behaviors-9526-
pick ef1e0a216 feat: Upgrade Zoe
pick e4cc97ceb Revert "fix(a3p-integration): workaround zoe issues"
pick 84dd2297e feat: repair KREAd contract on zoe upgrade
pick cb771609c test: validate KREAd character purchase
pick e1d961ec1 test: move vault upgrade from test to use phase (#9536)
label Fix-upgrade-behaviors-9526-
reset base-Fix-upgrade-behaviors-9526-
merge -C 8d05faf1d Fix-upgrade-behaviors-9526- # Fix upgrade behaviors (#9526)

# Branch Support-for-snapshots-export-command-9563-
label base-Support-for-snapshots-export-command-9563-
pick 2a3976e776 refactor(cosmos): use DefaultBaseappOptions for newApp
pick 84208e99f5 fix(cosmos): use dedicated dedicate app creator for non start commands
pick 8c1a62ddbf chore(cosmos): refactor cosmos command extension
pick 4386f8e671 feat(cosmos): support snapshot export
pick 2dabb52c4e test(a3p): add test for snapshots export and restore
label Support-for-snapshots-export-command-9563-
reset base-Support-for-snapshots-export-command-9563-
merge -C 309c7e1a94 Support-for-snapshots-export-command-9563- # Support for `snapshots export` command (#9563)

# Branch Swing-store-export-data-outside-of-genesis-file-9549-
label base-Swing-store-export-data-outside-of-genesis-file-9549-
pick f1eacbec22 fix(x/swingset): handle defer errors on export write
pick 496a430c77 feat(cosmos): add hooking kv reader
pick f476bd50fb feat(cosmos): separate swing-store export data from genesis file
pick 17a53741b7 test(a3p): add genesis fork acceptance test
label Swing-store-export-data-outside-of-genesis-file-9549-
reset base-Swing-store-export-data-outside-of-genesis-file-9549-
merge -C 3aa5d6635d Swing-store-export-data-outside-of-genesis-file-9549- # Swing-store export data outside of genesis file (#9549)

# Branch Remove-scaled-price-authority-upgrade-9585-
label base-Remove-scaled-price-authority-upgrade-9585-
pick bce49e3cb6 test: add test during upgradeVaults; vaults detect new prices
pick 88f6500c05 test: repair test by dropping upgrade of scaledPriceAuthorities
label Remove-scaled-price-authority-upgrade-9585-
reset base-Remove-scaled-price-authority-upgrade-9585-
merge -C 8376991ec4 Remove-scaled-price-authority-upgrade-9585- # Remove scaled price authority upgrade (#9585)

# Branch feat-make-software-upgrade-coreProposals-conditional-on-upgrade-plan-name-9575-
label base-feat-make-software-upgrade-coreProposals-conditional-on-upgrade-plan-name-9575-
pick 95174a2367 feat(builders): non-ambient `strictPriceFeedProposalBuilder` in `priceFeedSupport.js`
pick 5cc190d96c feat(app): establish mechanism for adding core proposals by `upgradePlan.name`
pick 52f02b75b6 fix(builders): use proper `oracleBrand` subkey case
pick ddc072d7f0 chore(cosmos): extract `app/upgrade.go`
pick b3182a49eb chore: fix error handling of upgrade vaults proposal
pick ea568a2665 fix: retry upgrade vaults price quote
label feat-make-software-upgrade-coreProposals-conditional-on-upgrade-plan-name-9575-
reset base-feat-make-software-upgrade-coreProposals-conditional-on-upgrade-plan-name-9575-
merge -C cbe061cdb8 feat-make-software-upgrade-coreProposals-conditional-on-upgrade-plan-name-9575- # feat: make software upgrade `coreProposals` conditional on upgrade plan name (#9575)

This is followed by a commit to remove the orchestration and async-flow packages from the release, as these are not baked in yet and are not deployed anyway.

@mhofman mhofman added the force:integration Force integration tests to run on PR label Jun 20, 2024
Copy link

cloudflare-workers-and-pages bot commented Jun 20, 2024

Deploying agoric-sdk with  Cloudflare Pages  Cloudflare Pages

Latest commit: 01d104e
Status: ✅  Deploy successful!
Preview URL: https://2ebe656c.agoric-sdk.pages.dev
Branch Preview URL: https://mhofman-u16-cherry-pick.agoric-sdk.pages.dev

View logs

@mhofman mhofman added force:integration Force integration tests to run on PR and removed force:integration Force integration tests to run on PR labels Jun 20, 2024
@mhofman mhofman force-pushed the mhofman/u16-cherry-pick branch 2 times, most recently from 4afb5c5 to bd14cb1 Compare June 21, 2024 00:18
mhofman and others added 24 commits June 22, 2024 18:45
closes: #9466

Expand `isRetryableReason` to take the prior return value into account
to prevent infinite loops when `watch`ing or `when`ing a plain
`Promise.reject(reasonThatIsRetryable)`. Before this, when the
watch/when would perform the retry loop, it incorrectly assumed that a
non-advancing series of retryable `shorten` attempts just needed more
retries.

Also, reject watch/when of certain promises that don't resolve promptly
(such as when the watching vat is upgraded, even if the promise's
decider has not been upgraded).

Finally, remove the deprecated `@agoric/vat-data/vow.js`, since that has
been superseded by `@agoric/vow/vat.js`.

### Security Considerations

Fixes some availability problems.

### Scaling Considerations

Less resource consumption.

### Documentation Considerations

By design, the `@agoric/vat` package assumes that all pending promises
in watch/when chains will settle promptly (such as by the end of the
crank), producing only long-lived vows or terminal results.

### Testing Considerations

New bootstrap tests added.

### Upgrade Considerations

None additional.
Tell Mergify to remove noisy HTML comments in the PR description from
the Git merge commit.

As per the NOTE in [Mergify Template
docs](https://docs.mergify.com/configuration/data-types/#template):
> By default, the HTML comments are stripped from `body`. To get the
full body, you can use the `body_raw` attribute.
refs: #9153

## Description

This partially reverts the behavioral change from `deviceTools` from #9153.

### Security Considerations

None

### Scaling Considerations

None

### Documentation Considerations

None

### Testing Considerations

None

### Upgrade Considerations

DeviceTools are bundled into device bundles, so we want to minimize difference between code executing and code on disk.
…ndler (#9507)

refs: #8596

## Description

Reverts a behavioral change questioned in #8596 (comment), which was indeed not appropriate.

If the reject handler is missing, it should result in an unhandled rejection.

### Security Considerations
None

### Scaling Considerations
None

### Documentation Considerations
None

### Testing Considerations
No coverage

### Upgrade Considerations
Maintains currently deployed behavior on mainnet
closes: #9484 closes: #9497

## Description

 - Changes the type of every return in `localchain.js` to `PromiseVow`
 - Uses `watch` for `.query()` and `.deposit()` methods and returns a `Vow`
 - Updates call sites outside `vat-localchain` to expect and unwrap PromiseVows (using `V`) 

### Upgrade Considerations

These changes better prepare us for upgrades - consumers will know to expect a `PromiseVow`.
Liveslots uses an internal `Cache` utility to hold data about virtual
objects, backed by the vatstore. PR #8752 included a change to make
its `delete()` method return a "did exist" boolean, just like a
JavaScript `Map`. Unfortunately the cache does not know whether a key
is present/absent in the backing store, so `delete()` will return an
erroneous value. It would cost an extra DB query to make this behave
as expected, and liveslots does not have a need for it.

So this commit reverts that change, and makes `delete()` return void
as before.
`queue_conditions` prevent a PR from entering the merge queue. What we
really want is `merge_conditions`, which allows PRs to embark on the
merge train while integration tests are still running, and only actually
merge when the tests are successful.

Also, we can use `queue_conditions` to consolidate and clean up the
condition logic in our `pull_request_rules`.

https://docs.mergify.com/merge-queue/setup/#configuring-the-merge-queue-rules

Tested successfully in Agoric/mergify-experiements#13
mhofman and others added 2 commits June 23, 2024 03:58
closes: #9100 
closes: #9425

## Description
Until we have better layering of the snapshots extensions in cosmos-sdk, this replaces the implementation of the `snapshots export` command to initiate the snapshot creation through our front-running mechanism used for state-sync.

Drive-by cleanup of the root command as well to address #9425

### Security Considerations
None,

### Scaling Considerations
None

### Documentation Considerations
Communicate to validators that we fixed the command and that it's usable as expected with any cosmos chain.

### Testing Considerations
Added an integration test since this only modifies the command line handling

Manually tested with the following
```
cd packages/cosmic-swingset
make scenario2-setup
make scenario2-run-chain
# wait until there are blocks, then kill
agd --home t1/n0 snapshots export
rm -rf t1/n0/data/application.db t1/n0/data/agoric
# the above removes app state without removing cometbft state as that cannot
# be restored easily in a single node chain
agd --home t1/n0 snapshot restore $snapshot_height 2
make scenario2-run-chain
# verify blocks are being produced
```

### Upgrade Considerations
Since we replace the upstream sdk command handling, any future changes upstream would have to be reflected in the override.
mhofman and others added 5 commits June 25, 2024 07:46
Closes #9567

## Description
Stores the swing-store export data outside of the genesis file, and only include a hash of it in genesis for validation.

This is the bulk of the data in the genesis file (on mainnet 6GB vs 300MB for the rest), and it serves little purpose to keep it in there. Furthermore golevelDB chokes when cosmos attempts to store the genesis file inside the DB (limit of 4GB documents)

### Security Considerations
None, the data remains validated

### Scaling Considerations
Reduces memory usage on genesis export / import. Furthermore we avoid iterating over the data twice on import, which is painfully slow for IAVL.

### Documentation Considerations
None

### Testing Considerations
Added a3p acceptance test

Manually tested as follow:
```
cd packages/cosmic-swingset
make scenario2-setup
make scenario2-run-chain
# wait until there are blocks, then kill
mkdir t1/n0/export
agd --home t1/n0 export --export-dir t1/n0/export
agd --home t1/n0 tendermint unsafe-reset-all
mv t1/n0/export/* t1/n0/config
make scenario2-run-chain
# verify blocks are being produced after genesis restart
```

### Upgrade Considerations
There is a compatibility mode to load genesis files with export data embedded.
@mhofman mhofman force-pushed the mhofman/u16-cherry-pick branch 2 times, most recently from 6818dd2 to 78d7794 Compare June 26, 2024 02:18
Chris-Hibbert and others added 11 commits June 26, 2024 04:38
This test fails, and will be fixed in the subsequent commit
refs: #9382
refs: #9584

## Description

Add a test that was supposed to be in #9283, where it says 

> A3P tests that verify that vaultFactory has been upgraded, that a new Auctioneer is running and is receiving prices. Verify that when prices drop, assets are sold via the auction, the bidder gets the proceeds, and the vaults are liquidated or reconstituted appropriately.

It was too hard to verify the results of the auction, because of the timing of vault liquidations and auction runs, so the actual check was dropped. 

The subsequent PR (#9371) that upgraded scaledPriceAuthorities seems to have broken the upgrade, and the missing test failed to warn us.

Here we test that vaultFactory was actually upgraded by verifying that it's getting prices from the new price feeds, and drop the upgrade of scaledPriceAuthority until we can figure out how to make that upgrade compatible.

### Security Considerations

Not relevant

### Scaling Considerations

Drops the upgrade of scaledPriceAuthority, which fixed part of the memory growth. This was the smaller portion of the growth, so it's more important to get the rest of the fixes in than to also include this.

### Documentation Considerations

None.

### Testing Considerations

Replaces a missing test.

### Upgrade Considerations

Repairs upgrade.
…an name (#9575)

closes: #9571

## Description

Parametrize the price feed core proposals by the upgrade handler name. The proposal for a3p remains with the deprecated ambient (from environment) oracle addresses.

Additionally fix and add some logging to the upgrade vault proposal

### Security Considerations
None

### Scaling Considerations
Not a typical scaling consideration, but this introduces a "basic" upgrade name that skips all vaults related proposals. The correctly parametrized info can be provided in the chain software upgrade proposal's upgrade info for any chain that has different configuration.

### Documentation Considerations
Need to communicate to validators that the pattern of upgrade names has changed.

### Testing Considerations
A3P and manually tested on mainfork

### Upgrade Considerations
See above
@mhofman mhofman marked this pull request as ready for review June 26, 2024 04:49
@mhofman mhofman merged commit 0549112 into dev-upgrade-16 Jun 26, 2024
79 of 86 checks passed
@mhofman mhofman deleted the mhofman/u16-cherry-pick branch June 26, 2024 05:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
force:integration Force integration tests to run on PR
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants