refactor(vm): Unify the vm interface for two latest vms #325
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Co-authored-by: Igor Borodin <[email protected]>
# What ❔ After transferring to vise metrics, `mempool_` transaction stage labels became incorrect (e.g., `mempool_Added` instead of `mempool_added`). This PR changes them back. ## Why ❔ These labels are used on some internal Grafana dashboards. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] ~Tests for the changes have been added / updated.~ *not applicable* - [ ] ~Documentation comments have been added / updated.~ *not applicable* - [x] Code has been formatted via `zk fmt` and `zk lint`.
# What ❔ Data for mainnet upgrade to version 16 ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`.
# What ❔ Refactors action queue used by EN to be more idiomatic. Covers sync logic with tests. ## Why ❔ The idea is to reuse EN code other than the fetcher with the new sync mechanism that will be brought by consensus. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`.
# What ❔ With boojum upgrades will go through governor. PR adds functionality to process such upgrades while leaving old processor as well. ## Why ❔ Prepare for boojum upgrade ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`.
# What ❔ - Removes `fee_ticker` module. - Removes `token_trading_volume` fetcher. - Removes some (now unused) methods from DAL. - Removes some (now unused) types. ## Why ❔ Dead code. ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] ~~Tests for the changes have been added / updated.~~ - [ ] ~~Documentation comments have been added / updated.~~ - [x] Code has been formatted via `zk fmt` and `zk lint`.
# What ❔ Allows to recover a Merkle tree from a snapshot (collection of tree entries ordered by ascending key). The recovery procedure is fault-tolerant (may be paused and restarted). ## Why ❔ This is one of components for recovering a node from a snapshot. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`.
… proof wrapping (#240) # What ❔ * fix(crypto): update snark-vk to be used in server and update args for proof wrapping ## Why ❔ * Rn the server sends the blob with protocol version using scheduler_vk which is different from the one used in prover(snark_scheduler_vk) ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ *] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ *] Tests for the changes have been added / updated. - [ *] Documentation comments have been added / updated. - [ *] Code has been formatted via `zk fmt` and `zk lint`.
# What ❔ - fix broken links ## Why ❔ - The link is no longer valid and cannot be opened. ## Checklist - [ ✅ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ✅ ] Tests for the changes have been added / updated. - [ ✅ ] Documentation comments have been added / updated. - [ ✅ ] Code has been formatted via `zk fmt` and `zk lint`.
# What ❔ - update Security link in README.MD ## Why ❔ - when you click on Security on README. it takes you to invalid link ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`.
# What ❔ - fix documentation link ## Why ❔ - The link cannot be accessed. ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`.
# What ❔ - Update docs ## Why ❔ <img width="1184" alt="image" src="https://github.com/matter-labs/zksync-era/assets/148654386/0d1efcd4-4009-4f91-9f29-0b445bd2863a"> ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`.
# What ❔ Updated the era-zkevm_opcode_defs dependency and minimal set of other dependencies which required update to a compatible version. This unlocked updated sha2 and sha3 dependencies to 0.10.8. Prover integration test run for extra safety: https://github.com/matter-labs/zksync-era-workflows/actions/runs/6612607366 ## Why ❔ To include matter-labs/era-zkevm_opcode_defs#16 which in turn allows us to depend on up-to-date versions of sha2 and sha3.
…refunds (#274) # What ❔ During VM run in state keeper we save all storage refunds that are returned by oracle. Later in basic witness generation these refunds are reused. Also, `zk_evm` is updated so that it now passes correct timestamps for `estimate_refunds_for_write` queries. Correct timestamps are necessary for VM storage oracle to rollback `returned_refunds` ## Why ❔ With new VM, pubdata tracer mutates oracle affecting storage refunds. As this tracer is only run in state keeper and not in witness generator we must save refunds from state keeper and use them in wit-gen. ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`.
# What ❔ RocksDB write stalls are still happening, this time for a different reason. Previously, they were caused by too many immutable memtables, this time – by too many level-0 SST files. This PR: - Tunes RocksDB options some more (the main tuning point is [optimizing level-style compaction](https://docs.rs/rocksdb/latest/rocksdb/struct.Options.html#method.optimize_level_style_compaction)). - Increases the number of retries on stall and introduces exponential backoff. - Introduces a dozen of RocksDB metrics that should help monitoring RocksDB health. ## Why ❔ Having write stalls leads to panics and is obviously bad. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. --------- Co-authored-by: AnastasiiaVashchuk <[email protected]>
# What ❔ - Tests actions produced by the main node fetcher. As a side effect, adds a way to launch a JSON-RPC HTTP server in-process and adds a primitive test for it as well. - Refactors the main node fetcher slightly, extracting the EN–main node client interface to a trait. ## Why ❔ - Creates reference action queue contents that may be tested against when an alternative fetcher based on the consensus gossip network is implemented. - Test coverage good. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`.
…related) (#273) # What ❔ - Fixes some flow errors on the hyerpchain initializer - Changes the hyperchain command name to a "zk stack" umbrella command - Addes docker steup command - builds server and generates a docker compose file ## Why ❔ - Improves the experience of starting hyperchains ## Checklist - [X] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [X] Tests for the changes have been added / updated. - [X] Documentation comments have been added / updated. - [X] Code has been formatted via `zk fmt` and `zk lint`.
# What ❔ - fix broken links ## Why ❔ - The link is no longer valid and cannot be opened. ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ✅ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ✅ ] Tests for the changes have been added / updated. - [ ✅ ] Documentation comments have been added / updated. - [ ✅ ] Code has been formatted via `zk fmt` and `zk lint`.
# What ❔ - update link ## Why ❔ - Unable to open, 404 ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`.
# What ❔ - Update contact_us link in 01_intro.md ## Why ❔  ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`.
# What ❔ [A previous fix](#265) didn't really work judging by Merkle tree behavior on the stage env. This PR makes the initialization timeout configurable (and increases the default value from 10s to 30s; 30s is approximately equal to the compaction duration) and slightly increases the number of retries on stalled writes. ## Why ❔ Having write stalls leads to panics and is obviously bad. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`.
# What ❔ - Bump zksolc and zksvyper binaries, build solidity from source (to support arm64 in future) - Minor CI and Dockerfiles refactoring here and there - Bump ancient gcloud CLI in zk-env publish CI - Bump Ubuntu in zk-env to 22.04 - **Build zk-environment (non-CUDA) on arm64** ## Why ❔ To unblock building of the core services on arm64 arch ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`.
# What ❔ Add EU GAR registry for publish images ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`.
# What ❔ Subj ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`.
# What ❔ Pre-pulling Docker manifests before trying to glue multi-arch image from them ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`.
# What ❔ Subj ## Why ❔ All of our JS machinery is now expecting Node 18, so this aligns Node version with what's considered being "our" version ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`.
🤖 I have created a release *beep* *boop* --- ## [16.1.0](core-v16.0.2...core-v16.1.0) (2023-10-24) ### Features * Add new commitments ([#219](#219)) ([a19256e](a19256e)) * arm64 zk-environment rust Docker images and other ([#296](#296)) ([33174aa](33174aa)) * **config:** Extract everything not related to the env config from zksync_config crate ([#245](#245)) ([42c64e9](42c64e9)) * **eth-watch:** process governor upgrades ([#247](#247)) ([d250294](d250294)) * **merkle tree:** Expose Merkle tree API ([#209](#209)) ([4010c7e](4010c7e)) * **merkle tree:** Snapshot recovery for Merkle tree ([#163](#163)) ([9e20703](9e20703)) * **multivm:** Remove lifetime from multivm ([#218](#218)) ([7eda27c](7eda27c)) * Remove fee_ticker and token_trading_volume fetcher modules ([#262](#262)) ([44f7179](44f7179)) * **reorg_detector:** compare miniblock hashes for reorg detection ([#236](#236)) ([2c930b2](2c930b2)) * Rewrite server binary to use `vise` metrics ([#120](#120)) ([26ee1fb](26ee1fb)) * **types:** introduce state diff record type and compression ([#194](#194)) ([ccf753c](ccf753c)) * **vm:** Improve tracer trait ([#121](#121)) ([ff60138](ff60138)) * **vm:** Move all vm versions to the one crate ([#249](#249)) ([e3fb489](e3fb489)) ### Bug Fixes * **crypto:** update snark-vk to be used in server and update args for proof wrapping ([#240](#240)) ([4a5c54c](4a5c54c)) * **db:** Fix write stalls in RocksDB ([#250](#250)) ([650124c](650124c)) * **db:** Fix write stalls in RocksDB (again) ([#265](#265)) ([7b23ab0](7b23ab0)) * **db:** Fix write stalls in RocksDB (for real this time) ([#292](#292)) ([0f15919](0f15919)) * Fix `TxStage` string representation ([#255](#255)) ([246b5a0](246b5a0)) * fix typos ([#226](#226)) ([feb8a6c](feb8a6c)) * **witness-generator:** Witness generator oracle with cached storage refunds ([#274](#274)) ([8928a41](8928a41)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
fix: fix the broken link for get_txs_fee_in_wei # What ❔ - fix the broken link for get_txs_fee_in_wei ## Why ❔ - Documentation error ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`.
# What ❔ - update docs ## Why ❔ - Unreachable link ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`.
# What ❔ - Do not cancel workflows running in main - Make manual trigger actually work ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`.
# What ❔ Subj ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`.
# What ❔ Subj ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`.
# What ❔ push FRI docker image to Asia and Europe ## Why ❔ ability to spawn FRI provers in other region ## Checklist - [ *] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ *] Tests for the changes have been added / updated. - [ *] Documentation comments have been added / updated. - [* ] Code has been formatted via `zk fmt` and `zk lint`.
# What ❔ Subj ## Why ❔ We need to debug recent load test performance degradation. As a first step, we're going to try switching back to using previous iteration of CI runner image ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`.
# What ❔ ## Update setup loading for node agg circuit ## Why ❔ One one setup key exist for all node agg ## Checklist - [ *] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [* ] Tests for the changes have been added / updated. - [* ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`.
This PR changes the shutdown order of the external node to a more graceful one. # Why? Currently sutting down the EN because of reorgs and inconsistency checks may be raceful and a shutting down consistency checker may bring down the reorg detector with it which would prevent the reorg detector from doing it's job in the legitimate cases of reorgs where the node actually needs to be rolled back to some previous state. # What happens currently? The concerns have been focused around the logic of consistency checker that used to panic when an inconsistency was found. That is no longer the case (`anyhow::bail!` used to be `panic`): https://github.com/matter-labs/zksync-era/blob/feb8a6c7053cc5e0202088f6a1f7644316e1ad05/core/lib/zksync_core/src/consistency_checker/mod.rs#L181-L183 however, this error would still unconditionally bring the node down because currently the handler for `ConsistencyChecker` is a part of `wait_tasks` that would resolve if any of the tasks has resolved: https://github.com/matter-labs/zksync-era/blob/feb8a6c7053cc5e0202088f6a1f7644316e1ad05/core/bin/external_node/src/main.rs#L401-L404 The proposed logic works as follows: The handle to reorg detector is fused and on the node shutdown, after all the components are gracefully shutdown we manually check the result of the reorg detector. This logic handles both situations: 1. node is shutting down 2. reorg detector has found a reorg and is performing a rollback and shutdown.
…component (#156) # What ❔ This PR introduces a new component that will be ran alongside all components in the service. The component extracts all information needed for running Basic Witness Generator in a binary serialized Input and uploads it to Object Store (GCS for now). ## Why ❔ Part of work decoupling Prover Subsystems from Server Core. Today, Witness Generators need Server Core Database's access to generate witnesses. More information can be found [here](https://www.notion.so/matterlabs/Decouple-Witness-Generator-from-Core-Database-313ece3293d74ee8aea1d1fca823df15) for the Witness Generator split and [here](https://www.notion.so/matterlabs/Segregating-prover-and-server-to-use-API-for-communication-6cdcb7a508e84420bdfcb153af0e2524) for the Prover Subsystems and Server core split. ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`.
# What ❔ Made unit tests use separate real dbs created in a temporary postgres container. ## Why ❔ It makes the tests more realistic and greatly simplifies the dal crate.
# What ❔ Fixes (hopefully) the root cause of recent RocksDB misbehavior. After recent refactoring, it improperly cancels background compactions / flushes each time a `RocksDB` instance is dropped, which leads to compactions / flushes being constantly interrupted. (After refactoring, `RocksDB` instances are `Clone`eable and act essentially as `Arc` wrappers; a new instance is created and dropped, in particular, [when processing](https://github.com/matter-labs/zksync-era/blob/ec41692063909742005acd656694a8f0838fcf33/core/lib/zksync_core/src/metadata_calculator/updater.rs#L385) each chunk of L1 batches.) A proper solution, implemented by this PR, is to cancel background work when dropping an internal, non-`Arc`'d DB instance. ## Why ❔ That's a bug that makes RocksDB behave *very* unstably, as witnessed by recent issues with it. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Code has been formatted via `zk fmt` and `zk lint`. --------- Co-authored-by: Igor Borodin <[email protected]>
# What ❔ Setting the ground for bigger boojum changes. Adding boojum related fields to the data model and types. Extending state tree to have a concept of `StateDiffs`. ## Why ❔ We want to reduce the size of the overall boojum merge, this sets the basis for more extensive changes. These changes are all meant to be non-intrusive. ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. --------- Co-authored-by: “perekopskiy” <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What ❔
Implement one common interface for all vms. It's the first attempt to do it and it's done only for two latest vms.
In the future all vms will be migrated to the one interface.
Why ❔
We need to unify our work with VM now it's always unpredicable what are we calling and when
Checklist
zk fmtandzk lint.