feat(eigen-client-m0): merge extra features #364
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…merge-main Eigen client extra features merge main
## What ❔
Upgrade rustls
## Why ❔
```
error[vulnerability]: rustls network-reachable panic in `Acceptor::accept`
┌─ /github/workspace/Cargo.lock:601:1
│
601 │ rustls 0.23.16 registry+https://github.com/rust-lang/crates.io-index
│ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ security vulnerability detected
│
├ ID: RUSTSEC-[20](https://github.com/matter-labs/zksync-era/actions/runs/12011183823/job/33479628638?pr=3199#step:4:21)24-0399
├ Advisory: https://rustsec.org/advisories/RUSTSEC-2024-0399
├ A bug introduced in rustls 0.23.13 leads to a panic if the received
TLS ClientHello is fragmented. Only servers that use
`rustls::server::Acceptor::accept()` are affected.
Servers that use `tokio-rustls`'s `LazyConfigAcceptor` API are affected.
Servers that use `tokio-rustls`'s `TlsAcceptor` API are not affected.
Servers that use `rustls-ffi`'s `rustls_acceptor_accept` API are affected.
├ Announcement: https://github.com/rustls/rustls/issues/[22](https://github.com/matter-labs/zksync-era/actions/runs/12011183823/job/33479628638?pr=3199#step:4:23)27
├ Solution: Upgrade to >=0.[23](https://github.com/matter-labs/zksync-era/actions/runs/12011183823/job/33479628638?pr=3199#step:4:24).18 (try `cargo update -p rustls`)
```
## What ❔ <!-- What are the changes this PR brings about? --> <!-- Example: This PR adds a PR template to the repo. --> <!-- (For bigger PRs adding more context is appreciated) --> Add three new components to node's healthcheck: - General (i.e., version, last migration) - State Keeper - Eth Sender ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zkstack dev fmt` and `zkstack dev lint`. --------- Co-authored-by: Danil <[email protected]>
…atter-labs#3321) ## What ❔ This PR adds more information to the response of the `unstable_getTeeProofs` API endpoint, enabling the [client][1] that sent the [request][2] to determine whether it makes sense to retry fetching the TEE proof for a particular batch number. ## Why ❔ Currently, the [TEE verifier][1] – the tool for continuous SGX attestation and batch signature verification – is [stuck][3] on batches that failed to be proven and are marked as `permanently_ignored`. The tool should be able to distinguish between batches that are permanently ignored (and should be skipped) and batches that have failed but will be retried. This PR enables that distinction. Example use cases: - requesting TEE proof for a batch with the `permanently_ignored` status ``` $ curl -i -X POST -H "Content-Type: application/json" --data '{"jsonrpc": "2.0", "id": 1, "method": "unstable_getTeeProofs", "params": [14, "sgx"] }' 'http://localhost:3152' HTTP/1.1 200 OK content-type: application/json; charset=utf-8 vary: origin, access-control-request-method, access-control-request-headers access-control-allow-origin: * content-length: 207 date: Tue, 26 Nov 2024 12:42:48 GMT {"jsonrpc":"2.0","result":[{"l1BatchNumber":14,"teeType":"sgx","pubkey":null,"signature":null,"proof":null,"provedAt":"2024-11-20T15:43:46.112146Z","status":"permanently_ignored","attestation":null}],"id":1} ``` - requesting TEE proof for a batch with the `failed` status ``` $ curl -i -X POST -H "Content-Type: application/json" --data '{"jsonrpc": "2.0", "id": 1, "method": "unstable_getTeeProofs", "params": [15, "sgx"] }' 'http://localhost:3152' HTTP/1.1 200 OK content-type: application/json; charset=utf-8 vary: origin, access-control-request-method, access-control-request-headers access-control-allow-origin: * content-length: 194 date: Tue, 26 Nov 2024 12:44:19 GMT {"jsonrpc":"2.0","result":[{"l1BatchNumber":15,"teeType":"sgx","pubkey":null,"signature":null,"proof":null,"provedAt":"2024-11-20T15:43:46.121432Z","status":"failed","attestation":null}],"id":1} ``` - requesting TEE proof for a batch with the `generated` status ``` $ curl -i -X POST -H "Content-Type: application/json" --data '{"jsonrpc": "2.0", "id": 1, "method": "unstable_getTeeProofs", "params": [28, "sgx"] }' 'http://localhost:3152' HTTP/1.1 200 OK content-type: application/json; charset=utf-8 vary: origin, access-control-request-method, access-control-request-headers access-control-allow-origin: * content-length: 229 date: Tue, 26 Nov 2024 12:45:27 GMT {"jsonrpc":"2.0","result":[{"l1BatchNumber":28,"teeType":"sgx","pubkey":"0506070809","signature":"0001020304","proof":"1011121314","provedAt":"2024-11-20T15:21:16.129128Z","status":"generated","attestation":"0403020100"}],"id":1} ``` - requesting TEE proof for a non-existent batch ``` $ curl -i -X POST -H "Content-Type: application/json" --data '{"jsonrpc": "2.0", "id": 1, "method": "unstable_getTeeProofs", "params": [1337, "sgx"] }' 'http://localhost:3152' HTTP/1.1 200 OK content-type: application/json; charset=utf-8 vary: origin, access-control-request-method, access-control-request-headers access-control-allow-origin: * content-length: 36 date: Tue, 26 Nov 2024 12:46:08 GMT {"jsonrpc":"2.0","result":[],"id":1} ``` Relevant database entries for the use cases mentioned above: ``` zksync_server_localhost_legacy=# SELECT * FROM tee_proof_generation_details WHERE l1_batch_number IN (14, 15, 28, 1337); l1_batch_number | status | signature | pubkey | proof | tee_type | created_at | updated_at | prover_taken_at -----------------+---------------------+--------------+--------------+--------------+----------+----------------------------+----------------------------+---------------------------- 14 | permanently_ignored | | | | sgx | 2023-11-20 15:27:47.281293 | 2024-11-20 15:43:46.112146 | 2024-11-20 15:43:46.106042 15 | failed | | | | sgx | 2024-11-20 15:27:47.287777 | 2024-11-20 15:43:46.121432 | 2024-11-20 15:43:46.115853 28 | generated | \x0001020304 | \x0506070809 | \x1011121314 | sgx | 2024-11-20 12:56:33.055642 | 2024-11-20 15:21:16.129128 | 2024-11-20 14:53:14.25949 (3 rows) ``` [1]: https://github.com/matter-labs/teepot/blob/main/bin/verify-era-proof-attestation/src/main.rs [2]: https://github.com/matter-labs/teepot/blob/1a8a9f17fa7284f83c41a63d37fe380aef6d550d/bin/verify-era-proof-attestation/src/proof.rs#L15-L21 [3]: https://grafana.matterlabs.dev/goto/unFqf57Hg?orgId=1 ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [x] Code has been formatted via `zkstack dev fmt` and `zkstack dev lint`.
🤖 I have created a release *beep* *boop* --- ## [17.1.1](matter-labs/zksync-era@prover-v17.1.0...prover-v17.1.1) (2024-11-26) ### Bug Fixes * Add prometheus exporter to circut prover ([matter-labs#3311](matter-labs#3311)) ([920eba1](matter-labs@920eba1)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
…#3266) ## What ❔ Records root hashes of the last hard-pruned L1 batch in pruning logs. ## Why ❔ This allows to check tree integrity after recovery after pruning. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zkstack dev fmt` and `zkstack dev lint`.
## What ❔ Create reqwest client only once. Additionally HttpClient exports metric `calls` with all the requests and correct status codes. <!-- What are the changes this PR brings about? --> <!-- Example: This PR adds a PR template to the repo. --> <!-- (For bigger PRs adding more context is appreciated) --> ## Why ❔ Creating reqwest client is expensive because it initializes TLS, loads certificates, etc. So it should be create only once and reused. Create new internal mod http_client instead of patching zksync_utils because fn `send_request_with_retries` is used only in prover_autoscaler and outdated prover_fri, which will be removed soon. <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [x] Code has been formatted via `zkstack dev fmt` and `zkstack dev lint`. ref ZKD-1855
## What ❔ Generally moves some code from gateway integration branch into main. High-level list of changes: - `eth_watch`: refactors watcher's client so it can use L2-specific features if gateway is queried. New event processor is added for chains that settle on gateway, it calculates part of merkle proof for L2->L1 logs. - `api`: new method is added to `unstable` namespace that is utilized by the new event processor. This method encapsulates a few storage reads + builds merkle tree. This is a temporary measure and method will likely be removed (that's why it's in unstable namespace) pretty soon after a contract getter that does exactly the same is merged into contracts, replacing unstable_getChainLogProof invocation with a simple `eth_call` - `consistency_checker`, `tree_data_fetcher`: adds support for reading batch commitment from either L1 or GW depending on what the SL is for a batch. - adds support for new format of commitBatches, proveBatches, executeBatches data encoding - bug fixes: `L2_NATIVE_TOKEN_VAULT_ADDRESS` was used instead of `L2_ASSET_ROUTER_ADDRESS` in `core/bin/external_node/src/node_builder.rs`, mixed up if/else branches in `core/node/commitment_generator/src/lib.rs` - some clean up, renaming, refactoring: `resolve_l1_batch_to_l2_block` + `get_block_details` are replaced with just `get_batch_details` in `batch_status_updater`, some code for old protocol versions (pre-boojum or pre-shared-bridge) removed, `l1_` prefix is added to some addresses to avoid confusion ## Why ❔ Finalize and merge some gateway features ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zkstack dev fmt` and `zkstack dev lint`.
## What ❔ Update consensus dependencies to the latest version. This effectively deploys ChonkyBFT.
## What ❔ - replaces sk l1 gas criterion with l1 l2 txs criterion - removes commit aggregation in eth sender - enables execute aggregation only for chains that settle on L1 - adds sk criterion for l2 l1 logs ## Why ❔ - make criteria work with gateway ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zkstack dev fmt` and `zkstack dev lint`.
…r-labs#3341) ## What ❔ Ability to publish zk-environment from manual trigger ## Why ❔ Sometimes we need to force-update zk-environment ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zkstack dev fmt` and `zkstack dev lint`.
* Move inclusion logic * Remove unused import
* Move inclusion logic * Remove verify cert
* Change eth conf depth for settlement layer * Remove blob size limit * Move reserved * Change function name * Add constants to lib * Fix compilation
* initial commit * remove anyhow err * Format code * Fix compilation --------- Co-authored-by: Gianbelinche <[email protected]>
* Add option to download points * Fix test * Fix clippy * Change name to points source * Fix test * Fix test
…s with mock tests (#356) * initial commit * create verifier client trait * remove unwraps * Update core/node/da_clients/src/eigen/verifier.rs Ignore test Co-authored-by: Gianbelinche <[email protected]> * Update core/node/da_clients/src/eigen/sdk.rs Co-authored-by: Gianbelinche <[email protected]> * change ignore test comment * Format code --------- Co-authored-by: Gianbelinche <[email protected]>
…on docs (#360) * Remove steps to run from integration docs * Combine readme * Clarify readme
…and-main feat(eigen-client-extra-features): Merge Main
gianbelinche
changed the title
juanbono
approved these changes
Dec 6, 2024
gianbelinche
merged commit 0169b1e
into
eigen-client-m0-implementation
15 of 35 checks passed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What ❔
This PR merges the new changes on extra features
It reverts changes to dispatcher back to the original sequential one, since there is no performance gain on the concurrent one anymore.
Why ❔
Checklist
zkstack dev fmtandzkstack dev lint.