[ZK Stack] - Why the zksync external node requires 64GB of Ram to run?

[sachanayush47]

Team or Project

No response

Module Affected

• System contracts (era-system-contracts)
• Smart contracts (era-contracts)
• zkEVM (era-zk_evm)
• Server (zksync-era)
• Prover (era-boojum)
• Fee withdrawer (era-fee-withdrawer)
• Block Explorer (clock-explorer)
• Portal - Wallet + Bridge (dapp-portal)
• Other (please specify below)

Rust Version

1.80.1

Operating System

• Ubuntu
• MacOS
• Windows
• Other (Please specify below)

Issue Description

We are attempting to run an external node for a self-deployed zkSync Hyperchain, but the node crashes (process killed by the OS) within 30 seconds due to low system memory i.e 8GB. According to the zkSync documentation, the external node requires 64GB of RAM. However, running an Ethereum node only requires 8GB of RAM, as mentioned in this article.

Our question is: why does a zkSync node require 4x the memory compared to an Ethereum node? Are there any lower minimum system requirements, such as 16GB of RAM? And is there any way to optimize the node to run on systems with lower specifications?

Edit:
Our main node is running on 8GB without any problem.
Prover is not running
L1 batch commit data generator mode: Validium

Expected Behavior

External node should be keep on running.

Current Behavior

External node process killed by OS

Repository Link (if applicable)

No response

Additional Details

Some logs from external node:

024-10-23T11:16:24.327758Z  INFO access_storage{l1_batch_number=L1BatchNumber(51)}: zksync_state::rocksdb: Secondary storage for L1 batch #51 initialized, size is 329
2024-10-23T11:16:24.328515Z DEBUG access_storage{l1_batch_number=L1BatchNumber(51)}: zksync_state::storage_factory: Using RocksDB-based storage rocksdb_l1_batch_number=52
2024-10-23T11:16:24.328822Z  INFO zksync_vm_executor::batch::factory: Starting executing L1 batch #52
2024-10-23T11:16:24.344251Z  INFO zksync_state_keeper::io::seal_logic: Sealing L2 block 104 with timestamp 2024-10-21 16:15:03 UTC (L1 batch 52) with 1 (1 L2 + 0 L1) txs, 4 events
2024-10-23T11:16:25.244140Z  INFO zksync_node_sync::batch_status_updater: Batch 51: committed
2024-10-23T11:16:25.245448Z  INFO zksync_node_sync::batch_status_updater: Batch 51: proven
2024-10-23T11:16:25.245472Z  INFO zksync_node_sync::batch_status_updater: Batch 51: executed
2024-10-23T11:16:25.252071Z  INFO zksync_node_sync::batch_status_updater: Commit status change: number 51, hash 0x1fb2…1281, happened at 2024-10-21 16:10:55.607414 UTC
2024-10-23T11:16:25.475586Z  INFO zksync_node_sync::batch_status_updater: Prove status change: number 51, hash 0xfa8a…133f, happened at 2024-10-21 16:11:35.725615 UTC
2024-10-23T11:16:25.518773Z  INFO zksync_node_sync::batch_status_updater: Execute status change: number 51, hash 0x6283…b273, happened at 2024-10-21 16:12:01.787273 UTC
2024-10-23T11:16:25.574137Z DEBUG zksync_health_check: Changed health of `batch_status_updater` from {"status":"ready","details":{"last_committed_l1_batch":50,"last_executed_l1_batch":50,"last_proven_l1_batch":50}} to {"status":"ready","details":{"last_committed_l1_batch":51,"last_executed_l1_batch":51,"last_proven_l1_batch":51}}
2024-10-23T11:16:26.361328Z  WARN TreeUpdater::step{next_l1_batch_to_process=L1BatchNumber(52)}: zksync_db_connection::instrument: Query get_sealed_l1_batch_number called at /home/zksync/ecosystem_holesky/zksync-era/core/lib/dal/src/blocks_dal.rs:117 [requested by `metadata_calculator` at /home/zksync/ecosystem_holesky/zksync-era/core/node/metadata_calculator/src/updater.rs:205] is executing for more than 100ms
2024-10-23T11:16:26.361244Z  WARN zksync_db_connection::instrument: Query get_sealed_l2_block_number called at /home/zksync/ecosystem_holesky/zksync-era/core/lib/dal/src/blocks_dal.rs:134 [requested by `api` at core/node/node_framework/src/implementations/layers/web3_api/server/sealed_l2_block.rs:29] is executing for more than 100ms
│
■  Command failed to run
│  
■    Status:
│      signal: 9 (SIGKILL)
│    Stdout:
│      
│  
│    Stderr:
│      
│  
■  Failed to start server
│  
▲    0: Failed to run external node
│    1: Command failed to run: cargo run --release --bin zksync_external_node -- --config-path /home/zksync/ecosystem_holesky/chains/validium_holesky/configs/external_node/general.yaml --secrets-path /home/zksync/ecosystem_holesky/chains/validium_holesky/configs/external_node/secrets.yaml --external-node-config-path /home/zksync/ecosystem_holesky/chains/validium_holesky/configs/external_node/external_node.yaml
│  
│    2: Command failed to run: cargo run --release --bin zksync_external_node -- --config-path /home/zksync/ecosystem_holesky/chains/validium_holesky/configs/external_node/general.yaml --secrets-path /home/zksync/ecosystem_holesky/chains/validium_holesky/configs/external_node/secrets.yaml --external-node-config-path /home/zksync/ecosystem_holesky/chains/validium_holesky/configs/external_node/external_node.yaml
│  
└  Failed to run command

Prior Research

• Yes, I searched for similar issues and did not find any matching ones.

[gleenai[bot]]

The requirement for 64GB of RAM for running a ZKsync node, as opposed to the lower requirement for an Ethereum node, primarily stems from the differences in their underlying technologies and the operations they perform.

1. Complexity of Operations: ZKsync nodes handle more complex operations than typical Ethereum nodes. They are involved in generating and verifying zero-knowledge proofs, which are computationally intensive and require more memory to manage the data involved in these processes efficiently.

2. State Size and Management: ZKsync nodes may also manage larger state sizes compared to Ethereum nodes. The management of this state, especially when dealing with numerous or complex transactions within a block, can demand more memory.

3. Performance and Scalability: Higher RAM ensures better performance and scalability when processing transactions in ZKsync. This is crucial for maintaining the speed and efficiency of the network, especially under high transaction loads.

Unfortunately, reducing the RAM requirement significantly below 64GB might affect the node's ability to perform these operations effectively, leading to issues like the crashes you've experienced. Optimizing the node to run on systems with lower specifications might not be feasible without compromising on some aspects of performance or functionality.

For more detailed discussions and potential updates on system requirements, you might want to visit the ZKsync developers discussions on Github.

Your feedback helps us serve you better. If this fully addresses your question, please give it a thumbs up. If it doesn't, please give it a thumbs down. This will inform the community and team so we can improve our support.

[uF4No]

Reply from the bot is quite accurate. Eth and ZKsync nodes are very different hence the different requirements.

That said, we've teams running with less than 32GB of RAM but that's the minimum recommended.

[sachanayush47]

Reply from the bot is quite accurate. Eth and ZKsync nodes are very different hence the different requirements.

That said, we've teams running with less than 32GB of RAM but that's the minimum recommended.

How main node is able to operate with just 8GB but not external node?

[windycrypto]

The bot's answer doesn't clearly stated where the memory is consumed, we are running the external nodes for Era and Zero, the Zero chain, though memory increases, consumes about 8GB. We have external node for cronos zkevm and zklink , consumes 8GB and 4GB respectively.

So could we have a technical answer where does the >30GB memory consumes in Era?