[General] Local ZkSync Node Is Shutting Down When Sending Many Transactions At A Same Time

[FaizanNehal]

Team or Project

No response

Environment

Testnet

L2 block number

No response

Provide a brief description of the functionality you're trying to implement and the issue you are running into.

I created my own zksync ecosystem using the Zk Stack guide.
I am testing the scalability of ZkSync locally on my machine by deploying Uniswap v2 on it, creating a pair and sending many swaps transactions at a same time through a transaction generation script.
This transaction generation usually generates 100-300 transactions in one instance, and we can run as many instances as possible. I am running 6 instances so around 1000-2000 transactions in 1-2 minutes.

Below is the screenshot from node after it has shut down:

Repo Link (Optional)

No response

Additional Details

In the transaction generator, you can replicate real world traffic by running a instances that have different local IP. You can see below that I am using 6 instances.

You can also try the transaction generator from the repo below. You would have to make few changes in the code according to your local environment.

https://github.com/bogatyy/rollup-amm-tps-test/tree/main

[gleenai[bot]]

It seems like your local ZKsync node is experiencing shutdowns when handling a high volume of transactions. This could be due to several reasons such as resource limitations (CPU, RAM), improper node configuration, or issues within the node software itself that may not handle burst loads efficiently.

Here are a few steps you can take to troubleshoot and potentially resolve the issue:

1. Check System Resources: Ensure that your system has enough RAM and CPU power to handle the load. ZKsync nodes, especially when processing many transactions, can be resource-intensive.

2. Node Configuration: Review the node's configuration settings. Adjustments in the configuration might be needed to handle higher loads, such as increasing the number of worker threads or optimizing database settings.

3. Logs and Errors: Examine the node logs for any specific error messages or warnings that occur before the shutdown. This can provide clues on what might be causing the issue.

4. Update and Patch: Ensure that your ZKsync node and all related software are up to date with the latest patches and updates.

5. Stress Test Gradually: Instead of starting with a high number of transactions, gradually increase the load and monitor how the node behaves. This can help identify at what point the node struggles.

6. Consult Documentation and Community: Review the ZKsync documentation for any similar issues or optimization tips. Also, consider reaching out on the ZKsync developers discussions on Github for more specific advice or similar experiences from other developers.

If the problem persists, you might want to provide detailed logs and system information in a discussion on the ZKsync developers forum for more tailored support.

Your feedback helps us serve you better. If this fully addresses your question, please give it a thumbs up. If it doesn't, please give it a thumbs down. This will inform the community and team so we can improve our support.

[kaymomin]

Hi @FaizanNehal — It’s likely because, in a local setup, the file system is used as an object store. When the node tries to write too many files from a single process, the OS may shut it down.

An easy fix is to avoid running the bwip component since they probably don’t use proofs in their environment.

A more robust (but costlier) solution is to switch to using GCS as an object store.

[FaizanNehal]

Hi @kaymomin thanks for the info. How can I avoid running the bwip component?
I am already not running the prover.

I have followed this guideline:
https://docs.zksync.io/zk-stack/running-a-zk-chain/quickstart

[kaymomin]

Seems like by default, these components are started:

api, tree, eth, state_keeper, house_keeper, commitment_generator, da_dispatcher, vm_runner_protective_reads.

The bwip component isn’t part of this default set, so its not running anyway.

But if you want to avoid running any specific component, you’ll need to explicitly specify the components you do want to run. For example, if you don’t want to run the da_dispatcher, you’d use:

zkstack server --components api,tree,eth,state_keeper,housekeeper,commitment_generator,vm_runner_protective_reads

Try this approach, and let me know if it's still causing issues.

[kaymomin]

@FaizanNehal just checking if the solution helped in anyway?