[General] <Set up a archive node based on docker compose and pgdump>

[corerouter]

Team or Project

No response

Environment

Mainnet

L2 block number

No response

Provide a brief description of the functionality you're trying to implement and the issue you are running into.

I need to setup archive node based on docker. So I follow docker compose file here at https://github.com/matter-labs/zksync-era/blob/main/docs/guides/external-node/docker-compose-examples/mainnet-external-node-docker-compose.yml, and after all docker run successfully, I use pg_restore command to restore pgdump file from https://storage.googleapis.com/zksync-era-mainnet-external-node-backups/external_node_latest.pgdump to postgres.

But it looks to me the pgdump files restored all historical records on mainnet2 database, instead of zksync_local_ext_node used by zksync node, as shown below:

    datname        |  size   

-----------------------+---------
postgres | 8569 kB
zksync_local_ext_node | 229 GB
template1 | 8417 kB
template0 | 8417 kB
mainnet2 | 6094 GB

As a result, when I run rpc request to retrieve transactions long before the current block, the node replied error messages shown as below:

{"jsonrpc":"2.0","error":{"code":-32602,"message":"Block with such an ID is pruned; the first retained block is 46555998"},"id":1}

Any suggestions on how to fix this issue? Thanks!

Repo Link (Optional)

No response

Additional Details

No response

[gleenai[bot]]

It seems like the issue you're encountering is related to the database used by the ZKsync node not being correctly aligned with the restored data from the pgdump file. Here are a few steps you can take to address this issue:

1. Verify Database Connection: Ensure that the ZKsync node is configured to connect to the correct database (mainnet2 in your case) where the historical records are stored. You might need to adjust the database connection settings in your docker-compose.yml file or in the ZKsync node configuration.

2. Database Restoration: Double-check the pg_restore command you used. It should specify the correct target database. The command should look something like this:

   pg_restore --dbname=mainnet2 --verbose /path/to/your/dumpfile.pgdump

   Replace mainnet2 with the actual database name if it's different.

3. Pruning Settings: The error message about pruned blocks suggests that there might be a configuration setting in ZKsync that controls data retention or block pruning. Review the ZKsync node's configuration for any settings related to data pruning or block retention, and adjust them to retain more historical data if necessary.

4. Restart Node: After making changes to the database or configuration, restart your ZKsync node to ensure all settings are applied correctly.

If these steps do not resolve the issue, you might want to consult the ZKsync Era documentation for more detailed guidance on setting up and configuring archive nodes, or reach out to the ZKsync community or support for more specific assistance.

Your feedback helps us serve you better. If this fully addresses your question, please give it a thumbs up. If it doesn't, please give it a thumbs down. This will inform the community and team so we can improve our support.

[ischasny]

Hello!

As far as I understand you'd like to change pruning behaviour of your node. That can be done via adjusting the following configuraitons

pruning:
  enabled: true
  chunk_size: 10
  removal_delay_sec: 60
  data_retention_sec: 3600

[corerouter]

Thanks for your reply.

However, I set EN_PRUNING_ENABLED: true or EN_PRUNING_ENABLED: false at the docker compose file at https://github.com/matter-labs/zksync-era/blob/899ffc074ac21cf0c3c0bb1e0c876cfd3d8fda72/docs/guides/external-node/docker-compose-examples/mainnet-external-node-docker-compose.yml#L99
The rpc reply from the node was still:
{"jsonrpc":"2.0","error":{"code":-32602,"message":"Block with such an ID is pruned; the first retained block is 46555998"},"id":1}

Then I changed the database name for postgres database from zksync_local_ext_node to mainnet2 at
https://github.com/matter-labs/zksync-era/blob/899ffc074ac21cf0c3c0bb1e0c876cfd3d8fda72/docs/guides/external-node/docker-compose-examples/mainnet-external-node-docker-compose.yml#L88
in order to make the node use the data restored from the pgdump file. Then during the node startup, it show:
mainnet-node-external-node-1 | error: migration 20240803083814 was previously applied but is missing in the resolved migrations

Then I inspect table schema between database mainnet2 and zksync_local_ext_node and find some tables are different. So I can't simply append data from mainnet2 to zksync_local_ext_node in order to make node return earlier block info for rpc.

It looks to me I need to config the node to return latest data from zksync_local_ext_node database and earlier data from mainnet2 database. But I did not find such options in zksync node codes.

Any suggestions?
Thanks!

[ischasny]

Just to confirm - you want to set up a full archival node, pointed at the mainnet, that has the entire history of blocks with no pruning. Does that sound correct?

[corerouter]

Yes. This is my goal--Setup a zksync mainnet archival node based on docker compose.

Please advise. Thanks!

[ischasny]

I'd recommend going through this tutorial - it explains everything, including pruning.

[Ajes1337]

the link is broken, here is the correct link, afaik; https://github.com/matter-labs/zksync-era/blob/main/docs/src/guides/external-node/00_quick_start.md

I have read most of the relevant docs now, and I cant still find out how to configure a node as Archival. the AI bot on discord tells me the answer is here: https://github.com/matter-labs/ansible-en-role but I cant see that the guide state which flags/arguments/configuration I need todo in a docker-compose or such, to run zksync-era as a archival node.... 🤷‍♂️