Skip to content

Panthalassa worker installation

Jump to bottom
Anton Bernatskiy edited this page Apr 14, 2024 · 27 revisions

Worker installation instructions

================================

Requirements

To run a single worker you need:

  • 4 vCPU
  • 16GB RAM
  • 1TB SSD
  • Stable 24/7 Internet connection, at least 1Gbit
  • public IP and two ports open for incoming traffic:
    • one UDP port for p2p communication (default 12345),
    • one TCP port for Prometheus metrics (default 9090).
  • 100_000 tSQD tokens (funded to a special vesting contract owned by the Primary Address).
  • Some Arbitrum Sepolia ETH (for gas)

The same Primary Address can be used to register multiple workers (if the vesting contract has sufficient funds).

Worker itself is available in two forms:

  1. as a Docker image,
  2. as source code.

Please refer to the appropriate section for further instructions.

Running a worker Docker image

Prerequisites

  • Docker engine (tested with version 24.0.4)
  • Docker compose (tested with version 2.19.1)

Steps

  1. Create a new directory and save this run_worker.sh script in it.

  2. Generate your key file by running the following command in the newly created directory. It will display your peer ID – please copy this ID, as it will be needed for on-chain worker registration.

    $ docker run --rm subsquid/rpc-node:0.2.5 keygen >key
Your peer ID: <THIS IS WHAT YOU NEED TO COPY>

    ⚠️ Note: Please make sure that the generated file is safe and secure (i.e. it will not be deleted accidentally and cannot be accessed by unauthorized parties).

  3. Register your worker by logging into the testnet Web APP with the Primary Wallet you provided in the onboarding form. You will need the Peer ID from the previous step. See the detailed instructions here.

  4. Make the run_worker.sh script executable and run PUBLIC_IP=<your public ip> ./run_worker.sh <DATA_DIR> up -d to start the node. <DATA_DIR> is a directory where the downloaded data will be stored – please make sure it exists and can accommodate up to 1 TB of data. If you want to use a different port than the default 12345, set the LISTEN_PORT environment variable.

    ⚠️ Note: Setting PUBLIC_IP is currently required for the node to be publicly discoverable.

  5. Check the status of the containers with docker container ls. Check the worker logs using docker logs -f worker-1. After some time the worker should output some info on the downloaded data chunks. See Docker docs for a reference of Docker commands.

Building from the source

  1. Install prerequisites (Python, Rust, Protobuf).

    apt install python3.11 python3.11-venv gcc protobuf-compiler
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

  2. Clone & build a network node. Obtain GITHUB_TOKEN in the private node operator chat.

    git clone --branch 0.2.5 https://[email protected]/subsquid/subsquid-network.git
cd subsquid-network/transport
cargo build --release --bin node

  3. Create a new Python virtual env and install the worker package. Use the same GITHUB_TOKEN as on the previous step.

    python3.11 -m venv env
. env/bin/activate
pip install -U "subsquid-eth-archive[p2p-worker] @ git+https://[email protected]/subsquid/[email protected]"

  4. Enter the subsquid-network/transport directory and generate your key file by running the command below. It will display your peer ID – please copy this ID, as it will be needed for on-chain worker registration.

    cargo run --bin keygen > <FILE PATH WHERE YOUR KEY WILL BE SAVED>
Your peer ID: <THIS IS WHAT YOU NEED TO COPY>

  5. Register your worker by logging into the testnet Web APP with the Primary Wallet you provided in the onboarding form. You will need the Peer ID from the previous step. See the detailed instructions here.

  6. Run the network node using the command below (assuming you are in the subsquid-network/transport directory). It uses nohup, but you may also consider daemonizing it with systemd. The libp2p address+port combination, specified by the --p2p-listen-addr flag, should be available for external connections (in the reference command below the node listens on 12345):

    nohup cargo run --release --bin node -- \
    --p2p-listen-addrs /ip4/0.0.0.0/udp/12345/quic-v1 \
    --key <FILE PATH WHERE YOUR KEY IS SAVED> \
    --boot-nodes '12D3KooWSRvKpvNbsrGbLXGFZV7GYdcrYNh4W2nipwHHMYikzV58 /dns4/testnet.subsquid.io/udp/22445/quic-v1' \
    --boot-nodes '12D3KooWQC9tPzj2ShLn39RFHS5SGbvbP2pEd7bJ61kSW2LwxGSB /dns4/testnet.subsquid.io/udp/22446/quic-v1' \
    --bootstrap \
    > network.log 2>&1 &

    If you see the error below in the console, ignore it.

    2023-06-21T08:59:52.427Z ERROR [subsquid_network_transport::transport] Error broadcasting message: InsufficientPeers

Important

Be careful when supplying the path to the key you created at step 4 via --key. If you supply the wrong path, a new random key will be automatically created there and your node will attempt to operate with a new (unregistered) peer ID - unsuccessfully.

  1. Set up the credentials for read-only S3 access, sentry, and RPC via env variables:

    cat <<EOF > .worker-env
export AWS_ACCESS_KEY_ID=66dfc7705583f6fd9520947ac10d7e9f
export AWS_SECRET_ACCESS_KEY=a68fdd7253232e30720a4c125f35a81bd495664a154b1643b5f5d4a4a5280a4f
export AWS_S3_ENDPOINT=https://7a28e49ec5f4a60c66f216392792ac38.r2.cloudflarestorage.com
export AWS_REGION=auto
export SENTRY_DSN=https://[email protected]/4505589334081536
EOF

source .worker-env

  2. Run the worker process in a terminal where you have your virtualenv active. You may run it with systemd instead of nohup to tolerate server restarts.

    nohup python -m sqa.worker.p2p \
    --scheduler-id 12D3KooWQER7HEpwsvqSzqzaiV36d3Bn6DZrnwEunnzS76pgZkMU \
    --logs-collector-id 12D3KooWC3GvQVqnvPwWz23sTW8G8HVokMnox62A7mnL9wwaSujk \
    --data-dir </path/where/you/want/to/store/downloaded/data> \
    --proxy localhost:50051 \
    --prometheus-port <port_on_which_you_want_to_expose_prometheus_metrics> \
    --rpc-url https://sepolia-rollup.arbitrum.io/rpc \
    > worker.log 2>&1 &

Retiring the worker (deprecated)

Withdrawing an active worker is done in 2 steps:

  1. Execute this command to unregister the worker.

    docker run subsquid/worker-registration:mirovia <PEER_ID> <WALLET_PRIVATE_KEY> deregister

  2. Wait for an epoch to end (currently max 30 minutes) and execute this command. The bond will be be transferred back to your wallet immediately.

    docker run subsquid/worker-registration:mirovia <PEER_ID> <WALLET_PRIVATE_KEY> withdraw

FAQ

Where do I find PEER_ID ?

It is printed when you run keygen (see the steps for running worker).

It is also in the first line of the worker log output. For the docker setup, list the containers with docker container ls and inspect the rpc_node container logs with docker logs -f <rpc_node_container_id>

If you installed your worker from source, check the network.log file.

In both cases, the log line you are looking for should look like this

INFO  [subsquid_network_transport::transport] Local peer ID: <THIS IS WHAT YOU NEED TO COPY>

How do I get tSQD tokens ?

The tokens are distributed to dedicated vesting contracts owned by the Primary address. Only the node operators that were onboarded via the DeForm application received tSQD allocations and are thus capable of running worker(s)

I see Insufficient peers... error in the worker logs

Just ignore it

Can I move my worker to another server

Yes, copy the key file (located in the key file located at KEY_PATH) to the new working directory before starting. You don't need to re-register your worker.

I have just started my worker but see no logs

This is normal. Wait for a few minutes and the logs should show some data being downloaded.

Should I build it from source or run with Docker?

Docker makes for an easier setup. Building from sources is suitable only for experienced Linux/Unix users.

How do I check that my worker is updated to the latest version?

Check the pings endpoint and locate the version by your PeerID.

Which Linux distro is recommended?

We recommend Ubuntu 22.04 LTS

I see error from daemon in stream: Error grabbing logs

This is a docker issue, not a problem with the worker. Look at this GitHub issue and this Stackoverflow thread for more context.

How do I check if my worker is up-to-date and running?

Copy your peer ID and look for an entry on this page. If the last ping timestamp is <1 minute ago, and the listed version is the most recent one, you should be good. Alternatively, you can use this Python script.

Clone this wiki locally