Plugin causing validator to not keep up

[Dassy23]

Are you able to deploy the plugin and keep up with the network, I am consistently falling behind. If you are able to keep up would love to know what flags/settings you have set.

[lijunwangs]

Hi @Dassy23 -- The validator I ran with master branch has a plugin writing transactions, accounts, block, secondary indexing to the database and the validator was able to catch up with the network after a couple of hours. You will need to ensure the validator and the postgres server to have enough computing powers. I found that I had to give my nodes more memory to handle the load.

[Dassy23]

Thats awesome to hear! What version of the Solana validator did you run? Also if you don't mind can you copy your solana-validator command in this thread? My machine specs are pretty good I believe

• CPU 32 cores @ 2.9 GHz (Intel w/ AVX-512F & SHA-NI instruction sets)
• Storage 1 x 480 GB SDD + 2 x 1 TB NVMe
• Memory 256GB
• Network 2x 10 Gbps with unlimited free transfers

[lijunwangs]

The following is my validator run command:

cargo run --release --bin solana-validator -- --identity ~/validator-keypair.json --vote-account ~/vote-account-keypair.json --rpc-port 8899 --dynamic-port-range 8000-8100 --entrypoint mainnet-beta.solana.com:8001 --limit-ledger-size 50000000 --no-port-check --trusted-validator 7Np41oeYqPefeNQEHSv1UDhYrehxin3NStELsSKCT4K2 --trusted-validator GdnSyH3YtwcxFvQrVVJMm1JhTS4QVX7MFsX56uJLUfiZ --no-check-vote-account --minimal-snapshot-download-speed 40000000 --maximum-snapshot-download-abort 20 --accounts-shrink-optimize-total-space true --accounts-shrink-ratio 0.8 --accountsdb-plugin-config /home/lijun_solana_com/plugin-config-all-txns-and-accounts-idx.json --ledger validator-ledger-mainnet

My plugin config:

{
"libpath": "/home/lijun_solana_com/solana-accountsdb-plugin-postgres-1/target/release/libsolana_accountsdb_plugin_postgres.so",
"host": "lijun-dev-2",
"user": "solana",
"port": 5433,
"threads": 80,
"batch_size": 20,
"panic_on_db_errors": true,
"store_account_historical_data": true,
"accounts_selector" : {
"accounts" : [""]
},
"transaction_selector": {
"mentions" : [""]
},
"index_token_owner": true,
"index_token_mint": true
}

My virtual machines configuration: I run in google GCP:

Validator: 64 CPU Intel(R) Xeon(R) CPU @ 2.80GHz, Memory: 384 G.

PostgreSQL: 48 CPU Intel(R) Xeon(R) CPU @ 2.80GHz, Memory: 384 G.

[Dassy23]

thank you ser : )

[Dassy23]

@lijunwangs does the thread count(in the config) correlate to the the CPU at all? For example, would it make sense to have 64 CPU and 150 threads?

[lijunwangs]

The threads count in a system correlate to the count of the CPUs. But they do not have exact one to one mapping. You will see the system usually has much more threads than the CPUs. This makes sense as the thread is not necessarily using the CPU all the time. In our case, it will spend some time waiting on the network IOs (writing or waiting for results from other systems). It does not mean the more threads the better either -- because at certain point the threads has to compete for the CPU and the threads will spend more time context switching than doing real work. The optimal thread count has to be done with some performance measurement with work loads. In this case, you have to also tweak the PostgreSQL concurrent connection counts.