[Bug]: addValidator to a Fuji PoA L1 fails with Error: failed to connect to a threshold of stake

[leopaul36]

Describe the bug
addValidator to a Fuji PoA L1 fails with Error: failed to connect to a threshold of stake

To Reproduce

Just followed exactly the steps at https://docs.avax.network/avalanche-l1s/deploy-a-avalanche-l1/fuji-testnet

Logs

L1 create

❯ avalanche blockchain create testblockchain
✔ Subnet-EVM
✔ Proof Of Authority
✔ Get address from an existing stored key (created from avalanche key create or avalanche key import)
✔ validator_manager_owner
✓ Validator Manager Contract owner address 0xd6C9B994deCF937C69d3B1aBF5d8661c779b1D28
✔ I want to use defaults for a test environment
✔ Chain ID: 12345█
Token Symbol: TST
prefunding address 0x8db97C7cEcE249c2b98bDC0226Cc4C2A57BF52FC with balance 1000000000000000000000000
File /home/leo/.avalanche-cli/subnets/testblockchain/chain.json successfully written
✓ Successfully created blockchain configuration
Run 'avalanche blockchain describe' to view all created addresses and what their roles are

L1 deploy

❯ avalanche blockchain deploy testblockchain
✔ Fuji Testnet
Deploying [testblockchain] to Fuji
✔ Use stored key
✔ validator_manager_owner
Using [P-fuji1fa5wl78gj8xalm00x6z28wwfkf4xxcvpztkj43] to be set as a change owner for leftover AVAX
You can use your local machine as a bootstrap validator on the blockchain
This means that you don't have to to set up a remote server on a cloud service (e.g. AWS / GCP) to be a validator on the blockchain.
✔ Yes
Installing avalanchego-v1.12.0-fuji...
avalanchego-v1.12.0-fuji installation successful
Backend controller started, pid: 46294, output at: /home/leo/.avalanche-cli/local/testblockchain-local-node/server.log
✓ Local cluster testblockchain-local-node not found. Creating...
Warning: Fuji Bootstrapping can take several minutes
Starting local avalanchego node using root: /home/leo/.avalanche-cli/local/testblockchain-local-node ...
✓ Booting Network. Wait until healthy...
✓ Avalanchego started and ready to use from /home/leo/.avalanche-cli/local/testblockchain-local-node

Node logs directory: /home/leo/.avalanche-cli/local/testblockchain-local-node/node1/logs

Network ready to use.

URI: http://127.0.0.1:9650
NodeID: NodeID-HE2rcVNArcsb8Bi8Ls1uJyHZKCnQX2WHX

Your blockchain control keys: [P-fuji1fa5wl78gj8xalm00x6z28wwfkf4xxcvpztkj43]
Your subnet auth keys for chain creation: [P-fuji1fa5wl78gj8xalm00x6z28wwfkf4xxcvpztkj43]
CreateSubnetTx fee: 0.000010278 AVAX
Subnet has been created with ID: sBW8Awx2e5Jg9QduQJ6ytsULmfTYFkCQpRuaFMDcaytwBUzZ9
Now creating blockchain...
CreateChainTx fee: 0.000095460 AVAX
+--------------------+----------------------------------------------------+
| DEPLOYMENT RESULTS |                                                    |
+--------------------+----------------------------------------------------+
| Chain Name         | testblockchain                                     |
+--------------------+----------------------------------------------------+
| Subnet ID          | sBW8Awx2e5Jg9QduQJ6ytsULmfTYFkCQpRuaFMDcaytwBUzZ9  |
+--------------------+----------------------------------------------------+
| VM ID              | tGBrM94jbkesczgqsL1UaxjrdxRQQobs3MZTNQ4GrfhzvpiE8  |
+--------------------+----------------------------------------------------+
| Blockchain ID      | 2KZYGeh3tvjPb5wRRxNb3qREqfxr2oCA1BHcpz3iyr2RBpFLmq |
+--------------------+                                                    +
| P-Chain TXID       |                                                    |
+--------------------+----------------------------------------------------+
Now calling ConvertSubnetToL1Tx...
ConvertSubnetToL1Tx fee: 0.000042914 AVAX
ConvertSubnetToL1Tx ID: opHrdDPNeKdRcsNSgMQ4tSJxeaUqfo3PEwPZXao8CxbKwnrXw
Waiting for L1 to be converted into sovereign blockchain ... 100% [===============]

Restarting node node1 to track subnet
Waiting for rpc http://127.0.0.1:9650/ext/bc/2KZYGeh3tvjPb5wRRxNb3qREqfxr2oCA1BHcpz3iyr2RBpFLmq/rpc to be available
✓ testblockchain-local-node successfully tracking testblockchain
✓ Checking if node is healthy...
✓ Node is healthy after 0 seconds
Initializing Proof of Authority Validator Manager contract on blockchain testblockchain ...
✓ Proof of Authority Validator Manager contract successfully initialized on blockchain testblockchain

AddValidator

❯ avalanche blockchain addValidator testblockchain
✔ Fuji Testnet
✔ Use stored key
✔ validator_manager_owner
What is the NodeID of the node you want to add as a blockchain validator?: NodeID-A8rdxK5Yddh9kwbMkgkpb7oEN1GAxiBkz
Next, we need the public key and proof of possession of the node's BLS
Check https://docs.avax.network/api-reference/info-api#infogetnodeid for instructions on calling info.getNodeID API
What is the node's BLS public key?: 0xa2d8524afaa3954d6d030e372af6606e959cf62b5442be07d1113c75c67c4cf6742a3338180eec1a325e9010eaaa3656
What is the node's BLS proof of possession?: 0xaeb3362dba19f937825f671c2a8395e91b5350909bff25918eb759f7c79bbd7beb86f67d36082f63a98504c1e6ef3bcf0980ea141e8a583373f4294f01401644c2386f39c47b53aa91a47ae90310c2c29e536de4d12f862f7d74c11f5ab2899a
What is the node's BLS proof of possession?: 0xaeb3362dba19f937825f671c2a8395e91b5350909bff25918eb759f7c79bbd7beb86f67d36082f63a98504c1e6ef3bcf0980ea141e8a583373f4294f01401644c2386f39c47b53aa91a47ae90310c2c29e536de4d12f862f7d74c11f5ab2899a
Validation manager owner 0xd6C9B994deCF937C69d3B1aBF5d8661c779b1D28 pays for the initialization of the validator's registration (Blockchain gas token)
Validator's balance is used to pay for continuous fee to the P-Chain
When this Balance reaches 0, the validator will be considered inactive and will no longer participate in validating the L1
What balance would you like to assign to the validator (in AVAX)?: 1
?: 1█
✔ Get address from an existing stored key (created from avalanche key create or avalanche key import)
✔ validator_manager_owner
✔ Get address from an existing stored key (created from avalanche key create or avalanche key import)
✔ validator_manager_owner
Validator weight: 20
Error: failed to connect to a threshold of stake

Operating System
Ubuntu 22.04.4 LTS

Additional context
Running addValidator with --aggregator-log-level debug, logs show:

Failed to get API Node ID","error":"failed to decode client response: the method info.getNodeID is not available

After adding --aggregator-extra-endpoints http://127.0.0.1:9650, logs show:

{"level":"debug","timestamp":"2024-12-02T10:29:44.801+1100","logger":"p2p-network","caller":"peers/external_handler.go:94","msg":"Received AppError message","message":"chain_id:\"\\xad\\xd7VQ\\x9bb\\xc6\\xd2ȕ$\\x88x\\x86\\x1d\\x08\\x02.\\xe8\\xb2'\\xa4\\xa0-\\xbb\\x19\\xf6[\\xa3\\xc6\\xd8\\xf1\" request_id:1094531497 error_code:1 error_message:\"failed to parse addressed call message: couldn't unmarshal interface: unknown type ID 1\""}

[felipemadero]

Hi! (1) Could you pass it the full aggregator logs for each case? if possible with verbo log level (2) which is the avago release/version of your validator? (the coudn't unmarshall interface points to some avago compat issue)

[leopaul36]

Hi! (1) Could you pass it the full aggregator logs for each case? if possible with verbo log level (2) which is the avago release/version of your validator? (the coudn't unmarshall interface points to some avago compat issue)

Hey @felipemadero, it was using v1.12.0-fuji as show in the logs above.

My latest attempt runs with v1.12.0 since it was just released but it still does not work.

Attaching the new node's debug logs and full debug relayer output.

logs.tar.gz
relayer_debug_logs.txt

[leopaul36]

Update @felipemadero, it seems that adding --aggregator-extra-endpoints http://127.0.0.1:9650 on the first addValidator command attempt made it work 👍

[leopaul36]

New update: addValidator commands went through but something is definitely wrong. Look at the results for platform.getValidatorsAt of my L1:

curl --location --request POST https://api.avax-test.network/ext/bc/P \
--header 'Content-Type: application/json' \
--data-raw '{
    "jsonrpc":"2.0",
    "id"     :1,
    "method" :"platform.getValidatorsAt",
    "params" :{
        "height": "proposed",
        "subnetID":"uShcYQqf94FfcFqu2b1qfWDVFYkUkNYgjSVuTLpGrCj5WwKWS"
    }
}' | jq

Output:

{
  "jsonrpc": "2.0",
  "result": {
    "NodeID-111111111111111111116DBWJs": {
      "publicKey": null,
      "weight": "20"
    },
    "NodeID-Foi8JSSaPTLn2QbPxspeKWyqT3kEAZL1c": {
      "publicKey": "0xb136db9d1ea5ced511ddf721679de86bf741f4ff2b3383d05588d9fdeb870c97d36102c8ec712d331a8d5ccfc64f2cf2",
      "weight": "100"
    }
  },
  "id": 1
}

This NodeID is clearly not the one I added + publicKey is empty...

[felipemadero]

All issues fixed by #2396 and #2387

[felipemadero]

New update: addValidator commands went through but something is definitely wrong. Look at the results for platform.getValidatorsAt of my L1:

curl --location --request POST https://api.avax-test.network/ext/bc/P \
--header 'Content-Type: application/json' \
--data-raw '{
    "jsonrpc":"2.0",
    "id"     :1,
    "method" :"platform.getValidatorsAt",
    "params" :{
        "height": "proposed",
        "subnetID":"uShcYQqf94FfcFqu2b1qfWDVFYkUkNYgjSVuTLpGrCj5WwKWS"
    }
}' | jq

Output:

{
  "jsonrpc": "2.0",
  "result": {
    "NodeID-111111111111111111116DBWJs": {
      "publicKey": null,
      "weight": "20"
    },
    "NodeID-Foi8JSSaPTLn2QbPxspeKWyqT3kEAZL1c": {
      "publicKey": "0xb136db9d1ea5ced511ddf721679de86bf741f4ff2b3383d05588d9fdeb870c97d36102c8ec712d331a8d5ccfc64f2cf2",
      "weight": "100"
    }
  },
  "id": 1
}

This NodeID is clearly not the one I added + publicKey is empty...

empty node information related to a node that got out of funds