[Bug]: ~/.avalanche-cli/config.json seems to be ignored when using avalanche blockchain deploy <blockchain_name> --local

[ckz8780]

Describe the bug
I am probably misunderstanding the correct usage of the CLI since the Etna upgrade, but it seems that the CLI now ignores the config.json file in ~/.avalanche-cli/.

To Reproduce
Beginning from a completely wiped out ~/.avalanche-cli/ directory with no logs, binaries or anything except the empty directory structure and the .last_actions.json file:

~/.avalanche-cli# ls -la ./*
./bin:
total 12
drwxr-xr-x 2 root root 4096 Jan  8 13:58 .
drwxr-xr-x 1 root root 4096 Jan  8 13:56 ..

./key:
total 12
drwxr-xr-x 2 root root 4096 Jan  8 13:59 .
drwxr-xr-x 1 root root 4096 Jan  8 13:56 ..

./local:
total 12
drwx------ 2 root root 4096 Jan  7 21:00 .
drwxr-xr-x 1 root root 4096 Jan  8 13:56 ..

./logs:
total 12
drwxr-x--- 2 root root 4096 Jan  8 13:58 .
drwxr-xr-x 1 root root 4096 Jan  8 13:56 ..

./nodes:
total 12
drwxr-xr-x 2 root root 4096 Jan  8 13:58 .
drwxr-xr-x 1 root root 4096 Jan  8 13:56 ..

./plugins:
total 12
drwxr-xr-x 2 root root 4096 Jan  7 20:40 .
drwxr-xr-x 1 root root 4096 Jan  8 13:56 ..

./repos:
total 12
drwxr-xr-x 2 root root 4096 Jan  7 20:40 .
drwxr-xr-x 1 root root 4096 Jan  8 13:56 ..

./runs:
total 12
drwxr-xr-x 2 root root 4096 Jan  7 21:00 .
drwxr-xr-x 1 root root 4096 Jan  8 13:56 ..

./snapshots:
total 12
drwxr-xr-x 2 root root 4096 Jan  7 20:40 .
drwxr-xr-x 1 root root 4096 Jan  8 13:56 ..

./subnets:
total 12
drwxr-xr-x 2 root root 4096 Jan  7 20:59 .
drwxr-xr-x 1 root root 4096 Jan  8 13:56 ..

./vms:
total 12
drwxr-xr-x 2 root root 4096 Jan  7 20:40 .
drwxr-xr-x 1 root root 4096 Jan  8 13:56 ..

1. Create a config.json in ~/.avalanche-cli:

~/.avalanche-cli# vim config.json 

{
  "metricsenabled": true,
  "node-config": {
    "http-allowed-hosts": "*",
    "http-allowed-origins": "*",
    "http-host": "0.0.0.0"
  }
}

2. Run avalanche blockchain create myblockchain and with the following prompt responses:

   • Subnet-EVM
   • Proof of Authority
   • Custom controller of Validator Manager contract > <my desired admin address>
   • I don't want to use default values
   • Use latest release version
   • Chain ID: <my desired id>
   • Token Symbol <my desired symbol>
   • Define a custom token allocation
   • Add an address <my desired token holder address>
   • Amount to allocate: 100000000000 (100 billion)
   • Confirm and finalize > yes
   • Enable native minting
   • Add an address > Admin role > <my desired admin address>
   • Confirm Allow list > yes
   • Medium block size (C-chain settings)
   • Constant gas prices
   • Allow adjustment of tx fees as needed
   • Add an address > Admin role > <my desired admin address>
   • Confirm Allow list > yes
   • Burn transaction fees
   • Enable c-chain/other chain interoperability
   • Allow anyone to deploy/make transactions

   Output:

   Installing subnet-evm-v0.7.0...
   subnet-evm-v0.7.0 installation successful
   File /root/.avalanche-cli/subnets/myblockchain/chain.json successfully written
   ✓ Successfully created blockchain configuration
   

3. Run avalanche blockchain deploy myblockchain --local

   Relevant portion of output:

   +--------------------------------------------------------------------------+
   |                               PRIMARY NODES                              |
   +-------+------------------------------------------+-----------------------+
   | NAME  | NODE ID                                  | LOCALHOST ENDPOINT    |
   +-------+------------------------------------------+-----------------------+
   | node1 | NodeID-7Xhw2mDxuDS44j42TCB6U5579esbSt3Lg | http://127.0.0.1:9650 |
   +-------+------------------------------------------+-----------------------+
   | node2 | NodeID-MFrZFVCXPv5iCn6M9K6XduxGTYp891xXZ | http://127.0.0.1:9652 |
   +-------+------------------------------------------+-----------------------+
   
   +---------------------------------------------------------------------------+
   |                                  L1 NODES                                 |
   +-------+------------------------------------------+------------------------+
   | NAME  | NODE ID                                  | LOCALHOST ENDPOINT     |
   +-------+------------------------------------------+------------------------+
   | node1 | NodeID-KJnvmwp6VGkFc6phpjVPjEKuHMXTfhrSd | http://127.0.0.1:44973 |
   +-------+------------------------------------------+------------------------+
   
   +------------------------------------------------------------------------------------------------------+
   |                                           WALLET CONNECTION                                          |
   +-----------------+------------------------------------------------------------------------------------+
   | Network RPC URL | http://127.0.0.1:44973/ext/bc/zemLMedASTVzsdqDYwsb95p9vfwJQ9zHEYEDWGTYLmjFFirZ/rpc |
   +-----------------+------------------------------------------------------------------------------------+
   | Network Name    | myblockchain                                                                       |
   +-----------------+------------------------------------------------------------------------------------+
   | Chain ID        | <my chain id>                                                                      |
   +-----------------+------------------------------------------------------------------------------------+
   | Token Symbol    | MYTOKEN                                                                            |
   +-----------------+------------------------------------------------------------------------------------+
   | Token Name      | MYTOKEN Token                                                                      |
   +-----------------+------------------------------------------------------------------------------------+
   

4. Note the following:

   • The RPC URL and Node Endpoints are 127.0.0.1 and not 0.0.0.0 (as defined in the http-host setting in the config.json file)
   • The RPC URL's port is a random high port rather than 9650
   • Running netstat -tuln indicates that the service is bound to the loopback address rather than 0.0.0.0:

   ~/.avalanche-cli# netstat -tuln
   Active Internet connections (only servers)
   Proto Recv-Q Send-Q Local Address           Foreign Address         State      
   tcp        0      0 127.0.0.1:44977         0.0.0.0:*               LISTEN     
   tcp        0      0 127.0.0.1:37945         0.0.0.0:*               LISTEN     
   tcp        0      0 127.0.0.1:41949         0.0.0.0:*               LISTEN     
   tcp        0      0 127.0.0.1:33919         0.0.0.0:*               LISTEN     
   tcp        0      0 127.0.0.1:45507         0.0.0.0:*               LISTEN     
   tcp        0      0 127.0.0.1:38793         0.0.0.0:*               LISTEN     
   tcp        0      0 127.0.0.1:43305         0.0.0.0:*               LISTEN     
   tcp        0      0 127.0.0.1:33257         0.0.0.0:*               LISTEN     
   tcp        0      0 127.0.0.1:44973         0.0.0.0:*               LISTEN    <---------     
   tcp6       0      0 :::45905                :::*                    LISTEN     
   tcp6       0      0 :::9650                 :::*                    LISTEN     
   tcp6       0      0 :::9651                 :::*                    LISTEN     
   tcp6       0      0 :::9652                 :::*                    LISTEN     
   tcp6       0      0 :::9653                 :::*                    LISTEN     
   tcp6       0      0 :::8090                 :::*                    LISTEN     
   tcp6       0      0 :::8091                 :::*                    LISTEN     
   tcp6       0      0 :::39615                :::*                    LISTEN     
   tcp6       0      0 :::8097                 :::*                    LISTEN     
   tcp6       0      0 :::8098                 :::*                    LISTEN     
   tcp6       0      0 :::9092                 :::*                    LISTEN  
   

Expected behavior
I expected setting http-host to 0.0.0.0 in config.json to bind the deployed blockchain to 0.0.0.0 so that it could be accessed from the outside world (assuming my firewall rules allow it). I also expected the RPC URL to use port 9650. This is what used to happen with the CLI v1.7.1 which is what I was using pre-Etna, but since the Etna upgrade in AvalancheGo and upgrading the CLI to the latest (1.8.4), the local blockchain is configured with a different random high port each time, and I cannot figure out how to properly bind the local address to 0.0.0.0 so it's available from outside the local machine. This is a problem because a) the random high port means I have to reconfigure my clients to connect to a different RPC URL each time the local blockchain is stopped/started and b) currently I can't connect to the RPC URL from anything but localhost anyway.

Screenshots
N/A

Logs
N/A - Let me know if there's any generated files that would be helpful

Operating System
I have reproduced this both on Ubuntu 22.04.5 LTS and MacOS Sonoma 14.7.1

Additional context
In general, I'm having trouble determining where things are supposed to be configured in the latest version of the CLI, and the documentation is either not up to date or I'm looking in the wrong places. Just a few examples:

• How do I change the number of nodes via the CLI? There are two commands (--num-nodes and --num-local-nodes). The former seems to do nothing when used like avalanche blockchain deploy myblockchain --local --num-nodes 5 (it still starts two Primary Nodes and a single L1 node) and the latter sometimes works - starting two Primary Nodes and 5 L1 nodes on unpredictable ports, still ignoring the config.json file - but other times fails with "Error: failure waiting for local cluster myblockchain bootstrapping". There doesn't seem to be a good source of information on what I'm doing wrong anywhere that I can find
• How do I customize the RPC URL?
• How do I properly bind the running service to the wildcard address?
• How do I configure individual nodes to listen on specific ports?
• I keep seeing references to chain.json, subnet.json, node-config.json and other config files, but no definitive explanation anywhere of what they can do, where they should live, what their available config options are, etc etc.

Much of this stuff seems to have associated functionality in avalanchego itself, but it seems like the CLI is out of date or needs the docs updated to describe proper usage.

Any help would be GREATLY appreciated! Thank you!

[felipemadero]

Hi! Thanks for this detailed feedback. First point, yes, the local network deploy changed quite a bit in latest release.
README is outdated atm (#2542).
It tells to use the node-config section of the main config file to set http-host, but that currently only
affects the local network primary validators, not the L1 bootstrap validators.
Those take the input from the blockchain node config, as set using avalanche blockchain configure.
So, create a json with your desired contents and use avalanche blockchain configure cmd previously
to doing the deploy.

[felipemadero]

Secondly, yes currently L1 validator ports are dynamic, decided when creating the network (#2543). For this case I can make two comments (besides the team going to provide control of this):

• if you execute avalanche network stop and then avalanche network start you are expected to continue having the same port (it is only dynamic when doing the deploy). So maybe you can find a workaround for your case if you can use the assigned port in your configurations
• after a network stop, you can modify the port by hand on a file called ~/.avalanche-cli/local/blockchainName- local-node-local-network/network.json . This is by no means the expected workflow, just another indication that may help you atm.

[felipemadero]

Finally, some other responses:

• taking a big note on having clear and centralized documentation on how to use chain.json, subnet.json, node-config.json, and in general, CLI config options
• --num-nodes does affect the number of primary validators on blockchain deploy (probably you are operating on a local network already deployed that continues having 2 nodes. you can either user network stop to save local network state, or network clean to start again from scratch)
• --num-local-nodes does affect the number of L1 validators on blockchain deploy, if you are hitting am intermitent failure there Error: failure waiting for local cluster myblockchain bootstrapping this is bug ([Bug]: blockchain deploy with 5 bootstrap validators has intermittent failure #2544). we will try to reproduce, if that is not possible, I would like to ask you logs after such a failure
  (probably all ~/.avalanche-cli/local/blockchainName-local-node-local-network contents)
• taking a node on having clear doc on how to set both number of nodes
• RPC URL customization is not supported atm
• configuring individual node ports is not supported atm, but taking note (you can edit network.json file atm just as a dirt workaround)
• we changed information provided for the deployed local network on purpose, to stop showing http://0.0.0.0 urls and always show http://127.0.0.1 urls, even when http-host is set.

Thanks very much!

[esrasiler]

• after a network stop, you can modify the port by hand on a file called ~/.avalanche-cli/local/blockchainName- local-node-local-network/network.json . This is by no means the expected workflow, just another indication that may help you atm.

Can I ask a clarification about this? Let's say I've deployed an L1 network named myblockchain, and it is running on port 34567. If I stop the network using avalanche network stop and then edit the network.json file to change the port and set the IP to 0.0.0.0, will I be able to restart the network with the new settings by running avalanche network start ? Or should I make the change before running avalanche blockchain deploy myblockchain?

I am also having difficulties with this process and would appreciate any guidance.

[felipemadero]

1. pls don't set the ip to 0.0.0.0, but "http-host" to "0.0.0.0"
2. yes, if you change network.json, you will be able to start the same network with the new settings
   by running network start
3. if you run blockchain deploy again you will get a fresh new network, that will override the previous one

[felipemadero]

Pls feel free to ask any question to have on this. Happy to help

[ckz8780]

Thanks for these great responses! This is ultimately how I ended up figuring it out yesterday after much head scratching before I realized that you responded with the exact solutions I came up with 🤦‍♂️ 😆

In the end, the key thing that seems to have made it work is understanding that config.json in the ~/.avalanche-cli/ directory, which used to control the nodes via its node-config key, now doesn't work the same way, and in order to achieve the same I needed to set the config in ~/.avalanche-cli/subnets/<myblockchainname>/node-config.json

This is only partially explanatory though - since there seem to be some things that aren't supported in the same way they used to be. For example, if I understand it correctly setting http-port within this file is kind of pointless, since it's ignored for the L1 node and chosen randomly and the primary nodes all need different ports in order to work, so a single port setting doesn't really make sense. Either way though, like you said (which I did realize yesterday after tinkering for a while), the port/blockchain id/RPC url stay the same as long as you don't redeploy so that's helpful to know as it at least gives us some predictability once the blockchain is set up and deployed for the first time.

Ultimately I think documentation on the following would be really helpful:

• Example files for chain.json, node-config.json, per-node-chain.json, subnet.json, cluster-config.json, network.json, and config.json, if it can be used to combine all the former somehow
• Explanations of where those files should live, and what their available options are
• How those files interact with one another, which ones are copied into which directories upon deploying a blockchain, and which files they end up as (just as one example: it's clear now that setting http-allowed-hosts in node_config.json ultimately shows up in network.json both in the top level flags (which I'm guessing is the L1 node?) and in the nodeConfig level flags (which I think are the primary nodes?), but setting http-port only shows up at the nodeConfig flags level and is ignored at the outer level. It would be super helpful to have explanations of what each section means and how they all interconnect so that we can properly configure everything)

Simply documenting those three bullets in detail would GREATLY improve end users' ability to understand the system as a whole and their ability to properly configure it, and would probably lead to faster development of the CLI as a whole since it would become more clear what its capabilities already are.

Thank you again and please let me know if you have any more questions about my setup or I can help in any way.

EDIT: One other thing: is there an example blockchain aliases file (I think state.json?) somewhere? If we can't configure the RPC URL, it'd be handy to at least be able to set an alias and then if we have to recreate the network we can simply use the RPC server to get the id with info.getBlockchainID from /ext/info

[felipemadero]

Thanks for the detailed comments. Most of the documentation improvements are taken into account. We are
expecting to make some changes in the implementation of local networks, so probably the network.json configuration
file will change, and we will make documentation on those internals once the change is implemented.

Regarding blockchain aliases file, an example is:

{
  "q2aTwKuyzgs8pynF7UXBZCU7DejbZbZ6EUyHr3JQzYgwNPUPi": ["DFK"]
}

The expected location for such a file is

~/.avalanche-cli/local/blockchainName-local-node-local-network/nodeN/configs/chains/aliases.json

It is taken into account upon network restart.

Alternatively, you can also use an admin API call for that, that must be done after every restart:

https://docs.avax.network/api-reference/admin-api#adminaliaschain

Regarding http-port setting, it is not used at this moment, as you said. But we will provide some means for that
in the short term.

Regarding network.json structure, top level flags are applied to all l1 validator nodes, nodeConfig flags are applied to
each l1 validator separately. primary validators flags are not inside that file, but in ~/.avalanche-cli/snapshots/anr-snapshot-default-1654102509/network.json after a network stop. All of this is implementation dependent.

Agree the local network internals and file structure need improvement and documentation, for power users,
and we will work towards that.

Thanks again!