Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flow EVM Gateway doc changes #1282

Open
wants to merge 54 commits into
base: main
Choose a base branch
from
Open
Changes from 12 commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
096a925
Update flow.mdx
franklywatson Sep 19, 2024
a1e0901
WIP
franklywatson Sep 20, 2024
202630f
WIP
franklywatson Sep 20, 2024
3644f6d
WIP
franklywatson Sep 20, 2024
73e4e7f
WIP
franklywatson Sep 20, 2024
6f4ae5a
WIP
franklywatson Sep 20, 2024
8da6808
WIP
franklywatson Sep 20, 2024
688fd1f
WIP
franklywatson Sep 20, 2024
c4ca90e
Add Flow graphics
franklywatson Sep 20, 2024
1546daf
Add Flow guidance stake pending graphic
franklywatson Sep 20, 2024
fc5fe2d
Revise account creation flow
franklywatson Sep 21, 2024
fd6e077
Further fleshing out
franklywatson Sep 22, 2024
777379b
Last updates to main guidance
franklywatson Sep 22, 2024
bd81195
Added troubleshooting guidance
franklywatson Sep 22, 2024
f492651
Minor tidyup
franklywatson Sep 22, 2024
89920ea
Merge upstream
franklywatson Sep 22, 2024
e75b55b
Revert unwanted change
franklywatson Sep 22, 2024
6ab539e
Make naming consistent
franklywatson Sep 22, 2024
9cf845b
Fix indents
franklywatson Sep 22, 2024
f8a4f61
Minor edit
franklywatson Sep 22, 2024
91be36e
Minor edit
franklywatson Sep 22, 2024
bf01579
removing 'earn increased Flow fees'
vishalchangrani Sep 22, 2024
0b2c46a
ignoring IDE files
vishalchangrani Sep 22, 2024
8ba107f
removing the IDE files
vishalchangrani Sep 22, 2024
a58d5ba
updating the access node setup
vishalchangrani Sep 22, 2024
bd296d3
adding instructions to download the execution state
vishalchangrani Sep 23, 2024
54f00d1
adding a line about ingress rule
vishalchangrani Sep 23, 2024
9c50322
removing dynamic bootstrap flags and adding instructinos to pull down…
vishalchangrani Sep 23, 2024
6e2a600
adding mainnet and testnet tabs for transit script
vishalchangrani Sep 23, 2024
f8fea0a
fixing typo
vishalchangrani Sep 23, 2024
c489c61
callout for network upgrade
vishalchangrani Sep 23, 2024
1d59439
adding reference to execution data download
vishalchangrani Sep 23, 2024
00f835c
Merge pull request #2 from onflow/vishal/flow-node-bootstrap-update
vishalchangrani Sep 23, 2024
c2d807f
Merge branch 'main' into flow-node-bootstrap-update
vishalchangrani Sep 23, 2024
cef5b2f
Adjusting tab items
vishalchangrani Sep 23, 2024
aa439aa
Adjusting tab items
vishalchangrani Sep 24, 2024
7a5c61f
Adjusting tab items
vishalchangrani Sep 24, 2024
d8107cb
updating testnet instructions
vishalchangrani Sep 24, 2024
1561a98
Last tidyups
franklywatson Sep 24, 2024
ab3b211
Minor edits to fix errors and formatting
franklywatson Sep 24, 2024
e7bbf67
Add EVm block explorer
franklywatson Sep 24, 2024
04bf326
Tidy wording for EVM block explorer
franklywatson Sep 24, 2024
ab731b1
Small edit to clarify CLI usage
franklywatson Sep 24, 2024
50fc468
Minor edit
franklywatson Sep 24, 2024
da3fcce
chanding role to execution for transit script
vishalchangrani Sep 24, 2024
aca14c5
Specify correct account
franklywatson Sep 24, 2024
8cd9fb1
Merge branch 'main' into flow-node-bootstrap-update
franklywatson Oct 9, 2024
32e78d9
Merge pull request #3 from onflow/flow-node-bootstrap-update
franklywatson Oct 9, 2024
85eac43
Restore lost example
franklywatson Oct 9, 2024
237b1f0
Fixing links
franklywatson Oct 9, 2024
bb0b0dc
Merge branch 'main' into main
franklywatson Oct 9, 2024
3792102
Merge branch 'axelarnetwork:main' into main
franklywatson Jan 6, 2025
b1f05e7
Minor corrections and updates
franklywatson Jan 6, 2025
c6049a2
Merge branch 'axelarnetwork:main' into main
franklywatson Jan 9, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -19,3 +19,6 @@ pnpm-debug.log*

# macOS-specific files
.DS_Store

# goland ide
.idea/
163 changes: 113 additions & 50 deletions src/content/docs/validator/external-chains/flow.mdx
Original file line number Diff line number Diff line change
@@ -7,14 +7,14 @@ Node setup guide for Flow Testnet/Mainnet Axelar validator
## EVM on Flow equivalence

EVM on Flow integrates Ethereum's transaction execution core into Cadence's native transaction runtime and thus Flow's multi
node consensus. Access Nodes serve the Access API, the native entry point to Flow, accepting Cadence scripts or
node consensus. Access Nodes (AN) serve the Access API, the native entry point to Flow, accepting Cadence scripts or
transactions, into which signed transactions from the EVM Gateway are included. The EVM Gateway implements the Ethereum
JSON-RPC specification for web3.js clients. In the Flow context these two services together represent an Axelar validator.
Access Nodes, and by extension EVM Gateways, do not vote on the network - they are rpc-only node types.

Operators can choose to run the Access Node and EVM Gateway separately or on the same logical host. Consider that
Access Nodes are optimized for high throughput script/transaction ingress for Flow clients. Node operators which choose
to make AN Access APIs public earn increased Flow fees, for which recommended hardware requirements apply. If the validator's
to make AN Access APIs public should use the recommended hardware requirements. If the validator's
sole purpose is to participate in the Axelar network, recommended hardware requirements may be excess to operational needs.
Operators may opt to run only EVM Gateway if using [public](https://www.flowdiver.io/node?sortField=node.organization) Access Nodes.

@@ -29,20 +29,29 @@ Operators may opt to run only EVM Gateway if using [public](https://www.flowdive

## Steps

1. Generate node information
The steps first walk through the process of setting up an access node and then setting up the EVM gateway node.

### Access node setup
1. Generate access node information
2. Account creation
3. Stake your node
4. Get binaries
5. Create services
6. Build EVM Gateway
7. Start all services
3. Stake the access node
4. Build access node binary
5. Download access node bootstrap files
6. Create access node service

### EVM Gateway setup
7. Build EVM Gateway binary
8. Create EVM Gateway service

9. Start all services

We recommend first time operators to complete the guidance for node setup below in full using testnet.

## Generate Node Information
## 1. Generate Node Information

### Download the boot-tools utility
```bash
# get the binary
# get the boot-tools utility to generate node information
curl -sL -O storage.googleapis.com/flow-genesis-bootstrap/boot-tools.tar
# untar it
tar -xvf boot-tools.tar
@@ -54,7 +63,13 @@ mkdir ./bootstrap
./boot-tools/bootstrap key --address "<YOUR_NODE_ADDRESS_GOES_HERE>:3569" --role access -o ./bootstrap
```

The generated machine account bootstrap files include public and private machine account keys, staking key and the node id associated with your node address.
<Callout type="info">
The network address is the address that will be used by the other nodes in the network to talk to your node.

Please ensure that the address (hostname:port) that you use is accessible from outside and there is no firewall rule blocking egress and **ingress** traffic for that address.
</Callout>

This will generate the node information which includes public and private staking keys, public and private networking key and the node ID.

```bash
cat ./bootstrap/public-root-information/node-info.pub.[HASH].json
@@ -79,11 +94,11 @@ cat ./bootstrap/private-root-information/private-node-info_[HASH]/node-info.priv
"StakingPrivKey": "061d9bef99db63629ed28bb64ad0cc6b93418accc1a6c5e461dfcbdf087c2862"
}
```
Please secure/take a backup of the entire machine account bootstrap folder.
Please ensure you backup the bootstrap folder as it contains the node private keys. If they keys are lost, the node will not be able to run.

For further information on this step see [official documentation](https://developers.flow.com/networks/node-ops/access-onchain-data/access-nodes/access-node-setup#step--1---generate-node-information).

## Account Creation
## 2. Account Creation

Many operators will only require the first two steps below if aggregating operational costs across Flow into a single account is acceptable, or where only a
single logical host is being run for all services. Organizations who must separate operating costs between node types must include step (3) to create a
@@ -92,7 +107,7 @@ distinct account for the EVM Gateway.
If you are only running the EVM Gateway and using public Access Nodes the first two account creation steps below are still required. In this case the account
will only be used for the EVM Gateway.

1. Create Access Node operator account
### 1. Create Access Node operator account

Accounts need to be created on Flow before they can be funded and used. Visit [Flow Port](https://port.flow.com) and
select `mainnet` or `testnet` in the dropdown and click 'Sign Up'. Flow Port will guide you through Flow Wallet download,
@@ -101,7 +116,7 @@ to Flow Port and can copy your account address from the wallet or Flow Port dash

Note: to enable Flow testnet network in Flow Wallet you will need to toggle <pre>Settings -> Developer Mode -> Enabled</pre>

2. Fund Access Node operator account
### 2. Fund Access Node operator account

<tabs>

@@ -147,7 +162,7 @@ Index 0
Contracts Deployed: 0
```

3. Create EVM Gateway operator account \[Optional\]
### 3. Create EVM Gateway operator account \[Optional\]

Use the existing operator account to create new accounts.

@@ -212,13 +227,13 @@ Create EVM Gateway operator account

</tabs>

## Stake your node
## 3. Stake the access node

Access Nodes require a minimum 100 FLOW stake for each participating node to secure a slot on the network. Follow this
[guide](https://developers.flow.com/networks/node-ops/access-onchain-data/access-nodes/access-node-setup#step--2---stake-the-node) to stake using [Flow Port](https://port.flow.com).

<Callout type="info">
Use the previously generated public key values generated by the bootstrap when staking your node
Use the public key values previously generated by the bootstrap utility when staking your node

When staking for the first time please select 'Upgrade Account' in Flow Port when requested
</Callout>
@@ -230,7 +245,7 @@ status. Note the banner which outlines when your staking action will be included

Further information about staking on Flow can be found in the [Staking and Epochs](https://developers.flow.com/networks/staking#how-does-staking-work-on-flow) documentation.

## Verify That Your Node ID Was Selected
### Verify That Your Node ID Was Selected

On Wednesday at around 12:00 UTC, the staking auction for the current epoch will end and five nodes from candidate list of nodes will be chosen at random by the staking contract for participation in the next epoch.
If all 5 slots have been taken from the previous epoch no new Access Nodes will be chosen
@@ -264,17 +279,78 @@ Look for the 'Tokens Staked' field in the response of the above command to verif
In the event that the epoch passes and the node you registered did not get picked your stake will be returned. You can re-stake the
node using the same staking procedure as above.

## Get binaries
## 4. Build access node binary

```bash
curl -sL -O storage.googleapis.com/flow-genesis-bootstrap/binary/access/app
chmod +x app
mv app /usr/bin/access-node
Build the access node binary from the latest release tag. The access node source is available in the [onflow/flow-go](https://github.com/onflow/flow-go) repo.

1. Install the prerequisites mentioned [here](https://github.com/onflow/flow-go?tab=readme-ov-file#installation)
2. Checkout the latest release tag mentioned [here](https://github.com/onflow/flow-go/releases).
3. Build the access node binary by following this [step](https://github.com/onflow/flow-go?tab=readme-ov-file#building-a-binary-for-the-access-node) in the flow-go repo.

Alternatively, you can also run the access node and the EVM gateway as a docker container. See [here](https://developers.flow.com/networks/node-ops/access-onchain-data/access-nodes/access-node-setup#step-4---start-your-node) for more details.

## 5. Download access node bootstrap files

After the access node has been successfully staked, it requires two pieces of information to startup.
1. An initialization file called the `root-protocol-state-snapshot.json` to initialize its local database.
2. A checkpoint of the execution state which contains the chain state (account balances, contracts etc.). The checkpoint consists of 18 files named as `root.checkpoint`, `root.checkpoint.000`, `root.checkpoint.001`...`root.checkpoint.016`.

The `root-protocol-state-snapshot.json` and the checkpoint files are published by the Foundation to a google bucket.
Location of the bucket is available [here](https://github.com/onflow/flow/blob/master/sporks.json) under `rootProtocolStateSnapshot` and `rootCheckpointFile` for each of the past network upgrades.
For e.g. For mainnet25 the files are [here](https://github.com/onflow/flow/blob/master/sporks.json#L15-L16).

The `transit` script which is part of the `boot-tools` utility downloaded earlier during [first step](#download_the_boot_tools_utility) can be used to download these files and place them in the right location.

To download the files run when testnet/mainnet-x should be set to the latest version of testnet or mainnet, e.g. testnet-52, mainnet-26 etc.
```
./boot-tools/transit pull -b ./bootstrap -t [testnet/mainnett-x] -r access --concurrency 10 --timeout 30m
```

<tabs>

<tab-item title="Mainnet">

## Create services
```bash
./boot-tools/transit pull -b ./bootstrap -t mainnet-x -r access --concurrency 10 --timeout 30m
```

Use `systemctl` to set up services for `access-node`, `gateway`
Example for network upgrade mainnet-26,

```
./boot-tools/transit pull -b ./bootstrap -t mainnet-26-execution -r access --concurrency 10 --timeout 30m
```

</tab-item>

<tab-item title="Testnet">

```bash
./boot-tools/transit pull -b ./bootstrap -t testnet-x -r access --concurrency 10 --timeout 30m
```

Example for network upgrade testnet-52,

```
./boot-tools/transit pull -b ./bootstrap -t testnet-52-execution -r access --concurrency 10 --timeout 30m
```

</tab-item>

</tabs>

For more information on the steps to carry out after a network upgrade, please see [here](https://developers.flow.com/networks/node-ops/node-operation/spork).

<Callout type="info">
If the node was already running on the previous network before the upgrade then please follow [these](https://developers.flow.com/networks/node-ops/node-operation/spork) steps to sync the node on the upgraded network.
</Callout>

<Callout type="info">
If you are joining the network on any epoch after the last network upgrade, then please see [here](https://developers.flow.com/networks/node-ops/access-onchain-data/access-nodes/access-node-configuration-options#option-2-enabling-indexing-mid-spork) for instructions on downloading the bootstrap files.
</Callout>

## 6. Create access node service

Use `systemctl` to set up services for `access-node`.

### Access Node

@@ -318,6 +394,7 @@ ExecStart=/usr/bin/access-node \
--datadir=$PWD/data/protocol \
--secretsdir=$PWD/data/secrets \
--execution-data-dir=$PWD/data/execution_data \
--execution-state-checkpoint=$PWD/bootstrap/execution-state \
--rpc-addr=0.0.0.0:9000 \
--secure-rpc-addr=0.0.0.0:9001 \
--http-addr=0.0.0.0:8000 \
@@ -326,9 +403,6 @@ ExecStart=/usr/bin/access-node \
--bind=0.0.0.0:3569 \
--dht-enabled=false \
--grpc-compressor=gzip \
--dynamic-startup-access-address=$DYNAMIC_ACCESS_ADDRESS \
--dynamic-startup-access-publickey=$DYNAMIC_ACCESS_PUBLICKEY \
--dynamic-startup-epoch-phase=EpochPhaseStaking \
--execution-state-dir=<path-to-execution-state-dir> \
--script-execution-mode=failover \
--event-query-mode=failover \
@@ -350,22 +424,6 @@ cat /etc/systemd/system/access-node.service
sudo systemctl enable access-node
```

### Syncing full or partial spork history

Depending on the operator use case Access Node can sync the full history of the current spork or they can simply start indexing from any other arbitrary block you configure. For Axelar validators
it is not anticipated that full historu indexing is required. In this case it can use the [dynamic bootstrap](https://developers.flow.com/networks/node-ops/node-operation/protocol-state-bootstrap#using-dynamic-startup)
method to start serving data from the current block. Instructions on dynamic bootstrap can be found [here](https://developers.flow.com/networks/node-ops/node-operation/protocol-state-bootstrap).

[TODO] Root Checkpoint guidance

If block history from the beginning of the spork is required you will need to configure above

```bash
--execution-data-indexing-enabled=true \
```

### Other configuration flags

### Troubleshooting

The following Access Node log message indicates that the node has not yet been selected for participation in the network. If your node is staked it will hopefully be selected at the next epoch.
@@ -382,9 +440,12 @@ curl localhost:8080/metrics | grep consensus_compliance_sealed_height

Further details on Access Node setup can be found in the [Access Node setup](https://developers.flow.com/networks/node-ops/access-onchain-data/access-nodes/access-node-setup#how-to-run-a-permissionless-access-node)) documentation.

## Build EVM Gateway
## 7. Build EVM Gateway binary

Similar to the access node, build the EVM gateway binary from the latest release tag. The EVM gateway source is available in the [onflow/flow-go](https://github.com/onflow/flow-evm-gateway) repo.
```bash
git clone https://github.com/onflow/flow-evm-gateway.git
git checkout <latest release tag mentioned here https://github.com/onflow/flow-evm-gateway/releases>
cd flow-evm-gateway
go build -o evm-gateway cmd/main/main.go
mv evm-gateway /usr/bin/
@@ -452,6 +513,7 @@ Flow operator address and private key is configured for `--coa-address` & `--coa

</tabs>

## 8. Create EVM Gateway service

```bash
sudo tee <<EOF >/dev/null /etc/systemd/system/gateway.service
@@ -511,12 +573,13 @@ The following log entry will occur when the EVM Gateway attempts to sync with th
failure in event subscription at height ${INIT-CADENCE-HEIGHT}, with: recoverable: disconnected: error receiving event: rpc error: code = FailedPrecondition desc = could not get start height: failed to get lowest indexed height: index not initialized
```

## Start all services
## 9. Start all services

Order of operations:

1. `access-node`: ensure it's fully synced before proceeding
3. `gateway`
1. `access-node`: ensure it's fully synced in terms of block height before proceeding.
- The latest chain height can be queried through flow cli (`flow blocks get latest - mainnet`) and also available on the [explorer](https://www.flowscan.io/).
2. `gateway`

```bash
sudo systemctl daemon-reload
@@ -533,7 +596,7 @@ For gateway
curl -s -XPOST 'localhost:8545' --header 'Content-Type: application/json' --data-raw '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' | jq
```

## Check logs
### Check logs

```bash
# change log settings to persistent if not already