2.1.1

CODE_COLOR: CODE_YELLOW_MAINNET
RELEASE_VERSION: 2.1.1
PROTOCOL_UPGRADE: TRUE
DATABASE_UPGRADE: TRUE
SECURITY_UPGRADE: FALSE

Protocol Changes

The protocol upgrade introduces two features:

• Eth-Implicit Accounts NEP-0518
• Host Functions for BLS12-381 Curve Operations NEP-0488

Non-protocol changes

#11711 enforces the SHA-NI extensions. Without this, nodes are more likely to fall out of sync.
Make sure that the CPU has support for SHA-NI : lscpu | grep sha.

Protocol Upgrade Voting

Voting for upgrading to protocol version 70 will start on Monday 2024-08-26 10:00:00 AM UTC.

2.1.0-rc.3

CODE_COLOR: CODE_GREEN_TESTNET
RELEASE_VERSION: 2.1.0-rc.3
PROTOCOL_UPGRADE: FALSE
DATABASE_UPGRADE: FALSE
SECURITY_UPGRADE: FALSE

Fixes

This release addresses the graceful shutdown of the awc::Client.

Non-protocol changes

#11711 enforces the SHA-NI extentions. Whithout this, nodes are more likely to fall out of sync.
Make sure that the CPU has support for SHA-NI : lscpu | grep sha.

Full Changelog: 2.1.0-rc.2...2.1.0-rc.3

2.0.0

CODE_COLOR: CODE_YELLOW_MAINNET
RELEASE_VERSION: 2.0.0
PROTOCOL_UPGRADE: TRUE
DATABASE_UPGRADE: TRUE
SECURITY_UPGRADE: FALSE

Release note revision history

August 12th

The release note was revised on August 12th to update the "Failover node" section.
If you haven't reviewed it yet, please revisit the changes mentioned below for the latest updates.

• Failover node
  • Old failover procedure (Option 1)
    • Removed the "copy node_key.json" step.
  • Validator key hot swap (Option 2)
    • Added "Required Changes After Protocol Version Upgrade to 69".
    • Before protocol version upgrade to 69, the backup node's config.json should just match the primary node's config.json.

August 9th

The release note was revised on August 9th to include additional information. If you haven't reviewed it yet, please revisit the sections mentioned below for the latest updates.

• Config changes - Important notes
  • What recommended changes to config.json imply
  • Notes on memtries
  • State sync config
• Failover node

Protocol Upgrade Voting

This release contains two protocol upgrades. The first protocol upgrade introduces the congestion control feature, and the second upgrade introduces a set of protocol changes for stateless validation. There are therefore two relevant protocol version voting dates. This doesn't require any action on the part of node operators, as your node will follow the current protocol version's rules as long as you upgrade. Validator nodes will vote first for protocol version 68, and then for version 69 after 68 is adopted.

Voting for upgrading to protocol version 68 will start on Monday 2024-08-12 14:00:00 UTC and voting for upgrading to protocol version 69 will start on Wednesday 2024-08-14 04:00:00 UTC

Congestion Control

Relevant NEP: near/NEPs#539
RPC API change: #11419

Congestion control introduces a feature that's meant to limit the transactions and receipts included in chunks when a shard is under load. Before this protocol change, when a particular shard is under heavy load, users can often experience frustrating experiences where transactions take a long time to be included because there is a very long queue of delayed receipts that take up chunk space. When this delayed receipt queue grows to an unreasonable size, it can be difficult for users to understand when things will settle down and their transactions will begin to be included again.

The congestion control feature adds fields to the chunk header that indicate the load on the corresponding shard in terms of delayed and outgoing receipt gas, and the protocol rules limit transactions and receipts destined for congested shards, mitigating the problem of delayed receipt queues growing unbounded to unreasonable levels.

Things to note

Transactions can be rejected with a “ShardCongested” error if the chain is congested heavily

Stateless Validation

Relevant NEP: near/NEPs#509

This protocol upgrade is the main reason for designating this release as version 2.0.0. It will alter the roles of validators within the network and fundamentally change the process of validating state transitions. Currently when a node (validator or not) receives a new block and downloads all its chunks, it applies the transactions and receipts contained in those chunks by looking up and modifying values in the state of each shard. This means nodes need to keep the full state of all shards on disk. The stateless validation protocol upgrade will introduce a change that allows nodes to check the validity of chunks without needing to keep a local copy of the full state. There will be two roles after the upgrade: chunk producers and chunk validators. Chunk producers hold the state of the shard they are assigned to in memory and produce chunks for that shard. Chunk validators do not maintain any state locally and rotate on every single block to verify the state transition of a shard. A specific node may function both as a chunk producer and a chunk validator in the network. Whether a node is a chunk producer is based on stake: top 100 nodes by stake are chunk producers.

Changes to validator roles

Here is a high level summary of changes to validator roles

TL;DR;

• All validators
  • Serve as chunk validators
  • Additional network usage is expected
  • Less disk usage is expected
• Top 100 validators by stake:
  • In addition to chunk validator role, serve as chunk/block producers and block validators
  • Do not have to track all shards
  • State of tracked shard is loaded into memory
• All other validators
  • Don’t need to track any shard

Overview of different validator roles

• [NO CHANGES] Block producers (top 100 validators):
  • (Same as today) Produce blocks, (new) including waiting for chunk endorsements
  • (Same as today) Maintain chunk parts (i.e. participates in data availability based on Reed-Solomon erasure encoding)
  • (Same as today) Should have a high barrier of entry for security reasons, to make block double signing harder.
  • (New) No longer require tracking any shard
• Chunk producers (top 100 validators):
  • (Same as today) Produce chunks
  • (Same as today) Must track the shard it produces the chunk for
  • (New) Produces and distributes state witnesses to chunk validators
• [New] Chunk validators (all validators):
  • Validate state witnesses and sends chunk endorsements to block producers
  • Do not require tracking any shard
  • Must collectively have a majority of all the validator stake, to ensure the security of chunk validation.

Reward calculation

Reward calculation will remain the same and be based on online ratio.

Changes to contract requirements

• New size limits were introduced to limit the size of state witness:
  • Transaction size must not exceed 1.5MB (used to be 4MB).
  • Receipt size must not exceed 4MB (possibly it may be reduced further down to 1.5MB).
  • A receipt is not allowed to read more than 4MB of state during a single function call.
  • Total size of transactions included in two consecutive chunks is limited to 4MB
  • Cross shard bandwidth is more limited than before
• Increased cost of sending receipts
  • Sending a receipt to a shard (cross contract call) will now cost 50 TGas / MiB. Previously the cost varied between 2-18 TGas / MiB

Config changes

This section outlines the configuration changes that node operators must apply to their nodes.

Important notes (PLEASE READ BEFORE PROCEEDING)

What recommended changes to config.json imply

Please note that the existing behavior of the tracked_shards config option referred to below is a bit counter-intuitive. For historical reasons, there are only two meaningful kinds of values, empty and nonempty lists. Any non-empty list (e.g. [0] or [0,1,2,3]) will have the node track all shards. An empty list will indicate that we don't need to track all shards, which will be the case for validators after the network has reached protocol version 69. The store.load_mem_tries_for_tracked_shards config option referred to below is a boolean that indicates whether the full state should be loaded into memory. This should only be necessary for validators. RPC node operators should keep this option set to false. If you set this to true for an RPC node, it may fail to respond to certain RPC requests.

Notes on memtries

As noted in the previous section, there may be issues if you set store.load_mem_tries_for_tracked_shards to true for RPC nodes. Until these issues are fixed, RPC node operators should refrain from setting this config option, and validators should refrain from using their validator nodes as RPC nodes. Please note that if you set store.load_mem_tries_for_tracked_shards=true on your RPC node, it may reach an inconsistent state that will cause issues even after reverting back to store.load_mem_tries_for_tracked_shards=false.

State sync config

In a previous release, we discouraged enabling state sync, as there were issues with state sync being enabled during a resharding. For this release, those issues are no longer present, and validators should re-enable state sync by setting state_sync_enabled=true in the config, as well as setting the state_sync config option to an appropriate value. The value of the state_sync key in the current default config at https://s3-us-west-1.amazonaws.com/build.nearprotocol.com/nearcore-deploy/mainnet/config.json is the following:

{
  "sync": {
    "ExternalStorage": {
      "location": {
        "GCS": {
          "bucket": "state-parts"
        }
      },
      "num_concurrent_requests": 25,
      "num_concurrent_requests_during_catchup": 5
    }
  }
}

Estimated protocol upgrade timeline

• Current Protocol version: 67
• Voting on protocol version 68 upgrade: Monday, August 12, 2024, at 14:00 UTC
• Protocol version 68 upgrade: approximately Tuesday, August 13, 2024, at 17:00 UTC (2 epochs after voting)
• Voting on protocol version 69 upgrade: Wednesday, August 14, 2024, at 04:00 UTC
• Protocol version 69 upgrade: approximately Thursday, August 15, 2024, at 07:00 UTC (2 epochs after voting)

About transition period:
The transition period is the time between the initial voting on Monday, August 12, 2024, at 14:00 UTC and the upgrade to Protocol Version 69, which occurs 2 epochs after the second voting. Each epoch is approximately 13.5 hours, but may vary slightly. We anticipate the upgrade will happen around Thursday, August 15, 2024, at 07:00 UTC. To verify if the transition is complete, simply check the protocol version; it should read 69.

When to make configuration change

• After upgrading to neard binary 2.0.0: Make your co...

2.1.0-rc.2

CODE_COLOR: CODE_GREEN_TESTNET
RELEASE_VERSION: 2.1.0-rc.2
PROTOCOL_UPGRADE: FALSE
DATABASE_UPGRADE: FALSE
SECURITY_UPGRADE: FALSE

Fixes

This release addresses the RPC endpoint issue where all handler errors are misclassified as 500.

Non-protocol changes

#11711 enforces the SHA-NI extentions. Whithout this, nodes are more likely to fall out of sync.
Make sure that the CPU has support for SHA-NI : lscpu | grep sha.

Full Changelog: 2.1.0-rc.1...2.1.0-rc.2

2.1.0-rc.1

CODE_COLOR: CODE_YELLOW_TESTNET
RELEASE_VERSION: 2.1.0-rc.1
PROTOCOL_UPGRADE: TRUE
DATABASE_UPGRADE: TRUE
SECURITY_UPGRADE: FALSE

Protocol Changes

The protocol upgrade introduces two features:

• Eth-Implicit Accounts NEP-0518
• Host Functions for BLS12-381 Curve Operations NEP-0488

Non-protocol changes

#11711 enforces the SHA-NI extentions. Whithout this, nodes are more likely to fall out of sync.
Make sure that the CPU has support for SHA-NI : lscpu | grep sha.

Protocol Upgrade Voting

Voting for upgrading to protocol version 70 will start on Tuesday 2024-08-06 10:00:00 UTC.

2.0.0-rc.5

CODE_COLOR: CODE_GREEN_TESTNET
RELEASE_VERSION: 2.0.0-rc.5
PROTOCOL_UPGRADE: FALSE
DATABASE_UPGRADE: FALSE
SECURITY_UPGRADE: FALSE

Fixes:

Enforce tracked_shards config option only for current validators, not just any node with a validator key.

2.0.0-rc.3

CODE_COLOR: CODE_GREEN_TESTNET
RELEASE_VERSION: 2.0.0-rc.3
PROTOCOL_UPGRADE: FALSE
DATABASE_UPGRADE: FALSE
SECURITY_UPGRADE: FALSE

Fixes:

Enforce tracked_shards config option only for validators. Before the stateless validation protocol version, we enforce non-empty tracked_shards. After we enter the first epoch with stateless validation we enforce empty tracked_shards.

2.0.0-rc.2

CODE_COLOR: CODE_YELLOW_TESTNET
RELEASE_VERSION: 2.0.0-rc.2
PROTOCOL_UPGRADE: FALSE
DATABASE_UPGRADE: FALSE
SECURITY_UPGRADE: FALSE

Fixes:

Fix an issue where setting empty tracked shards in config.json causes nodes to fail to start

2.0.0-rc.1

CODE_COLOR: CODE_YELLOW_TESTNET
RELEASE_VERSION: 2.0.0
PROTOCOL_UPGRADE: TRUE
DATABASE_UPGRADE: TRUE
SECURITY_UPGRADE: FALSE

Protocol Upgrade Voting

This release contains two protocol upgrades. The first protocol upgrade introduces the congestion control feature, and the second upgrade introduces a set of protocol changes for stateless validation. There are therefore two relevant protocol version voting dates. This doesn't require any action on the part of node operators, as your node will follow the current protocol version's rules as long as you upgrade. Validator nodes will vote first for protocol version 68, and then for version 69 after 68 is adopted.

Voting for upgrading to protocol version 68 will start on Tuesday 2024-07-09 17:00:00 UTC and voting for upgrading to protocol version 69 will start on Wednesday 2024-07-10 17:00:00 UTC

Congestion Control

Relevant NEP: near/NEPs#539
RPC API change: #11419

Congestion control introduces a feature that's meant to limit the transactions and receipts included in chunks when a shard is under load. Before this protocol change, when a particular shard is under heavy load, users can often experience frustrating experiences where transactions take a long time to be included because there is a very long queue of delayed receipts that take up chunk space. When this delayed receipt queue grows to an unreasonable size, it can be difficult for users to understand when things will settle down and their transactions will begin to be included again.

The congestion control feature adds fields to the chunk header that indicate the load on the corresponding shard in terms of delayed and outgoing receipt gas, and the protocol rules limit transactions and receipts destined for congested shards, mitigating the problem of delayed receipt queues growing unbounded to unreasonable levels.

Things to note

Transactions can be rejected with a “ShardCongested” error if the chain is congested heavily

Stateless Validation

This protocol upgrade is the main reason for designating this release as version 2.0.0. It will alter the roles of validators within the network and fundamentally change the process of validating state transitions. Currently when a node (validator or not) receives a new block and downloads all its chunks, it applies the transactions and receipts contained in those chunks by looking up and modifying values in the state of each shard. This means nodes need to keep the full state of all shards on disk. The stateless validation protocol upgrade will introduce a change that allows nodes to check the validity of chunks without needing to keep a local copy of the full state. There will be two roles after the upgrade: chunk producers and chunk validators. Chunk producers hold the state of the shard they are assigned to in memory and produce chunks for that shard. Chunk validators do not maintain any state locally and rotate on every single block to verify the state transition of a shard. A specific node may function both as a chunk producer and a chunk validator in the network. Whether a node is a chunk producer is based on stake: top 100 nodes by stake are chunk producers.

Changes to validator roles

Here is a high level summary of changes to validator roles

TL;DR;

• All validators
  • Serve as chunk validators
  • Additional network usage is expected
  • Less disk usage is expected
• Top 100 validators by stake:
  • In addition to chunk validator role, serve as chunk/block producers and block validators
  • Do not have to track all shards
  • State of tracked shard is loaded into memory
• All other validators
  • Don’t need to track any shard

Overview of different validator roles

• [NO CHANGES] Block producers (top 100 validators):
  • (Same as today) Produce blocks, (new) including waiting for chunk endorsements
  • (Same as today) Maintain chunk parts (i.e. participates in data availability based on Reed-Solomon erasure encoding)
  • (Same as today) Should have a high barrier of entry for security reasons, to make block double signing harder.
  • (New) No longer require tracking any shard
• Chunk producers (top 100 validators):
  • (Same as today) Produce chunks
  • (Same as today) Must track the shard it produces the chunk for
  • (New) Produces and distributes state witnesses to chunk validators
• [New] Chunk validators (all validators):
  • Validate state witnesses and sends chunk endorsements to block producers
  • Do not require tracking any shard
  • Must collectively have a majority of all the validator stake, to ensure the security of chunk validation.

Reward calculation

Reward calculation will remain the same and be based on online ratio.

Changes to contract requirements

• New size limits were introduced to limit the size of state witness:
  • Transaction size must not exceed 1.5MB (used to be 4MB).
  • Receipt size must not exceed 4MB (possibly it may be reduced further down to 1.5MB).
  • A receipt is not allowed to read more than 4MB of state during a single function call.
  • Total size of transactions included in two consecutive chunks is limited to 4MB
  • Cross shard bandwidth is more limited than before
• Increased cost of sending receipts
  • Sending a receipt to a shard (cross contract call) will now cost 50 TGas / MiB. Previously the cost varied between 2-18 TGas / MiB

Config changes

This section describes the configuration change a node operator needs to apply to their node before and after the stateless validation launch.
Let’s say that the current mainnet protocol version is X (before stateless validation).

When Top 100 validators, adopt the stateless validation release, they need to:

• Change their nodes to meet hardware requirements (can be found below)
• Mainnet: Set store.load_mem_tries_for_tracked_shards in config.json to true. Testnet: Set store.load_mem_tries_for_tracked_shards in config.json to true only after protocol version 68 is adopted by the network. otherwise nodes will crash attempting to load tries to memory
• Set tracked_shards in config.json to [0] (This config will be updated after the protocol upgrade, but it should NOT be changed at this point.)
• Recommendation: apply network optimization below

When Non-validator node operators, adopt the stateless validation release, they need to:

• Change their nodes to meet hardware requirements (can be found below)
• Set store.load_mem_tries_for_tracked_shards in config.json to false

The stateless validation release has two protocol upgrades, congestion control (protocol version 68), and stateless validation (protocol version 69). After these two protocol upgrades are done,

Top 100 validators must:

• Downgrade their node to meet hardware requirements (can be found below)
• Set tracked_shards in config.json to []
• Recommendation: apply network optimization below

This means that validators only need machines with a lot of memory during the upgrade process.

All other validators must:

• Set store.load_mem_tries_for_tracked_shards in config.json to true
• Set tracked_shards in config.json to []
• Recommendation: apply network optimization below

Non-validator node operators do NOT have any further action needed.

Network optimization

sudo sysctl -w net.core.rmem_max=8388608 && sudo sysctl -w net.core.wmem_max=8388608 && sudo sysctl -w net.ipv4.tcp_rmem='4096 87380 8388608' && sudo sysctl -w net.ipv4.tcp_wmem='4096 16384 8388608' && sudo sysctl -w net.ipv4.tcp_slow_start_after_idle=0

Validator key hot swap

There is an option to change a node or its configuration with near-zero downtime.
Let’s assume we created a new node with desired configuration and we want to move the validator key from the current validator to the new node.

If this is before stateless validation launch, then the new node is configured with:

• set tracked_shards in config.json to [0]

and we have to wait until the new node finishes state sync for all shards.

If this is after stateless validation launch, the new node is configured with:

• set tracked_shards in config.json to []
  then we should also:
• set tracked_shadow_validator in config.json to “<validator_id>”
  where <validator_id> is the validator account ID of the current validator, and wait until the new node finishes state sync for the shard assigned to the current validator.

Steps to move validator key from current validator to the new node:

1. Copy .near/validator_key.json to the new node.
2. Stop the current validator.
3. Send a SIGHUP signal to the new node.

The new node will pick up the validator key without restart and it will start validating.

Hardware requirements

Validators:

• RAM:
  • During transition time:
    • MainNet: 80+GB
    • TestNet: 40+GB
  • After transition:
    • MainNet: 32+GB
    • TestNet: 32+GB
• Network bandwidth:
  • Recommended burst capacity is 1 Gbps up/down.
  • Sustained usage rate will be below 100 Mbps up/down.
• Backup nodes
  • Hot-switch to validator-key
  • Track all shards (to be configured in config.json)
  • Before and after transition: Mainnet: 80+GB, Testnet: 40+ GB

RPC nodes

• RAM:
  • If in-mem trie is enabled:
    • MainNet: 80+GB
    • TestNet: 40+GB
  • If in-mem trie is NOT enabled:
    • MainNet and TestNet 24+GB
• No changes to disk size and network bandwidth requirements

Archival nodes

• RAM:
  • If in-mem trie is enabled:
    • MainNet: 80+GB
    • TestNet: 40+GB
  • If in-mem trie is NOT enabled:
    • MainNet and TestNet 24+GB
• No changes to disk size and network bandwidth requirements

1.40.0

CODE_COLOR: CODE_YELLOW_MAINNET
RELEASE_VERSION: 1.40.0
PROTOCOL_UPGRADE: TRUE
DATABASE_UPGRADE: TRUE
SECURITY_UPGRADE: FALSE

Protocol Upgrade Voting

This release contains a protocol upgrade. Voting for upgrading to protocol version 67 will start on Monday 2024-06-24 12:00:00 UTC.

Protocol Changes

Yield/resume NEP is stabilised in protocol version 67. This introduces two new host functions promise_yield_create and promise_yield_resume.

Fixes:

• routing: finish deprecation of old-style nonces #11321
• prefetch: remove code for prefetching claim.sweat::claim #11329
• sweat: update prefetcher for record_batch_for_hold #11338
• routing: check edge nonce before signature #11396