Add EIP: Forward compatible consensus data structures

EIPS/eip-7688.md

@@ -0,0 +1,296 @@
+---
+eip: 7688
+title: Forward compatible consensus data structures
+description: Transition consensus SSZ data structures to StableContainer
+author: Etan Kissling (@etan-status), Cayman (@wemeetagain)
+discussions-to: https://ethereum-magicians.org/t/eip-7688-forward-compatible-consensus-data-structures/19673
+status: Draft
+type: Standards Track
+category: Core
+created: 2024-04-15
+requires: 7495
+---
+
+## Abstract
+
+This EIP defines the changes needed to adopt `StableContainer` from [EIP-7495](./eip-7495.md) in consensus data structures.
+
+## Motivation
+
+Ethereum's consensus data structures make heavy use of [Simple Serialize (SSZ)](https://github.com/ethereum/consensus-specs/blob/ef434e87165e9a4c82a99f54ffd4974ae113f732/ssz/simple-serialize.md) `Container`, which defines how they are serialized and merkleized. The merkleization scheme allows application implementations to verify that individual fields (and partial fields) have not been tampered with. This is useful, for example, in smart contracts of decentralized staking pools that wish to verify that participating validators have not been slashed.
+
+While SSZ `Container` defines how data structures are merkleized, the merkleization is prone to change across the different forks. When that happens, e.g., because new features are added or old features get removed, existing verifier implementations need to be updated to be able to continue processing proofs.
+
+`StableContainer`, of [EIP-7495](./eip-7495.md), is a forward compatible alternative that guarantees a forward compatible merkleization scheme. By transitioning consensus data structures to use `StableContainer`, smart contracts that contain verifier logic no longer have to be maintained in lockstep with Ethereum's fork schedule as long as the underlying features that they verify don't change. For example, as long as the concept of slashing is represented using the boolean `slashed` field, existing verifiers will not break when unrelated features get added or removed. This is also true for off-chain verifiers, e.g., in hardware wallets or in operating systems for mobile devices that are on a different software update cadence than Ethereum.
+
+## Specification
+
+The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 and RFC 8174.
+
+### Conversion procedure
+
+For each converted data structure, a new `StableContainer` type `S` is introduced that serves as the primary definition of each data structure.
+
+- Each `StableContainer` is assigned a capacity to represent its potential design space that SHALL NOT change across future forks; if it is later determined that it is insufficient, a new field can be added to contain additional fields in a sub-container.
+- The `StableContainer` starts as a copy of the latest fork's `Container` equivalent.
+- To guarantee forward and backward compatibility, new fields from future forks MUST be appended to the end of the `StableContainer` definition. Existing fields MAY be converted to `Optional`.
+
+Furthermore, a `Variant` type is defined that is specific to the fork at which the conversion is applied. This `Variant` is the equivalent of the legacy `Container` type, except that it inherits from `Variant[S]`. The SSZ serialization of `Variant` is compatible with `Container`, but the merkleization and `hash_tree_root` are computed differently. Furthermore, `Variant` MAY use fields of `Optional` type if necessary.
+
+If subsequent forks add / remove fields, they specify a new `Variant`. This is similar to the previous approach of specifying a new `Container`. The `StableContainer` is the superset across all `Variant`; fields not present across all `Variant` use an `Optional` type.
+
+The fork specific `Variant` definitions use the same serialization as the legacy `Container` type. Only merkleization and `hash_tree_root` are computed differently when switching to the new scheme.
+
+### Limits
+
+| Name | Value | Description |
+| - | - | - |
+| `MAX_EXECUTION_PAYLOAD_FIELDS` | `uint64(2**6)` (= 64) | Maximum number of fields to which `ExecutionPayload` can ever grow in the future |
+| `MAX_BEACON_BLOCK_BODY_FIELDS` | `uint64(2**6)` (= 64) | Maximum number of fields to which `BeaconBlockBody` can ever grow in the future |
+| `MAX_BEACON_STATE_FIELDS` | `uint64(2**7)` (= 128) | Maximum number of fields to which `BeaconState` can ever grow in the future |
+
+### Fork-agnostic `StableContainer` definitions
+
+These type definitions are fork independent and shared across all forks. They are not exchanged over libp2p.
+
+```python
+class StableExecutionPayload(StableContainer[MAX_EXECUTION_PAYLOAD_FIELDS]):
+    # Execution block header fields
+    parent_hash: Hash32
+    fee_recipient: ExecutionAddress  # 'beneficiary' in the yellow paper
+    state_root: Bytes32
+    receipts_root: Bytes32
+    logs_bloom: ByteVector[BYTES_PER_LOGS_BLOOM]
+    prev_randao: Bytes32  # 'difficulty' in the yellow paper
+    block_number: uint64  # 'number' in the yellow paper
+    gas_limit: uint64
+    gas_used: uint64
+    timestamp: uint64
+    extra_data: ByteList[MAX_EXTRA_DATA_BYTES]
+    base_fee_per_gas: uint256
+    # Extra payload fields
+    block_hash: Hash32  # Hash of execution block
+    transactions: List[Transaction, MAX_TRANSACTIONS_PER_PAYLOAD]
+    withdrawals: List[Withdrawal, MAX_WITHDRAWALS_PER_PAYLOAD]
+    blob_gas_used: uint64
+    excess_blob_gas: uint64
+    ...  # Additional features from the fork that introduces this EIP
+
+class StableExecutionPayloadHeader(StableContainer[MAX_EXECUTION_PAYLOAD_FIELDS]):
+    # Execution block header fields
+    parent_hash: Hash32
+    fee_recipient: ExecutionAddress
+    state_root: Bytes32
+    receipts_root: Bytes32
+    logs_bloom: ByteVector[BYTES_PER_LOGS_BLOOM]
+    prev_randao: Bytes32
+    block_number: uint64
+    gas_limit: uint64
+    gas_used: uint64
+    timestamp: uint64
+    extra_data: ByteList[MAX_EXTRA_DATA_BYTES]
+    base_fee_per_gas: uint256
+    # Extra payload fields
+    block_hash: Hash32  # Hash of execution block
+    transactions_root: Root
+    withdrawals_root: Root
+    blob_gas_used: uint64
+    excess_blob_gas: uint64
+    ... # Additional features from the fork that introduces this EIP
+
+class StableBeaconBlockBody(StableContainer[MAX_BEACON_BLOCK_BODY_FIELDS]):
+    randao_reveal: BLSSignature
+    eth1_data: Eth1Data  # Eth1 data vote
+    graffiti: Bytes32  # Arbitrary data
+    # Operations
+    proposer_slashings: List[ProposerSlashing, MAX_PROPOSER_SLASHINGS]
+    attester_slashings: List[AttesterSlashing, MAX_ATTESTER_SLASHINGS]
+    attestations: List[Attestation, MAX_ATTESTATIONS]
+    deposits: List[Deposit, MAX_DEPOSITS]
+    voluntary_exits: List[SignedVoluntaryExit, MAX_VOLUNTARY_EXITS]
+    sync_aggregate: SyncAggregate
+    # Execution
+    execution_payload: StableExecutionPayload
+    bls_to_execution_changes: List[SignedBLSToExecutionChange, MAX_BLS_TO_EXECUTION_CHANGES]
+    blob_kzg_commitments: List[KZGCommitment, MAX_BLOB_COMMITMENTS_PER_BLOCK]
+    ...  # Additional features from the fork that introduces this EIP
+
+class StableBeaconState(StableContainer[MAX_BEACON_STATE_FIELDS]):
+    # Versioning
+    genesis_time: uint64
+    genesis_validators_root: Root
+    slot: Slot
+    fork: Fork
+    # History
+    latest_block_header: BeaconBlockHeader
+    block_roots: Vector[Root, SLOTS_PER_HISTORICAL_ROOT]
+    state_roots: Vector[Root, SLOTS_PER_HISTORICAL_ROOT]
+    historical_roots: List[Root, HISTORICAL_ROOTS_LIMIT]  # Frozen in Capella, replaced by historical_summaries
+    # Eth1
+    eth1_data: Eth1Data
+    eth1_data_votes: List[Eth1Data, EPOCHS_PER_ETH1_VOTING_PERIOD * SLOTS_PER_EPOCH]
+    eth1_deposit_index: uint64
+    # Registry
+    validators: List[Validator, VALIDATOR_REGISTRY_LIMIT]
+    balances: List[Gwei, VALIDATOR_REGISTRY_LIMIT]
+    # Randomness
+    randao_mixes: Vector[Bytes32, EPOCHS_PER_HISTORICAL_VECTOR]
+    # Slashings
+    slashings: Vector[Gwei, EPOCHS_PER_SLASHINGS_VECTOR]  # Per-epoch sums of slashed effective balances
+    # Participation
+    previous_epoch_participation: List[ParticipationFlags, VALIDATOR_REGISTRY_LIMIT]
+    current_epoch_participation: List[ParticipationFlags, VALIDATOR_REGISTRY_LIMIT]
+    # Finality
+    justification_bits: Bitvector[JUSTIFICATION_BITS_LENGTH]  # Bit set for every recent justified epoch
+    previous_justified_checkpoint: Checkpoint
+    current_justified_checkpoint: Checkpoint
+    finalized_checkpoint: Checkpoint
+    # Inactivity
+    inactivity_scores: List[uint64, VALIDATOR_REGISTRY_LIMIT]
+    # Sync
+    current_sync_committee: SyncCommittee
+    next_sync_committee: SyncCommittee
+    # Execution
+    latest_execution_payload_header: StableExecutionPayloadHeader
+    # Withdrawals
+    next_withdrawal_index: WithdrawalIndex
+    next_withdrawal_validator_index: ValidatorIndex
+    # Deep history valid from Capella onwards
+    historical_summaries: List[HistoricalSummary, HISTORICAL_ROOTS_LIMIT]
+    ...  # Additional features from the fork that introduces this EIP
+```
+
+### Fork-specific `Variant` definitions
+
+These type definitions are specific to the fork that introduces this EIP. They are used in place of the legacy fork-specific `Container` definitions.
+
+```python
+class ExecutionPayload(Variant[StableExecutionPayload]):
+    # Execution block header fields
+    parent_hash: Hash32
+    fee_recipient: ExecutionAddress  # 'beneficiary' in the yellow paper
+    state_root: Bytes32
+    receipts_root: Bytes32
+    logs_bloom: ByteVector[BYTES_PER_LOGS_BLOOM]
+    prev_randao: Bytes32  # 'difficulty' in the yellow paper
+    block_number: uint64  # 'number' in the yellow paper
+    gas_limit: uint64
+    gas_used: uint64
+    timestamp: uint64
+    extra_data: ByteList[MAX_EXTRA_DATA_BYTES]
+    base_fee_per_gas: uint256
+    # Extra payload fields
+    block_hash: Hash32  # Hash of execution block
+    transactions: List[Transaction, MAX_TRANSACTIONS_PER_PAYLOAD]
+    withdrawals: List[Withdrawal, MAX_WITHDRAWALS_PER_PAYLOAD]
+    blob_gas_used: uint64
+    excess_blob_gas: uint64
+    ...  # Additional features from the fork that introduces this EIP
+
+class ExecutionPayloadHeader(Variant[StableExecutionPayloadHeader]):
+    # Execution block header fields
+    parent_hash: Hash32
+    fee_recipient: ExecutionAddress
+    state_root: Bytes32
+    receipts_root: Bytes32
+    logs_bloom: ByteVector[BYTES_PER_LOGS_BLOOM]
+    prev_randao: Bytes32
+    block_number: uint64
+    gas_limit: uint64
+    gas_used: uint64
+    timestamp: uint64
+    extra_data: ByteList[MAX_EXTRA_DATA_BYTES]
+    base_fee_per_gas: uint256
+    # Extra payload fields
+    block_hash: Hash32  # Hash of execution block
+    transactions_root: Root
+    withdrawals_root: Root
+    blob_gas_used: uint64
+    excess_blob_gas: uint64
+    ... # Additional features from the fork that introduces this EIP
+
+class BeaconBlockBody(Variant[StableBeaconBlockBody]):
+    randao_reveal: BLSSignature
+    eth1_data: Eth1Data  # Eth1 data vote
+    graffiti: Bytes32  # Arbitrary data
+    # Operations
+    proposer_slashings: List[ProposerSlashing, MAX_PROPOSER_SLASHINGS]
+    attester_slashings: List[AttesterSlashing, MAX_ATTESTER_SLASHINGS]
+    attestations: List[Attestation, MAX_ATTESTATIONS]
+    deposits: List[Deposit, MAX_DEPOSITS]
+    voluntary_exits: List[SignedVoluntaryExit, MAX_VOLUNTARY_EXITS]
+    sync_aggregate: SyncAggregate
+    # Execution
+    execution_payload: ExecutionPayload
+    bls_to_execution_changes: List[SignedBLSToExecutionChange, MAX_BLS_TO_EXECUTION_CHANGES]
+    blob_kzg_commitments: List[KZGCommitment, MAX_BLOB_COMMITMENTS_PER_BLOCK]
+    ...  # Additional features from the fork that introduces this EIP
+
+class BeaconState(Variant[StableBeaconState]):
+    # Versioning
+    genesis_time: uint64
+    genesis_validators_root: Root
+    slot: Slot
+    fork: Fork
+    # History
+    latest_block_header: BeaconBlockHeader
+    block_roots: Vector[Root, SLOTS_PER_HISTORICAL_ROOT]
+    state_roots: Vector[Root, SLOTS_PER_HISTORICAL_ROOT]
+    historical_roots: List[Root, HISTORICAL_ROOTS_LIMIT]  # Frozen in Capella, replaced by historical_summaries
+    # Eth1
+    eth1_data: Eth1Data
+    eth1_data_votes: List[Eth1Data, EPOCHS_PER_ETH1_VOTING_PERIOD * SLOTS_PER_EPOCH]
+    eth1_deposit_index: uint64
+    # Registry
+    validators: List[Validator, VALIDATOR_REGISTRY_LIMIT]
+    balances: List[Gwei, VALIDATOR_REGISTRY_LIMIT]
+    # Randomness
+    randao_mixes: Vector[Bytes32, EPOCHS_PER_HISTORICAL_VECTOR]
+    # Slashings
+    slashings: Vector[Gwei, EPOCHS_PER_SLASHINGS_VECTOR]  # Per-epoch sums of slashed effective balances
+    # Participation
+    previous_epoch_participation: List[ParticipationFlags, VALIDATOR_REGISTRY_LIMIT]
+    current_epoch_participation: List[ParticipationFlags, VALIDATOR_REGISTRY_LIMIT]
+    # Finality
+    justification_bits: Bitvector[JUSTIFICATION_BITS_LENGTH]  # Bit set for every recent justified epoch
+    previous_justified_checkpoint: Checkpoint
+    current_justified_checkpoint: Checkpoint
+    finalized_checkpoint: Checkpoint
+    # Inactivity
+    inactivity_scores: List[uint64, VALIDATOR_REGISTRY_LIMIT]
+    # Sync
+    current_sync_committee: SyncCommittee
+    next_sync_committee: SyncCommittee
+    # Execution
+    latest_execution_payload_header: ExecutionPayloadHeader
+    # Withdrawals
+    next_withdrawal_index: WithdrawalIndex
+    next_withdrawal_validator_index: ValidatorIndex
+    # Deep history valid from Capella onwards
+    historical_summaries: List[HistoricalSummary, HISTORICAL_ROOTS_LIMIT]
+    ...  # Additional features from the fork that introduces this EIP
+```
+
+## Rationale
+
+### Best timing?
+
+Applying this EIP breaks `hash_tree_root` and Merkle tree verifiers a single time, while promising forward compatibility from the fork going forward. It is best to apply it before merkleization would be broken by different changes. Merkleization is broken by a `Container` reaching a new power of 2 in its number of fields.
+
+### Can this be applied retroactively?
+
+While `Variant` serializes in the same way as the legacy `Container`, the merkleization and `hash_tree_root` of affected data structures changes. Therefore, verifiers that wish to process Merkle proofs of legacy variants still need to support the corresponding legacy schemes.
+
+## Backwards Compatibility
+
+Existing Merkle proof verifiers need to be updated to support the new Merkle tree shape. This includes verifiers in smart contracts on different blockchains and verifiers in hardware wallets, if applicable.
+
+Note that backwards compatibility is also broken when one of the converted `Container` data structures would reach a new power of 2 in its number of fields.
+
+## Security Considerations
+
+None
+
+## Copyright
+
+Copyright and related rights waived via [CC0](../LICENSE.md).