Skip to content

Commit

Permalink
Update EIP-6493: Remove the need for a wrapper type
Browse files Browse the repository at this point in the history
Merged by EIP-Bot.
  • Loading branch information
etan-status authored Sep 1, 2023
1 parent 7040c4b commit 1e1fe7b
Show file tree
Hide file tree
Showing 7 changed files with 82 additions and 112 deletions.
30 changes: 10 additions & 20 deletions EIPS/eip-6493.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ Definitions from existing specifications that are used throughout this document

### SSZ `SignedTransaction` container

All SSZ transactions are represented as a single, normalized SSZ container. The definition uses the `StableContainer[T, N]` SSZ type and `Optional[E]` as defined in [EIP-7495](./eip-7495.md).
All SSZ transactions are represented as a single, normalized SSZ container. The definition uses the `StableContainer[N]` SSZ type and `Optional[T]` as defined in [EIP-7495](./eip-7495.md).

| Name | Value | Description |
| - | - | - |
Expand All @@ -93,8 +93,7 @@ class AccessTuple(Container):
address: ExecutionAddress
storage_keys: List[Hash32, MAX_ACCESS_LIST_STORAGE_KEYS]

@dataclass
class TransactionPayload:
class TransactionPayload(StableContainer[MAX_TRANSACTION_PAYLOAD_FIELDS]):
nonce: uint64
max_fee_per_gas: uint256
gas: uint64
Expand All @@ -112,17 +111,16 @@ class TransactionPayload:
max_fee_per_blob_gas: Optional[uint256]
blob_versioned_hashes: Optional[List[VersionedHash, MAX_BLOB_COMMITMENTS_PER_BLOCK]]

@dataclass
class TransactionSignature:
class TransactionSignature(StableContainer[MAX_TRANSACTION_SIGNATURE_FIELDS]):
from_: ExecutionAddress
ecdsa_signature: ByteVector[ECDSA_SIGNATURE_SIZE]

# EIP-2718
type_: Optional[TransactionType]

class SignedTransaction(Container):
payload: StableContainer[TransactionPayload, MAX_TRANSACTION_PAYLOAD_FIELDS]
signature: StableContainer[TransactionSignature, MAX_TRANSACTION_SIGNATURE_FIELDS]
payload: TransactionPayload
signature: TransactionSignature

def check_transaction_supported(tx: SignedTransaction):
if tx.payload.max_fee_per_blob_gas is not None:
Expand Down Expand Up @@ -242,7 +240,7 @@ See [EIP assets](../assets/eip-6493/tx_hashes.py) for a definition of `compute_s

### SSZ `PooledTransaction` container

During transaction gossip responses ([`PooledTransactions`](https://github.com/ethereum/devp2p/blob/6b259a7003b4bfb18365ba690f4b00ba8a26393b/caps/eth.md#pooledtransactions-0x0a)), each `SignedTransaction` is wrapped into a `PooledTransaction`. The definition uses the `StableContainer[T, N]` SSZ type and `Optional[E]` as defined in [EIP-7495](./eip-7495.md).
During transaction gossip responses ([`PooledTransactions`](https://github.com/ethereum/devp2p/blob/6b259a7003b4bfb18365ba690f4b00ba8a26393b/caps/eth.md#pooledtransactions-0x0a)), each `SignedTransaction` is wrapped into a `PooledTransaction`. The definition uses the `StableContainer[N]` SSZ type and `Optional[T]` as defined in [EIP-7495](./eip-7495.md).

| Name | Value | Description |
| - | - | - |
Expand All @@ -254,13 +252,9 @@ class BlobPayload(Container):
kzg_commitment: KZGCommitment
kzg_proof: KZGProof

@dataclass
class PooledTransactionPayload:
class PooledTransaction(StableContainer[MAX_POOLED_TRANSACTION_FIELDS]):
tx: SignedTransaction
blob_payloads: Optional[List[BlobPayload, MAX_BLOB_COMMITMENTS_PER_BLOCK]]

class PooledTransaction(StableContainer[PooledTransactionPayload, MAX_POOLED_TRANSACTION_FIELDS]):
pass
```

The same additional validation constraints as defined in [EIP-4844](./eip-4844.md) also apply to transactions that define `tx.payload.blob_versioned_hashes` or `blob_payloads`.
Expand All @@ -274,7 +268,7 @@ Such changes [do not affect](./eip-7495.md) how existing pooled transactions ser

### SSZ `Receipt` container

All SSZ receipts are represented as a single, normalized SSZ container. The definition uses the `StableContainer[T, N]` SSZ type and `Optional[E]` as defined in [EIP-7495](./eip-7495.md).
All SSZ receipts are represented as a single, normalized SSZ container. The definition uses the `StableContainer[N]` SSZ type and `Optional[T]` as defined in [EIP-7495](./eip-7495.md).

| Name | Value | Description |
| - | - | - |
Expand All @@ -289,8 +283,7 @@ class Log(Container):
topics: List[Bytes32, MAX_TOPICS_PER_LOG]
data: ByteList[MAX_LOG_DATA_SIZE]

@dataclass
class ReceiptPayload:
class Receipt(StableContainer[MAX_RECEIPT_FIELDS]):
root: Optional[Hash32]
gas_used: uint64
contract_address: Optional[ExecutionAddress]
Expand All @@ -299,14 +292,11 @@ class ReceiptPayload:

# EIP-658
status: Optional[boolean]

class Receipt(StableContainer[ReceiptPayload, MAX_RECEIPT_FIELDS]):
pass
```

Future specifications MAY:

- Add fields to the end of `ReceiptPayload`
- Add fields to the end of `Receipt`
- Convert existing fields to `Optional`

Such changes [do not affect](./eip-7495.md) how existing receipts serialize, merkleize, or validate.
Expand Down
75 changes: 38 additions & 37 deletions EIPS/eip-7495.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,17 +12,17 @@ created: 2023-08-18

## Abstract

This EIP introduces a new [Simple Serialize (SSZ) type](https://github.com/ethereum/consensus-specs/blob/67c2f9ee9eb562f7cc02b2ff90d92c56137944e1/ssz/simple-serialize.md) to represent `StableContainer[T, N]` values.
This EIP introduces a new [Simple Serialize (SSZ) type](https://github.com/ethereum/consensus-specs/blob/67c2f9ee9eb562f7cc02b2ff90d92c56137944e1/ssz/simple-serialize.md) to represent `StableContainer[N]` values.

A `StableContainer[T, N]` is an SSZ `Container` with stable serialization and merkleization even when individual fields become optional or new fields are introduced in the future.
A `StableContainer[N]` is an SSZ `Container` with stable serialization and merkleization even when individual fields become optional or new fields are introduced in the future.

## Motivation

Stable containers are currently not representable in SSZ. Adding support provides these benefits:

1. **Stable signatures:** Signing roots derived from a `StableContainer[T, N]` never change. In the context of Ethereum, this is useful for transaction signatures that are expected to remain valid even when future updates introduce additional transaction fields. Likewise, the overall transaction root remains stable and can be used as a perpetual transaction ID.
1. **Stable signatures:** Signing roots derived from a `StableContainer[N]` never change. In the context of Ethereum, this is useful for transaction signatures that are expected to remain valid even when future updates introduce additional transaction fields. Likewise, the overall transaction root remains stable and can be used as a perpetual transaction ID.

2. **Stable merkle proofs:** Merkle proof verifiers that check specific fields of a `StableContainer[T, N]` do not need continuous updating when future updates introduce additional fields. Common fields always merkleize at the same [generalized indices](https://github.com/ethereum/consensus-specs/blob/67c2f9ee9eb562f7cc02b2ff90d92c56137944e1/ssz/merkle-proofs.md).
2. **Stable merkle proofs:** Merkle proof verifiers that check specific fields of a `StableContainer[N]` do not need continuous updating when future updates introduce additional fields. Common fields always merkleize at the same [generalized indices](https://github.com/ethereum/consensus-specs/blob/67c2f9ee9eb562f7cc02b2ff90d92c56137944e1/ssz/merkle-proofs.md).

3. **Compact serialization:** SSZ serialization is compact; inactive fields do not consume space.

Expand All @@ -32,29 +32,36 @@ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "S

### Type definition

`StableContainer[T, N]` defines a wrapper around an SSZ `Container` type indicated by `T`. `N` indicates the maximum number of fields to which `T` can grow in the future. `N` MUST be `> 0`.
Similar to the regular [SSZ `Container`](https://github.com/ethereum/consensus-specs/blob/67c2f9ee9eb562f7cc02b2ff90d92c56137944e1/ssz/simple-serialize.md#composite-types), `StableContainer[N]` defines an ordered heterogeneous collection of fields. `N` indicates the potential maximum number of fields to which it can ever grow in the future. `N` MUST be `> 0`.

When wrapped in a `StableContainer[T, N]`, `T` MAY define fields of type `Optional[E]`. Such fields can either represent a value of SSZ type `E`, or absence of a value (indicated by `None`). The [default value](https://github.com/ethereum/consensus-specs/blob/67c2f9ee9eb562f7cc02b2ff90d92c56137944e1/ssz/simple-serialize.md#default-values) of an `Optional[E]` is `None`.
As part of a `StableContainer[N]`, fields of type `Optional[T]` MAY be defined. Such fields can either represent a present value of SSZ type `T`, or indicate absence of a value (indicated by `None`). The [default value](https://github.com/ethereum/consensus-specs/blob/67c2f9ee9eb562f7cc02b2ff90d92c56137944e1/ssz/simple-serialize.md#default-values) of an `Optional[T]` is `None`.

For the purpose of serialization, `StableContainer[T, N]` is always considered ["variable-size"](https://github.com/ethereum/consensus-specs/blob/67c2f9ee9eb562f7cc02b2ff90d92c56137944e1/ssz/simple-serialize.md#variable-size-and-fixed-size) regardless of `T`.
```python
class Example(StableContainer[32]):
a: uint64
b: Optional[uint32]
c: uint16
```

For the purpose of serialization, `StableContainer[N]` is always considered ["variable-size"](https://github.com/ethereum/consensus-specs/blob/67c2f9ee9eb562f7cc02b2ff90d92c56137944e1/ssz/simple-serialize.md#variable-size-and-fixed-size) regardless of the individual field types.

### Stability guarantees

The serialization and merkleization of a `StableContainer[T, N]` remains stable as long as:
The serialization and merkleization of a `StableContainer[N]` remains stable as long as:

- The maximum capacity `N` does not change
- The order of fields within `T` does not change
- New fields are always added to the end of `T`
- Required fields of `T` remain required `E`, or become an `Optional[E]`
- Optional fields of `T` remain `Optional[E]`, or become a required `E`
- The order of fields does not change
- New fields are always added to the end
- Required fields remain required `T`, or become an `Optional[T]`
- Optional fields remain `Optional[T]`, or become a required `T`

When an optional field becomes required, existing messages still have stable serialization and merkleization, but will be rejected on deserialization if not present.

### Serialization

Serialization of `StableContainer[T, N]` is defined similarly to the [existing logic](https://github.com/ethereum/consensus-specs/blob/67c2f9ee9eb562f7cc02b2ff90d92c56137944e1/ssz/simple-serialize.md#vectors-containers-lists) for `Container`. Notable changes are:
Serialization of `StableContainer[N]` is defined similarly to the [existing logic](https://github.com/ethereum/consensus-specs/blob/67c2f9ee9eb562f7cc02b2ff90d92c56137944e1/ssz/simple-serialize.md#vectors-containers-lists) for `Container`. Notable changes are:

- A [`Bitvector[N]`](https://github.com/ethereum/consensus-specs/blob/67c2f9ee9eb562f7cc02b2ff90d92c56137944e1/ssz/simple-serialize.md#composite-types) is constructed, indicating active fields within the `StableContainer[T, N]`. For required fields `E` and optional fields `Optional[E]` with a present value (not `None`), a `True` bit is included. For optional fields `Optional[E]` with a `None` value, a `False` bit is included. The `Bitvector[N]` is padded with `False` bits up through length `N`
- A [`Bitvector[N]`](https://github.com/ethereum/consensus-specs/blob/67c2f9ee9eb562f7cc02b2ff90d92c56137944e1/ssz/simple-serialize.md#composite-types) is constructed, indicating active fields within the `StableContainer[N]`. For required fields `T` and optional fields `Optional[T]` with a present value (not `None`), a `True` bit is included. For optional fields `Optional[T]` with a `None` value, a `False` bit is included. The `Bitvector[N]` is padded with `False` bits up through length `N`
- Only active fields are serialized, i.e., fields with a corresponding `True` bit in the `Bitvector[N]`
- The serialization of the `Bitvector[N]` is prepended to the serialized active fields
- If variable-length fields are serialized, their offsets are relative to the start of serialized active fields, after the `Bitvector[N]`
Expand Down Expand Up @@ -87,37 +94,31 @@ return serialize(active_fields) + b"".join(fixed_parts + variable_parts)

### Deserialization

Deserialization of a `StableContainer[T, N]` starts by deserializing a `Bitvector[N]`. That value MUST be validated against the definition of `T`:
Deserialization of a `StableContainer[N]` starts by deserializing a `Bitvector[N]`. That value MUST be validated:

- For each required field within `T`, the corresponding bit in the `Bitvector[N]` MUST be `True`
- For each optional field within `T`, the corresponding bit in the `Bitvector[N]` is not restricted
- All extra bits in the `Bitvector[N]` that exceed the number of fields within `T` MUST be `False`
- For each required field, the corresponding bit in the `Bitvector[N]` MUST be `True`
- For each optional field, the corresponding bit in the `Bitvector[N]` is not restricted
- All extra bits in the `Bitvector[N]` that exceed the number of fields MUST be `False`

The rest of the data is [deserialized](https://github.com/ethereum/consensus-specs/blob/67c2f9ee9eb562f7cc02b2ff90d92c56137944e1/ssz/simple-serialize.md#deserialization) as `T`, consulting the `Bitvector[N]` to determine what optional fields are present. Absent fields are skipped during deserialization and assigned `None` values.
The rest of the data is [deserialized](https://github.com/ethereum/consensus-specs/blob/67c2f9ee9eb562f7cc02b2ff90d92c56137944e1/ssz/simple-serialize.md#deserialization) same as a regular [SSZ `Container`](https://github.com/ethereum/consensus-specs/blob/67c2f9ee9eb562f7cc02b2ff90d92c56137944e1/ssz/simple-serialize.md#vectors-containers-lists), consulting the `Bitvector[N]` to determine what optional fields are present in the data. Absent fields are skipped during deserialization and assigned `None` values.

### Merkleization

To merkleize a `StableContainer[T, N]`, a `Bitvector[N]` is constructed, indicating active fields within the `StableContainer[T, N]`, using the same process as during serialization.

```python
class StableContainer(Generic[T, N]):
data: T
active_fields: Bitvector[N]
```

The [merkleization specification](https://github.com/ethereum/consensus-specs/blob/67c2f9ee9eb562f7cc02b2ff90d92c56137944e1/ssz/simple-serialize.md#merkleization) is extended with the following helper functions:

- `chunk_count(type)`: calculate the amount of leafs for merkleization of the type.
- `StableContainer[T, N]`: always `N`, regardless of the actual number of fields in `T`
- `mix_in_object`: Given a Merkle root `root` and an auxiliary SSZ object `obj` return `hash(root + hash_tree_root(obj))`.
- `StableContainer[N]`: always `N`, regardless of the actual number of fields in the type definition
- `mix_in_aux`: Given a Merkle root `root` and an auxiliary SSZ object root `aux` return `hash(root + aux)`.

To merkleize a `StableContainer[N]`, a `Bitvector[N]` is constructed, indicating active fields within the `StableContainer[N]`, using the same process as during serialization.

Merkleization `hash_tree_root(value)` of an object `value` is then extended with:
Merkleization `hash_tree_root(value)` of an object `value` is extended with:

- `mix_in_object(merkleize(([hash_tree_root(element) if is_active_field(element) else Bytes32() for element in value.data] + [Bytes32()] * N)[:N]), value.active_fields)` if `value` is a `StableContainer[T, N]`.
- `mix_in_aux(merkleize(([hash_tree_root(element) if is_active_field(element) else Bytes32() for element in value.data] + [Bytes32()] * N)[:N]), hash_tree_root(value.active_fields))` if `value` is a `StableContainer[N]`.

## Rationale

### What are the problems solved by `StableContainer[T, N]`?
### What are the problems solved by `StableContainer[N]`?

Current SSZ types are only stable within one version of a specification, i.e., one fork of Ethereum. This is alright for messages pertaining to a specific fork, such as attestations or beacon blocks. However, it is a limitation for messages that are expected to remain valid across forks, such as transactions or receipts. In order to support evolving the features of such perpetually valid message types, a new SSZ scheme needs to be defined.

Expand All @@ -127,17 +128,17 @@ To avoid restricting design space, the scheme has to support extension with new

Typically, the individual `Union` cases share some form of thematic overlap, sharing certain fields with each other. In a `Union`, shared fields are not necessarily merkleized at the same [generalized indices](https://github.com/ethereum/consensus-specs/blob/67c2f9ee9eb562f7cc02b2ff90d92c56137944e1/ssz/merkle-proofs.md). Therefore, Merkle proof systems would have to be updated each time that a new flavor is introduced, even when the actual changes are not of interest to the particular system.

Furthermore, SSZ Union types are currently not used in any final Ethereum specification and do not have a finalized design themselves. The `StableContainer[T, N]` serializes very similar to current `Union[T, U, V]` proposals, with the difference being a `Bitvector[N]` as a prefix instead of a selector byte. This means that the serialized byte lengths are comparable.
Furthermore, SSZ Union types are currently not used in any final Ethereum specification and do not have a finalized design themselves. The `StableContainer[N]` serializes very similar to current `Union[T, U, V]` proposals, with the difference being a `Bitvector[N]` as a prefix instead of a selector byte. This means that the serialized byte lengths are comparable.

### Why not a `Container` full of `Optional[E]`?
### Why not a `Container` full of `Optional[T]`?

If `Optional[E]` is modeled as an SSZ type, each individual field introduces serialization and merkleization overhead. As an `Optional[E]` would be required to be ["variable-size"](https://github.com/ethereum/consensus-specs/blob/67c2f9ee9eb562f7cc02b2ff90d92c56137944e1/ssz/simple-serialize.md#variable-size-and-fixed-size), lots of additional offset bytes would have to be used in the serialization. For merkleization, each individual `Optional[E]` would require mixing in a bit to indicate presence or absence of the value.
If `Optional[T]` is modeled as an SSZ type, each individual field introduces serialization and merkleization overhead. As an `Optional[T]` would be required to be ["variable-size"](https://github.com/ethereum/consensus-specs/blob/67c2f9ee9eb562f7cc02b2ff90d92c56137944e1/ssz/simple-serialize.md#variable-size-and-fixed-size), lots of additional offset bytes would have to be used in the serialization. For merkleization, each individual `Optional[T]` would require mixing in a bit to indicate presence or absence of the value.

Additionally, every time that the number of fields reaches a new power of 2, the Merkle roots break, as the number of chunks doubles. The `StableContainer[T, N]` solves this by artificially extending the Merkle tree to `N` chunks regardless of the actual number of fields defined by the current version of `T`. Therefore, the number of fields is constant and the Merkle tree shape is stable. The overhead of the additional empty placeholder leaves only affects serialization of the `Bitvector[N]` (1 byte per 8 leaves); the number of required hashes during merkleization only grows logarithmically with the total number of leaves.
Additionally, every time that the number of fields reaches a new power of 2, the Merkle roots break, as the number of chunks doubles. The `StableContainer[N]` solves this by artificially extending the Merkle tree to `N` chunks regardless of the actual number of fields currently specified. Because `N` is constant across specification versions, the Merkle tree shape remains stable. The overhead of the additional empty placeholder leaves only affects serialization of the `Bitvector[N]` (1 byte per 8 leaves); the number of required hashes during merkleization only grows logarithmically with `N`.

## Backwards Compatibility

`StableContainer[T, N]` is a new SSZ type and does not conflict with other SSZ types currently in use.
`StableContainer[N]` is a new SSZ type and does not conflict with other SSZ types currently in use.

## Test Cases

Expand Down
Loading

0 comments on commit 1e1fe7b

Please sign in to comment.