Skip to content

Commit

Permalink
Update EIP-6493: Rename to PartialContainer to StableContainer
Browse files Browse the repository at this point in the history 
Merged by EIP-Bot.
  • Loading branch information
@etan-status
etan-status authored Sep 1, 2023
1 parent 045955d commit b170a3b
Show file tree
Hide file tree
Showing 6 changed files with 62 additions and 64 deletions.
14 changes: 7 additions & 7 deletions EIPS/eip-6493.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ Definitions from existing specifications that are used throughout this document

### SSZ `SignedTransaction` container

All SSZ transactions are represented as a single, normalized SSZ container. The definition uses the `PartialContainer[T, N]` SSZ type and `Optional[E]` as defined in [EIP-7495](./eip-7495.md).
All SSZ transactions are represented as a single, normalized SSZ container. The definition uses the `StableContainer[T, N]` SSZ type and `Optional[E]` as defined in [EIP-7495](./eip-7495.md).

| Name | Value | Description |
| - | - | - |
Expand Down Expand Up @@ -121,8 +121,8 @@ class TransactionSignature:
type_: Optional[TransactionType]

class SignedTransaction(Container):
payload: PartialContainer[TransactionPayload, MAX_TRANSACTION_PAYLOAD_FIELDS]
signature: PartialContainer[TransactionSignature, MAX_TRANSACTION_SIGNATURE_FIELDS]
payload: StableContainer[TransactionPayload, MAX_TRANSACTION_PAYLOAD_FIELDS]
signature: StableContainer[TransactionSignature, MAX_TRANSACTION_SIGNATURE_FIELDS]

def check_transaction_supported(tx: SignedTransaction):
if tx.payload.max_fee_per_blob_gas is not None:
Expand Down Expand Up @@ -242,7 +242,7 @@ See [EIP assets](../assets/eip-6493/tx_hashes.py) for a definition of `compute_s

### SSZ `PooledTransaction` container

During transaction gossip responses ([`PooledTransactions`](https://github.com/ethereum/devp2p/blob/6b259a7003b4bfb18365ba690f4b00ba8a26393b/caps/eth.md#pooledtransactions-0x0a)), each `SignedTransaction` is wrapped into a `PooledTransaction`. The definition uses the `PartialContainer[T, N]` SSZ type and `Optional[E]` as defined in [EIP-7495](./eip-7495.md).
During transaction gossip responses ([`PooledTransactions`](https://github.com/ethereum/devp2p/blob/6b259a7003b4bfb18365ba690f4b00ba8a26393b/caps/eth.md#pooledtransactions-0x0a)), each `SignedTransaction` is wrapped into a `PooledTransaction`. The definition uses the `StableContainer[T, N]` SSZ type and `Optional[E]` as defined in [EIP-7495](./eip-7495.md).

| Name | Value | Description |
| - | - | - |
Expand All @@ -259,7 +259,7 @@ class PooledTransactionPayload:
tx: SignedTransaction
blob_payloads: Optional[List[BlobPayload, MAX_BLOB_COMMITMENTS_PER_BLOCK]]

class PooledTransaction(PartialContainer[PooledTransactionPayload, MAX_POOLED_TRANSACTION_FIELDS]):
class PooledTransaction(StableContainer[PooledTransactionPayload, MAX_POOLED_TRANSACTION_FIELDS]):
pass
```

Expand All @@ -274,7 +274,7 @@ Such changes [do not affect](./eip-7495.md) how existing pooled transactions ser

### SSZ `Receipt` container

All SSZ receipts are represented as a single, normalized SSZ container. The definition uses the `PartialContainer[T, N]` SSZ type and `Optional[E]` as defined in [EIP-7495](./eip-7495.md).
All SSZ receipts are represented as a single, normalized SSZ container. The definition uses the `StableContainer[T, N]` SSZ type and `Optional[E]` as defined in [EIP-7495](./eip-7495.md).

| Name | Value | Description |
| - | - | - |
Expand All @@ -300,7 +300,7 @@ class ReceiptPayload:
# EIP-658
status: Optional[boolean]

class Receipt(PartialContainer[ReceiptPayload, MAX_RECEIPT_FIELDS]):
class Receipt(StableContainer[ReceiptPayload, MAX_RECEIPT_FIELDS]):
pass
```

Expand Down
54 changes: 27 additions & 27 deletions EIPS/eip-7495.md
View file
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
---
eip: 7495
title: SSZ PartialContainer
title: SSZ StableContainer
description: New SSZ type to represent a flexible container with stable serialization and merkleization
author: Etan Kissling (@etan-status)
discussions-to: https://ethereum-magicians.org/t/eip-7495-ssz-partialcontainer/15476
discussions-to: https://ethereum-magicians.org/t/eip-7495-ssz-stablecontainer/15476
status: Review
type: Standards Track
category: Core
Expand All @@ -12,17 +12,17 @@ created: 2023-08-18

## Abstract

This EIP introduces a new [Simple Serialize (SSZ) type](https://github.com/ethereum/consensus-specs/blob/67c2f9ee9eb562f7cc02b2ff90d92c56137944e1/ssz/simple-serialize.md) to represent `PartialContainer[T, N]` values.
This EIP introduces a new [Simple Serialize (SSZ) type](https://github.com/ethereum/consensus-specs/blob/67c2f9ee9eb562f7cc02b2ff90d92c56137944e1/ssz/simple-serialize.md) to represent `StableContainer[T, N]` values.

A `PartialContainer[T, N]` is an SSZ `Container` with stable serialization and merkleization even when individual fields become optional or new fields are introduced in the future.
A `StableContainer[T, N]` is an SSZ `Container` with stable serialization and merkleization even when individual fields become optional or new fields are introduced in the future.

## Motivation

Partial containers are currently not representable in SSZ. Adding support provides these benefits:
Stable containers are currently not representable in SSZ. Adding support provides these benefits:

1. **Stable signatures:** Signing roots derived from a `PartialContainer[T, N]` never change. In the context of Ethereum, this is useful for transaction signatures that are expected to remain valid even when future updates introduce additional transaction fields. Likewise, the overall transaction root remains stable and can be used as a perpetual transaction ID.
1. **Stable signatures:** Signing roots derived from a `StableContainer[T, N]` never change. In the context of Ethereum, this is useful for transaction signatures that are expected to remain valid even when future updates introduce additional transaction fields. Likewise, the overall transaction root remains stable and can be used as a perpetual transaction ID.

2. **Stable merkle proofs:** Merkle proof verifiers that check specific fields of a `PartialContainer[T, N]` do not need continuous updating when future updates introduce additional fields. Common fields always merkleize at the same [generalized indices](https://github.com/ethereum/consensus-specs/blob/67c2f9ee9eb562f7cc02b2ff90d92c56137944e1/ssz/merkle-proofs.md).
2. **Stable merkle proofs:** Merkle proof verifiers that check specific fields of a `StableContainer[T, N]` do not need continuous updating when future updates introduce additional fields. Common fields always merkleize at the same [generalized indices](https://github.com/ethereum/consensus-specs/blob/67c2f9ee9eb562f7cc02b2ff90d92c56137944e1/ssz/merkle-proofs.md).

3. **Compact serialization:** SSZ serialization is compact; inactive fields do not consume space.

Expand All @@ -32,15 +32,15 @@ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "S

### Type definition

`PartialContainer[T, N]` defines a wrapper around an SSZ `Container` type indicated by `T`. `N` indicates the maximum number of fields to which `T` can grow in the future. `N` MUST be `> 0`.
`StableContainer[T, N]` defines a wrapper around an SSZ `Container` type indicated by `T`. `N` indicates the maximum number of fields to which `T` can grow in the future. `N` MUST be `> 0`.

When wrapped in a `PartialContainer[T, N]`, `T` MAY define fields of type `Optional[E]`. Such fields can either represent a value of SSZ type `E`, or absence of a value (indicated by `None`). The [default value](https://github.com/ethereum/consensus-specs/blob/67c2f9ee9eb562f7cc02b2ff90d92c56137944e1/ssz/simple-serialize.md#default-values) of an `Optional[E]` is `None`.
When wrapped in a `StableContainer[T, N]`, `T` MAY define fields of type `Optional[E]`. Such fields can either represent a value of SSZ type `E`, or absence of a value (indicated by `None`). The [default value](https://github.com/ethereum/consensus-specs/blob/67c2f9ee9eb562f7cc02b2ff90d92c56137944e1/ssz/simple-serialize.md#default-values) of an `Optional[E]` is `None`.

For the purpose of serialization, `PartialContainer[T, N]` is always considered ["variable-size"](https://github.com/ethereum/consensus-specs/blob/67c2f9ee9eb562f7cc02b2ff90d92c56137944e1/ssz/simple-serialize.md#variable-size-and-fixed-size) regardless of `T`.
For the purpose of serialization, `StableContainer[T, N]` is always considered ["variable-size"](https://github.com/ethereum/consensus-specs/blob/67c2f9ee9eb562f7cc02b2ff90d92c56137944e1/ssz/simple-serialize.md#variable-size-and-fixed-size) regardless of `T`.

### Stability guarantees

The serialization and merkleization of a `PartialContainer[T, N]` remains stable as long as:
The serialization and merkleization of a `StableContainer[T, N]` remains stable as long as:

- The maximum capacity `N` does not change
- The order of fields within `T` does not change
Expand All @@ -52,9 +52,9 @@ When an optional field becomes required, existing messages still have stable ser

### Serialization

Serialization of `PartialContainer[T, N]` is defined similarly to the [existing logic](https://github.com/ethereum/consensus-specs/blob/67c2f9ee9eb562f7cc02b2ff90d92c56137944e1/ssz/simple-serialize.md#vectors-containers-lists) for `Container`. Notable changes are:
Serialization of `StableContainer[T, N]` is defined similarly to the [existing logic](https://github.com/ethereum/consensus-specs/blob/67c2f9ee9eb562f7cc02b2ff90d92c56137944e1/ssz/simple-serialize.md#vectors-containers-lists) for `Container`. Notable changes are:

- A [`Bitvector[N]`](https://github.com/ethereum/consensus-specs/blob/67c2f9ee9eb562f7cc02b2ff90d92c56137944e1/ssz/simple-serialize.md#composite-types) is constructed, indicating active fields within the `PartialContainer[T, N]`. For required fields `E` and optional fields `Optional[E]` with a present value (not `None`), a `True` bit is included. For optional fields `Optional[E]` with a `None` value, a `False` bit is included. The `Bitvector[N]` is padded with `False` bits up through length `N`
- A [`Bitvector[N]`](https://github.com/ethereum/consensus-specs/blob/67c2f9ee9eb562f7cc02b2ff90d92c56137944e1/ssz/simple-serialize.md#composite-types) is constructed, indicating active fields within the `StableContainer[T, N]`. For required fields `E` and optional fields `Optional[E]` with a present value (not `None`), a `True` bit is included. For optional fields `Optional[E]` with a `None` value, a `False` bit is included. The `Bitvector[N]` is padded with `False` bits up through length `N`
- Only active fields are serialized, i.e., fields with a corresponding `True` bit in the `Bitvector[N]`
- The serialization of the `Bitvector[N]` is prepended to the serialized active fields
- If variable-length fields are serialized, their offsets are relative to the start of serialized active fields, after the `Bitvector[N]`
Expand All @@ -65,19 +65,19 @@ def is_active_field(element):

# Determine active fields
active_fields = Bitvector[N](([is_active_field(element) for element in value] + [False] * N)[:N])
partial_value = [element for element in value if is_active_field(element)]
active_values = [element for element in value if is_active_field(element)]

# Recursively serialize
fixed_parts = [serialize(element) if not is_variable_size(element) else None for element in partial_value]
variable_parts = [serialize(element) if is_variable_size(element) else b"" for element in partial_value]
fixed_parts = [serialize(element) if not is_variable_size(element) else None for element in active_values]
variable_parts = [serialize(element) if is_variable_size(element) else b"" for element in active_values]

# Compute and check lengths
fixed_lengths = [len(part) if part != None else BYTES_PER_LENGTH_OFFSET for part in fixed_parts]
variable_lengths = [len(part) for part in variable_parts]
assert sum(fixed_lengths + variable_lengths) < 2**(BYTES_PER_LENGTH_OFFSET * BITS_PER_BYTE)

# Interleave offsets of variable-size parts with fixed-size parts
variable_offsets = [serialize(uint32(sum(fixed_lengths + variable_lengths[:i]))) for i in range(len(partial_value))]
variable_offsets = [serialize(uint32(sum(fixed_lengths + variable_lengths[:i]))) for i in range(len(active_values))]
fixed_parts = [part if part != None else variable_offsets[i] for i, part in enumerate(fixed_parts)]

# Return the concatenation of the active fields `Bitvector` with the active
Expand All @@ -87,7 +87,7 @@ return serialize(active_fields) + b"".join(fixed_parts + variable_parts)

### Deserialization

Deserialization of a `PartialContainer[T, N]` starts by deserializing a `Bitvector[N]`. That value MUST be validated against the definition of `T`:
Deserialization of a `StableContainer[T, N]` starts by deserializing a `Bitvector[N]`. That value MUST be validated against the definition of `T`:

- For each required field within `T`, the corresponding bit in the `Bitvector[N]` MUST be `True`
- For each optional field within `T`, the corresponding bit in the `Bitvector[N]` is not restricted
Expand All @@ -97,27 +97,27 @@ The rest of the data is [deserialized](https://github.com/ethereum/consensus-spe

### Merkleization

To merkleize a `PartialContainer[T, N]`, a `Bitvector[N]` is constructed, indicating active fields within the `PartialContainer[T, N]`, using the same process as during serialization.
To merkleize a `StableContainer[T, N]`, a `Bitvector[N]` is constructed, indicating active fields within the `StableContainer[T, N]`, using the same process as during serialization.

```python
class PartialContainer(Generic[T, N]):
class StableContainer(Generic[T, N]):
data: T
active_fields: Bitvector[N]
```

The [merkleization specification](https://github.com/ethereum/consensus-specs/blob/67c2f9ee9eb562f7cc02b2ff90d92c56137944e1/ssz/simple-serialize.md#merkleization) is extended with the following helper functions:

- `chunk_count(type)`: calculate the amount of leafs for merkleization of the type.
- `PartialContainer[T, N]`: always `N`, regardless of the actual number of fields in `T`
- `StableContainer[T, N]`: always `N`, regardless of the actual number of fields in `T`
- `mix_in_object`: Given a Merkle root `root` and an auxiliary SSZ object `obj` return `hash(root + hash_tree_root(obj))`.

Merkleization `hash_tree_root(value)` of an object `value` is then extended with:

- `mix_in_object(merkleize(([hash_tree_root(element) if is_active_field(element) else Bytes32() for element in value.data] + [Bytes32()] * N)[:N]), value.active_fields)` if `value` is a `PartialContainer[T, N]`.
- `mix_in_object(merkleize(([hash_tree_root(element) if is_active_field(element) else Bytes32() for element in value.data] + [Bytes32()] * N)[:N]), value.active_fields)` if `value` is a `StableContainer[T, N]`.

## Rationale

### What are the problems solved by `PartialContainer[T, N]`?
### What are the problems solved by `StableContainer[T, N]`?

Current SSZ types are only stable within one version of a specification, i.e., one fork of Ethereum. This is alright for messages pertaining to a specific fork, such as attestations or beacon blocks. However, it is a limitation for messages that are expected to remain valid across forks, such as transactions or receipts. In order to support evolving the features of such perpetually valid message types, a new SSZ scheme needs to be defined.

Expand All @@ -127,25 +127,25 @@ To avoid restricting design space, the scheme has to support extension with new

Typically, the individual `Union` cases share some form of thematic overlap, sharing certain fields with each other. In a `Union`, shared fields are not necessarily merkleized at the same [generalized indices](https://github.com/ethereum/consensus-specs/blob/67c2f9ee9eb562f7cc02b2ff90d92c56137944e1/ssz/merkle-proofs.md). Therefore, Merkle proof systems would have to be updated each time that a new flavor is introduced, even when the actual changes are not of interest to the particular system.

Furthermore, SSZ Union types are currently not used in any final Ethereum specification and do not have a finalized design themselves. The `PartialContainer[T, N]` serializes very similar to current `Union[T, U, V]` proposals, with the difference being a `Bitvector[N]` as a prefix instead of a selector byte. This means that the serialized byte lengths are comparable.
Furthermore, SSZ Union types are currently not used in any final Ethereum specification and do not have a finalized design themselves. The `StableContainer[T, N]` serializes very similar to current `Union[T, U, V]` proposals, with the difference being a `Bitvector[N]` as a prefix instead of a selector byte. This means that the serialized byte lengths are comparable.

### Why not a `Container` full of `Optional[E]`?

If `Optional[E]` is modeled as an SSZ type, each individual field introduces serialization and merkleization overhead. As an `Optional[E]` would be required to be ["variable-size"](https://github.com/ethereum/consensus-specs/blob/67c2f9ee9eb562f7cc02b2ff90d92c56137944e1/ssz/simple-serialize.md#variable-size-and-fixed-size), lots of additional offset bytes would have to be used in the serialization. For merkleization, each individual `Optional[E]` would require mixing in a bit to indicate presence or absence of the value.

Additionally, every time that the number of fields reaches a new power of 2, the Merkle roots break, as the number of chunks doubles. The `PartialContainer[T, N]` solves this by artificially extending the Merkle tree to `N` chunks regardless of the actual number of fields defined by the current version of `T`. Therefore, the number of fields is constant and the Merkle tree shape is stable. The overhead of the additional empty placeholder leaves only affects serialization of the `Bitvector[N]` (1 byte per 8 leaves); the number of required hashes during merkleization only grows logarithmically with the total number of leaves.
Additionally, every time that the number of fields reaches a new power of 2, the Merkle roots break, as the number of chunks doubles. The `StableContainer[T, N]` solves this by artificially extending the Merkle tree to `N` chunks regardless of the actual number of fields defined by the current version of `T`. Therefore, the number of fields is constant and the Merkle tree shape is stable. The overhead of the additional empty placeholder leaves only affects serialization of the `Bitvector[N]` (1 byte per 8 leaves); the number of required hashes during merkleization only grows logarithmically with the total number of leaves.

## Backwards Compatibility

`PartialContainer[T, N]` is a new SSZ type and does not conflict with other SSZ types currently in use.
`StableContainer[T, N]` is a new SSZ type and does not conflict with other SSZ types currently in use.

## Test Cases

See [EIP assets](../assets/eip-7495/tests.py).

## Reference Implementation

See [EIP assets](../assets/eip-7495/partial_container.py), based on `protolambda/remerkleable`.
See [EIP assets](../assets/eip-7495/stable_container.py), based on `protolambda/remerkleable`.

## Security Considerations

Expand Down
Loading

0 comments on commit b170a3b

Please sign in to comment.