--- eip: 7688 title: Forward compatible consensus data structures description: Transition consensus SSZ data structures to ProgressiveContainer author: Etan Kissling (@etan-status), Cayman (@wemeetagain) discussions-to: https://ethereum-magicians.org/t/eip-7688-forward-compatible-consensus-data-structures/19673 status: Review type: Standards Track category: Core created: 2024-04-15 requires: 6110, 7002, 7251, 7495, 7549, 7569, 7916 --- ## Abstract This EIP defines the changes needed to adopt `ProgressiveContainer` from [EIP-7495](./eip-7495.md) and `ProgressiveList` from [EIP-7916](./eip-7916.md) in consensus data structures. ## Motivation Ethereum's consensus data structures make heavy use of [Simple Serialize (SSZ)](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/ssz/simple-serialize.md) `Container`, which defines how they are serialized and merkleized. The merkleization scheme allows application implementations to verify that individual fields (and partial fields) have not been tampered with. This is useful, for example, in smart contracts of decentralized staking pools that wish to verify that participating validators have not been slashed. While SSZ `Container` defines how data structures are merkleized, the merkleization is prone to change across the different forks. When that happens, e.g., because new features are added or old features get removed, existing verifier implementations need to be updated to be able to continue processing proofs. `ProgressiveContainer`, of [EIP-7495](./eip-7495.md), is a forward compatible alternative that guarantees a forward compatible merkleization scheme. By transitioning consensus data structures to use `ProgressiveContainer`, smart contracts that contain verifier logic no longer have to be maintained in lockstep with Ethereum's fork schedule as long as the underlying features that they verify don't change. For example, as long as the concept of slashing is represented using the boolean `slashed` field, existing verifiers will not break when unrelated features get added or removed. This is also true for off-chain verifiers, e.g., in hardware wallets or in operating systems for mobile devices that are on a different software update cadence than Ethereum. ## Specification The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 and RFC 8174. ### `Container` conversion `Container` types that are expected to evolve over forks SHALL be redefined as `ProgressiveContainer(active_fields=[1] * len(type.fields()))`. For example, given a type in the old fork: ```python class Foo(Container): a: uint8 b: uint16 ``` This type can be converted to support stable Merkleization in the new fork: ```python class Foo(ProgressiveContainer(active_fields=[1, 1])): a: uint8 b: uint16 ``` As part of the conversion, a stable [generalized index (gindex)](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/ssz/merkle-proofs.md#generalized-merkle-tree-index) is assigned to each field that remains valid in future forks. - If a fork appends a field, `active_fields` MUST be extended with a trailing `1`. - If a fork removes a field, the corresponding `active_fields` bit MUST be changed to `0`. - Compatibility rules SHOULD be enforced, e.g., by defining a `CompatibleUnion[fork_1.Foo, fork_2.Foo, fork_3.Foo, ...]` type in the unit test framework. ### `List[type, N]` / `Bitlist` conversion `List` types frequently have been defined with excessively large capacities `N` with the intention that `N` is never reached in practice. In other cases, the capacity itself has changed over time. - `List` types with dynamic or unbounded capacity semantics SHALL be redefined as `ProgressiveList[type]`, and the application logic SHALL be updated to check for an appropriate limit at runtime. - `Bitlist` types with dynamic or unbounded capacity semantics SHALL be redefined as `ProgressiveBitlist` As part of the conversion, a stable [generalized index (gindex)](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/ssz/merkle-proofs.md#generalized-merkle-tree-index) is assigned to each list element that remains valid regardless of the number of added elements. ### Converted types The following types SHALL be converted to `ProgressiveContainer`: - [`Attestation`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/electra/beacon-chain.md#attestation) - The `aggregation_bits` field is redefined to use `ProgressiveBitlist` - [`IndexedAttestation`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/electra/beacon-chain.md#indexedattestation) - The `attesting_indices` field is redefined to use `ProgressiveList` - [`ExecutionPayload`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/deneb/beacon-chain.md#executionpayload) - The `transactions` and `withdrawals` fields are redefined to use `ProgressiveList` - The `MAX_TRANSACTIONS_PER_PAYLOAD` (1M) limit is no longer enforced (the limit is unreachable with `MAX_PAYLOAD_SIZE` 10 MB) - [`ExecutionRequests`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/electra/beacon-chain.md#executionrequests) - The `deposits`, `withdrawals` and `consolidations` fields are redefined to use `ProgressiveList` - [`BeaconBlockBody`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/electra/beacon-chain.md#beaconblockbody) - The `proposer_slashings`, `attester_slashings`, `attestations`, `deposits`, `voluntary_exits` and `bls_to_execution_changes` fields are redefined to use `ProgressiveList` - [`BeaconState`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/electra/beacon-chain.md#beaconstate) - The `validators`, `balances`, `previous_epoch_participation`, `current_epoch_participation`, `inactivity_scores`, `pending_deposits`, `pending_partial_withdrawals` and `pending_consolidations` fields are redefined to use `ProgressiveList` - The `blob_kzg_commitments`, `kzg_proofs` and `column` fields are redefined to use `ProgressiveList` ### Immutable types These types are used as part of the `ProgressiveContainer` definitions, and, as they are not `ProgressiveContainer` themselves, are considered to have immutable Merkleization. If a future fork requires changing these types in an incompatible way, a new type SHALL be defined and assigned a new field name. | Type | Description | | - | - | | [`Slot`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/phase0/beacon-chain.md#custom-types) | Slot number on the beacon chain | | [`Epoch`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/phase0/beacon-chain.md#custom-types) | Epoch number on the beacon chain, a group of slots | | [`CommitteeIndex`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/phase0/beacon-chain.md#custom-types) | Index of a committee within a slot | | [`ValidatorIndex`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/phase0/beacon-chain.md#custom-types) | Unique index of a beacon chain validator | | [`Gwei`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/phase0/beacon-chain.md#custom-types) | Amount in Gwei (1 ETH = 10^9 Gwei = 10^18 Wei) | | [`Root`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/phase0/beacon-chain.md#custom-types) | Byte vector containing an SSZ Merkle root | | [`Hash32`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/phase0/beacon-chain.md#custom-types) | Byte vector containing an opaque 32-byte hash | | [`Version`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/phase0/beacon-chain.md#custom-types) | Consensus fork version number | | [`BLSPubkey`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/phase0/beacon-chain.md#custom-types) | Cryptographic type representing a BLS12-381 public key | | [`BLSSignature`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/phase0/beacon-chain.md#custom-types) | Cryptographic type representing a BLS12-381 signature | | [`KZGCommitment`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/_features/sharding/polynomial-commitments.md#custom-types) | G1 curve point for the KZG polynomial commitment scheme | | [`Fork`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/phase0/beacon-chain.md#fork) | Consensus fork information | | [`Checkpoint`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/phase0/beacon-chain.md#checkpoint) | Tuple referring to the most recent beacon block up through an epoch's start slot | | [`Validator`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/phase0/beacon-chain.md#validator) | Information about a beacon chain validator | | [`AttestationData`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/phase0/beacon-chain.md#attestationdata) | Vote that attests to the availability and validity of a particular consensus block | | [`Eth1Data`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/phase0/beacon-chain.md#eth1data) | Target tracker for importing deposits from transaction logs | | [`DepositData`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/phase0/beacon-chain.md#depositdata) | Log data emitted as part of a transaction's receipt when depositing to the beacon chain | | [`BeaconBlockHeader`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/phase0/beacon-chain.md#beaconblockheader) | Consensus block header | | [`ProposerSlashing`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/phase0/beacon-chain.md#proposerslashing) | Tuple of two equivocating consensus block headers | | [`Deposit`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/phase0/beacon-chain.md#deposit) | Tuple of deposit data and its inclusion proof | | [`VoluntaryExit`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/phase0/beacon-chain.md#voluntaryexit) | Consensus originated request to exit a validator from the beacon chain | | [`SignedVoluntaryExit`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/phase0/beacon-chain.md#signedvoluntaryexit) | Tuple of voluntary exit request and its signature | | [`SyncAggregate`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/altair/beacon-chain.md#syncaggregate) | Cryptographic type representing an aggregate sync committee signature | | [`ExecutionAddress`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/bellatrix/beacon-chain.md#custom-types) | Byte vector containing an account address on the execution layer | | [`Transaction`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/bellatrix/beacon-chain.md#custom-types) | Byte list containing an RLP encoded transaction | | [`WithdrawalIndex`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/capella/beacon-chain.md#custom-types) | Unique index of a withdrawal from any validator's balance to the execution layer | | [`Withdrawal`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/capella/beacon-chain.md#withdrawal) | Withdrawal from a beacon chain validator's balance to the execution layer | | [`DepositRequest`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/electra/beacon-chain.md#depositrequest) | Tuple of flattened deposit data and its sequential index | | [`WithdrawalRequest`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/electra/beacon-chain.md#withdrawalrequest) | Execution originated request to withdraw from a validator to the execution layer | | [`ConsolidationRequest`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/electra/beacon-chain.md#consolidation) | Execution originated request to consolidate two beacon chain validators | | [`BLSToExecutionChange`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/capella/beacon-chain.md#blstoexecutionchange) | Request to register the withdrawal account address of a beacon chain validator | | [`SignedBLSToExecutionChange`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/capella/beacon-chain.md#signedblstoexecutionchange) | Tuple of withdrawal account address registration request and its signature | | [`ParticipationFlags`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/altair/beacon-chain.md#custom-types) | Participation tracker of a beacon chain validator within an epoch | | [`HistoricalSummary`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/capella/beacon-chain.md#historicalsummary) | Tuple combining a historical block root and historical state root | | [`PendingDeposit`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/electra/beacon-chain.md#pendingdeposit) | Pending operation for depositing to a beacon chain validator | | [`PendingPartialWithdrawal`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/electra/beacon-chain.md#pendingpartialwithdrawal) | Pending operation for withdrawing from a beacon chain validator | | [`PendingConsolidation`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/electra/beacon-chain.md#pendingconsolidation) | Pending operation for consolidating two beacon chain validators | ## Rationale ### Best timing? Applying this EIP breaks `hash_tree_root` and Merkle tree verifiers a single time, while promising forward compatibility from the fork going forward. It is best to apply it before merkleization would be broken by different changes. Merkleization is broken by a `Container` reaching a new power of 2 in its number of fields. ### Can this be applied retroactively? While `ProgressiveContainer` serializes in the same way as the legacy `Container`, the merkleization and `hash_tree_root` of affected data structures changes. Therefore, verifiers that wish to process Merkle proofs of legacy variants still need to support the corresponding legacy schemes. ### Immutability Once a field in a `ProgressiveContainer` has been published, its name can no longer be used to represent a different type in the future. This is in line with historical management of certain cases: - Phase0: `BeaconState` contained `previous_epoch_attestations` / `current_epoch_attestations` - Altair: `BeaconState` replaced these fields with `previous_epoch_participation` / `current_epoch_participation` Furthermore, new fields have to be appended at the end of `ProgressiveContainer`. This is in line with historical management of other cases: - Capella appended `historical_summaries` to `BeaconState` instead of squeezing the new field next to `historical_roots` With `ProgressiveContainer`, stable Merkleization requires these rules to become strict. ### Cleanup opportunities #### `BeaconState` - The `eth1_data`, `eth1_data_votes`, `eth1_deposit_index` and `deposit_requests_start_index` fields could be dropped as they are no longer needed after the [EIP-6110](./eip-6110.md#eth1data-poll-deprecation) transition period finishes. - `historical_summaries` could be redefined to use `ProgressiveList` and also integrate the historical `historical_roots` data by merging in full `HistoricalSummary` data from an archive (`historical_root` is frozen since Capella), simplifying access to historical block and state roots. #### `Attestation` - The `committee_bits` is defined as a `Bitvector`, but the top bits are forced to 0 based on `get_committee_count_per_slot(state, data.target.epoch)`. It could be re-defined as a `ProgressiveBitlist`. #### `IndexedAttestation` The `attesting_indices` are limited to [`MAX_VALIDATORS_PER_COMMITTEE * MAX_COMMITTEES_PER_SLOT`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/electra/beacon-chain.md#indexedattestation), which is insufficient when the `IndexedAttestation` is formed from [`SingleAttestation`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/electra/beacon-chain.md#singleattestation) traffic. `SingleAttestation` allows validators that are not assigned to a slot to produce signatures that are not aggregatable into an `Attestation` (as such validators are not assigned), but that are still slashable. Further, [`MAX_ATTESTER_SLASHINGS_ELECTRA`](https://github.com/ethereum/consensus-specs/blob/b5c3b619887c7850a8c1d3540b471092be73ad84/specs/electra/beacon-chain.md#max-operations-per-block) at 1 limits inclusion efficiency of slashings in non-finality scenarios with a lot of forks where slashings happen across multiple different `AttestationData` values. Limits could be rethought to be based on actual resource usage, e.g., by limiting: - The total number of `attesting_indices` across all `AttesterSlashing` (shared total) - The total number of signature checks across all `AttesterSlashing`, `ProposerSlashing`, `VoluntaryExit`, and `SignedBLSToExecutionChange` messages Limiting totals rather than individual resources would allow extra attester slashings to be included at the cost of potentially delaying inclusion of a couple altruistic messages (if there even are any `VoluntaryExit` / `SignedBLSToExecutionChange` messages at that time), thus increasing security and block packing efficiency. #### `ExecutionPayload` The `block_hash` field could be moved to the top of `ExecutionPayload`, reducing the Merkle proof size. #### `ExecutionRequests` As deposits cannot be retried by the user (they pay the ETH upfront), deposit requests cannot fizzle like other requests; they are always included in the same block (since Pectra). For that reason, the current `MAX_DEPOSIT_REQUESTS_PER_PAYLOAD` (8192) is essentially unbounded at current gas limits, but may eventually become reachable (around ~192M gas, or earlier with gas repricings). The CL limit for deposit requests could be dropped to avoid scaling issues, instead solely relying on EL backpressure (gas fees, 1 ETH deposit minimum). For other requests (withdrawal / consolidation requests), a shared total limit based on the added CL state size could provide more flexibility than the current per-operation limits. For example, in times without consolidation requests, space could be used to enqueue more withdrawal requests. #### `BeaconBlockHeader` The `BeaconBlockHeader` is currently proposed to be kept as is. Updating the `BeaconBlockHeader` to `ProgressiveContainer` is tricky as it breaks `hash_tree_root(latest_block_header)` in the `BeaconState`. One option could be to store `latest_block_header_root` separately, possibly also incorporating the block proposer signature into the hash to avoid proposer signature checks while backfilling historical blocks. #### `Validator` The `Validator` container is currently proposed to be kept as is. Updating the `Validator` to `ProgressiveContainer` would add an extra hash for each validator; validators are mostly immutable so rarely need to be re-hashed. With a conversion, implementations may have to incrementally construct the new `Validator` entries ahead of the fork when validator counts are high. It should be evaluated whether the hashing overhead is worth a clean transition to future fields, e.g., for holding postquantum keys. Also consider that such a transition may also involve a new hash function, which is a breaking change to the Merkle proofs, so generalized indices do not have to be stable across that transition. ## Backwards Compatibility Existing Merkle proof verifiers need to be updated to support the new Merkle tree shape. This includes verifiers in smart contracts on different blockchains and verifiers in hardware wallets, if applicable. ## Security Considerations Before this EIP, all `List` types had a fixed upper bound, enabling implementations to reject messages exceeding that size early. With `ProgressiveList`, that is no longer possible, as there is no more maximum message size. Instead, the length checks have to be implemented as part of P2P gossip validation, and as part of the state transition logic. However, many of the limits are not practically reachable (e.g., the gas limit is reached before the maximum payload size). Further note that SSZ is simple enough that clever implementations could still enforce those length checks on the serialized data, before fully decoding it. All data inbound via libp2p is decrypted, then uncompressed with Snappy, then hashed with `MESSAGE_DOMAIN_VALID_SNAPPY` / `MESSAGE_DOMAIN_INVALID_SNAPPY` prefix depending on whether the decompression worked to compute the libp2p message ID, while honoring a global `MAX_PAYLOAD_SIZE` message size limit. This has to be done even if the underlying SSZ data ends up being invalid. As SSZ does not use variable-length encoding, it does not have uncontrolled blowup (no 1 byte becomes 100 MB). Therefore, attempting to decode a `MAX_PAYLOAD_SIZE` message before checking dynamic `List` limits does not decrease security. Any intentional DoS attacks can already target a heavier portion of the processing pipeline (e.g., by sending invalid BLS signatures, or by sending invalid Snappy data that still needs to be hashed to compute the message ID). Therefore, the EIP does not notably impact security. ## Copyright Copyright and related rights waived via [CC0](../LICENSE.md).