From 18b45d358a57905d1368146bd5404eb1dde8410c Mon Sep 17 00:00:00 2001 From: ian Date: Sun, 7 Feb 2021 11:23:38 +0800 Subject: [PATCH 1/6] RFC31: Add a variable length field in the block header --- .../0031-variable-length-header-field.md | 137 +++++++++++++++++- .../1-appending-the-field-at-the-end.md | 23 +++ ...ing-molecule-table-in-new-block-headers.md | 32 ++++ .../3-appending-a-hash-at-the-end.md | 31 ++++ 4 files changed, 222 insertions(+), 1 deletion(-) create mode 100644 rfcs/0031-variable-length-header-field/1-appending-the-field-at-the-end.md create mode 100644 rfcs/0031-variable-length-header-field/2-using-molecule-table-in-new-block-headers.md create mode 100644 rfcs/0031-variable-length-header-field/3-appending-a-hash-at-the-end.md diff --git a/rfcs/0031-variable-length-header-field/0031-variable-length-header-field.md b/rfcs/0031-variable-length-header-field/0031-variable-length-header-field.md index 25bad9671..e7b9a91bc 100644 --- a/rfcs/0031-variable-length-header-field/0031-variable-length-header-field.md +++ b/rfcs/0031-variable-length-header-field/0031-variable-length-header-field.md @@ -1 +1,136 @@ -In review, see +--- +Number: "0031" +Category: Consensus (Hard Fork) +Status: Draft +Author: Ian Yang +Organization: Nervos Foundation +Created: 2021-02-07 +--- + +# Add a variable length field in the block + +## Abstract + +This document proposes adding an optional variable length field to the block. + +## Motivation + +Currently, the block header is a fixed length structure. Each header consists of 208 bytes. + +Many extensions require adding new fields into the block. For example, PoA for testnet requires 65 bytes for each signature, and flyclient also needs to add a 64 bytes hash. + +There's no enough reserved bits in the header for these extensions. There's a workaround to store these data in the cellbase transaction, but this solution has a big overhead for clients which want to quickly verify the data using PoW only. If the data are stored in the cellbase transaction, the client has to download the cellbase transaction and the merkle tree proof of the cellbase transaction, which can be larger than the block header itself. + +This document proposes a solution to add a variable length field in the block. How the new field is interpreted is beyond the scope of this document and must be defined and deployed via a future soft fork. Although the field is added to the block body, nodes can synchronize the block header and this field together in the future version. + +## Specification + +The block header is encoded as a molecule struct, which consists of fixed length fields. The header binary is just the concatenation of all the fields in sequence. + +There are many different ways to add the variable length field to the block header. This RFC proposes to replace the `uncles_hash` in the header with the new field `extra_hash`, which is also a 32-bytes hash. The block will have a new field `extension`. + +There are two important time points to deploy this RFC, activation epoch A and extension application epoch B. + +In blocks before epoch A, the `extension` must be absent. The value of `extra_hash` is the same as the original `uncles_hash` in these blocks, so this RFC will not change the serialized headers of existing blocks. The field `extra_hash` is all zeros when `uncles` is empty, or `ckbhash` on all the uncle header hashes concatenated together. + +``` +uncles_hash = 0 when uncles is empty, otherwise + +uncles_hash = ckbhash(U1 || U2 || ... || Un) + where Ui is the header_hash of the i-th uncle in uncles +``` + +See Appendix for the default hash function `ckbhash`. The annotation `||` means bytes concatenation. + +In blocks generated since epoch A, `extension` can be absent, or any binary with 1 to 96 bytes. The upper limit 96 prevents abusing this field because there's no consensus rule to verify the content of `extension`. The 96 bytes limit allows storing the 64-bytes flyclient hash and extra 32-bytes as a hash on further extension bytes. + +The `extra_hash` is defined as: + +* When `extension` is empty, `extra_hash` is the same as the `uncles_hash`. +* Otherwise `extra_hash = ckbhash(uncles_hash || ckbhash(extension))` + +Since epoch B, consensus will define the schema of `extension` and verify the content. This is a soft fork if the `extension` is at most 96 bytes, because nodes deployed since epoch A do not verify the content of `extension`. + +### P2P Protocols Changes + +The field `uncles_hash` in the block header is renamed to `extra_hash`. + +``` +struct RawHeader { + version: Uint32, + compact_target: Uint32, + timestamp: Uint64, + number: Uint64, + epoch: Uint64, + parent_hash: Byte32, + transactions_root: Byte32, + proposals_hash: Byte32, + extra_hash: Byte32, + dao: Byte32, +} +``` + +The new field `extension` will be added to the block body and following data structures: + +``` +table Block { + header: Header, + uncles: UncleBlockVec, + transactions: TransactionVec, + proposals: ProposalShortIdVec, + extension: Bytes, +} + +table CompactBlock { + header: Header, + short_ids: ProposalShortIdVec, + prefilled_transactions: IndexTransactionVec, + uncles: Byte32Vec, + proposals: ProposalShortIdVec, + extension: Bytes, +} +``` + +For blocks before the activation epoch A, `extension` must be absent. After activation, the node must verify that `extension` is absent or a binary with 1 to 96 bytes, and `uncles` and `extension` match the `extra_hash` in the header. + +Pay attention that the `extension` field will occupy the block size. + +### RPC Changes + +* The `uncles_hash` is renamed to `extra_hash`. +* The new field `extension` is added to the block body RPC response. For blocks generated in ckb2019, it is always empty. + +## Comparison With Alternative Solutions + +1. [Appending the Field At the End](./1-appending-the-field-at-the-end.md) +2. [Using Molecule Table in New Block Headers](./2-using-molecule-table-in-new-block-headers.md) +3. [Appending a Hash At the End](./3-appending-a-hash-at-the-end.md) + +## Appendix + +### ckbhash + +CKB uses [blake2b](https://blake2.net/blake2.pdf) as the default hash algorithm with following configurations: + +- output digest size: 32 +- personalization: ckb-default-hash + +Python 3 Example and test vectors: + +```python +import hashlib +import unittest + +def ckbhash(): + return hashlib.blake2b(digest_size=32, person=b'ckb-default-hash') + +class TestCKBBlake2b(unittest.TestCase): + + def test_empty_message(self): + hasher = ckbhash() + hasher.update(b'') + self.assertEqual('44f4c69744d5f8c55d642062949dcae49bc4e7ef43d388c5a12f42b5633d163e', hasher.hexdigest()) + +if __name__ == '__main__': + unittest.main() +``` diff --git a/rfcs/0031-variable-length-header-field/1-appending-the-field-at-the-end.md b/rfcs/0031-variable-length-header-field/1-appending-the-field-at-the-end.md new file mode 100644 index 000000000..495f58a11 --- /dev/null +++ b/rfcs/0031-variable-length-header-field/1-appending-the-field-at-the-end.md @@ -0,0 +1,23 @@ +### Appending the Field At the End + +The block header size is at least 208 bytes. The first 208 bytes are encoded the same as the current header. The remaining bytes are the variable length field. + +``` ++-----------------------+-----------+ +| | | +| 208-bytes header | New Field | +| | | ++-----------------------+-----------+ +``` + + +Pros + +- Apps that are not interested in the new field can just read the first 208 bytes. + +Cons + +- It's not a valid Molecule buffer. +- It may break the old contract which assumes that the header has only 208 bytes. +- Nodes that do not need the new field still has to download it. +- Header is a variable length structure now. \ No newline at end of file diff --git a/rfcs/0031-variable-length-header-field/2-using-molecule-table-in-new-block-headers.md b/rfcs/0031-variable-length-header-field/2-using-molecule-table-in-new-block-headers.md new file mode 100644 index 000000000..d47ac2533 --- /dev/null +++ b/rfcs/0031-variable-length-header-field/2-using-molecule-table-in-new-block-headers.md @@ -0,0 +1,32 @@ +### Using Molecule Table in New Block Headers + +This solution uses a different molecule schema for the new block headers. If the block header size is 208 bytes, it's encoded using the old schema, otherwise it uses the new one. The new schema converts `RawHeader` into a molecule table and adds a variable length bytes field at the end of `RawHeader`. + +``` +old one: + 208 bytes ++-----+-----------------+ +| | | +|Nonce| RawHeader Stuct | +| | | ++-----+-----------------+ + +new one: + ++-----+-------------------------------+ +| | | +|Nonce| RawHeader Table | +| | | ++-----+-------------------------------+ +``` + +Pros + +- It is a valid Molecule buffer. + +Cons + +- It may break the old contract which assumes that the header has only 208 bytes and is just the concatenation of all members. +- Nodes that do not need the new field still has to download it. +- The molecule table header overhead. +- Header is a variable length structure now. \ No newline at end of file diff --git a/rfcs/0031-variable-length-header-field/3-appending-a-hash-at-the-end.md b/rfcs/0031-variable-length-header-field/3-appending-a-hash-at-the-end.md new file mode 100644 index 000000000..54c906d4f --- /dev/null +++ b/rfcs/0031-variable-length-header-field/3-appending-a-hash-at-the-end.md @@ -0,0 +1,31 @@ +### Appending a Hash At the End + +Instead of adding the new field directly at the end of the header, this solution adds a 32 bytes hash at the end of the header which is the hash of the new variable length field. The header is still a fixed length struct but is 32 bytes larger. If client does not need the extra field, it only has the 32 bytes overhead. Otherwise it has to download both the header and the extra field and verify that the hash matches. + +``` ++-----------------------+--+ +| | | +| 208-bytes header | +----+ +| | | | ++-----------------------+--+ | + | Hash of + | + v + +-----+-----+ + | | + | New Field | + | | + +-----------+ + +``` + +Pros + +- It is a valid Molecule buffer. +- The header still has the fixed length. +- Nodes that do not want the new field only need to download an extra hash to verify the PoW. + +Cons + +- It may break the old contract which assumes that the header has only 208 bytes. +- Extra P2P messages must be added to download the new extension field. From 9c86db3db2be97669ed7efe1a94f8daa63a27318 Mon Sep 17 00:00:00 2001 From: ian Date: Thu, 12 Aug 2021 18:53:23 +0800 Subject: [PATCH 2/6] add test vectors for block hash with extension --- .../0031-variable-length-header-field.md | 233 ++++++++++++++++++ 1 file changed, 233 insertions(+) diff --git a/rfcs/0031-variable-length-header-field/0031-variable-length-header-field.md b/rfcs/0031-variable-length-header-field/0031-variable-length-header-field.md index e7b9a91bc..f8665a7f4 100644 --- a/rfcs/0031-variable-length-header-field/0031-variable-length-header-field.md +++ b/rfcs/0031-variable-length-header-field/0031-variable-length-header-field.md @@ -106,6 +106,239 @@ Pay attention that the `extension` field will occupy the block size. 2. [Using Molecule Table in New Block Headers](./2-using-molecule-table-in-new-block-headers.md) 3. [Appending a Hash At the End](./3-appending-a-hash-at-the-end.md) +## Test Vectors + +### Block Hash + +
Block Template + +```json +{ + "version": "0x0", + "compact_target": "0x20010000", + "current_time": "0x17af3f66555", + "number": "0x3", + "epoch": "0x3e80003000000", + "parent_hash": "0xebf229020f333100942279dc33303ae0dfcbe720d8d11818687e6654c157294c", + "cycles_limit": "0x2540be400", + "bytes_limit": "0x91c08", + "uncles_count_limit": "0x2", + "uncles": [], + "transactions": [ + { + "hash": "0x9110ca9266f89938f09ae6f93cc914b2c856cc842440d56fda6d16ee62543f5c", + "required": false, + "cycles": "0x19f2d1", + "depends": null, + "data": { + "version": "0x0", + "cell_deps": [ + { + "out_point": { + "tx_hash": "0xace5ea83c478bb866edf122ff862085789158f5cbff155b7bb5f13058555b708", + "index": "0x0" + }, + "dep_type": "dep_group" + } + ], + "header_deps": [], + "inputs": [ + { + "since": "0x0", + "previous_output": { + "tx_hash": "0xa563884b3686078ec7e7677a5f86449b15cf2693f3c1241766c6996f206cc541", + "index": "0x7" + } + } + ], + "outputs": [ + { + "capacity": "0x2540be400", + "lock": { + "code_hash": "0x709f3fda12f561cfacf92273c57a98fede188a3f1a59b1f888d113f9cce08649", + "hash_type": "data", + "args": "0xc8328aabcd9b9e8e64fbc566c4385c3bdeb219d7" + }, + "type": null + }, + { + "capacity": "0x2540be400", + "lock": { + "code_hash": "0x9bd7e06f3ecf4be0f2fcd2188b23f1b9fcc88e5d4b65a8637b17723bbda3cce8", + "hash_type": "type", + "args": "0xc8328aabcd9b9e8e64fbc566c4385c3bdeb219d7" + }, + "type": null + }, + { + "capacity": "0x2540be400", + "lock": { + "code_hash": "0x709f3fda12f561cfacf92273c57a98fede188a3f1a59b1f888d113f9cce08649", + "hash_type": "data1", + "args": "0xc8328aabcd9b9e8e64fbc566c4385c3bdeb219d7" + }, + "type": null + } + ], + "outputs_data": [ + "0x", + "0x", + "0x" + ], + "witnesses": [ + "0x550000001000000055000000550000004100000070b823564f7d1f814cc135ddd56fd8e8931b3a7040eaf1fb828adae29736a3cb0bc7f65021135b293d10a22da61fcc64f7cb660bf2c3276ad63630dad0b6099001" + ] + } + } + ], + "proposals": [], + "cellbase": { + "hash": "0x185d1c46fe3c4a0a1a5ae47203df2aeebbb97ac353abcf2c6a3fc2548ecd4eda", + "cycles": null, + "data": { + "version": "0x0", + "cell_deps": [], + "header_deps": [], + "inputs": [ + { + "since": "0x3", + "previous_output": { + "tx_hash": "0x0000000000000000000000000000000000000000000000000000000000000000", + "index": "0xffffffff" + } + } + ], + "outputs": [], + "outputs_data": [], + "witnesses": [ + "0x590000000c00000055000000490000001000000030000000310000009bd7e06f3ecf4be0f2fcd2188b23f1b9fcc88e5d4b65a8637b17723bbda3cce80114000000c8328aabcd9b9e8e64fbc566c4385c3bdeb219d700000000" + ] + } + }, + "work_id": "0x2", + "dao": "0x105cabf31c1fa12eacfa6990f2862300bdaf44b932000000008d5fff03fbfe06", + "extension": "0x626c6f636b202333" +} +``` + +
+ +
Block + +```json +{ + "header": { + "version": "0x0", + "compact_target": "0x20010000", + "timestamp": "0x17af3f66555", + "number": "0x3", + "epoch": "0x3e80003000000", + "parent_hash": "0xebf229020f333100942279dc33303ae0dfcbe720d8d11818687e6654c157294c", + "transactions_root": "0x0bbf9d8946932c9c33a46c8d13b9ecfcf850ccc1728fc9c9c5d14710ad9428ad", + "proposals_hash": "0x0000000000000000000000000000000000000000000000000000000000000000", + "extra_hash": "0xfbbfbaaa0afac7730f4a6102b376986f1f288f3eccb18e0d16d58422aab28aad", + "dao": "0x105cabf31c1fa12eacfa6990f2862300bdaf44b932000000008d5fff03fbfe06", + "nonce": "0x6e43a02f3ed8bb00dea7f78c12fe94f5" + }, + "uncles": [], + "transactions": [ + { + "version": "0x0", + "cell_deps": [], + "header_deps": [], + "inputs": [ + { + "since": "0x3", + "previous_output": { + "tx_hash": "0x0000000000000000000000000000000000000000000000000000000000000000", + "index": "0xffffffff" + } + } + ], + "outputs": [], + "outputs_data": [], + "witnesses": [ + "0x590000000c00000055000000490000001000000030000000310000009bd7e06f3ecf4be0f2fcd2188b23f1b9fcc88e5d4b65a8637b17723bbda3cce80114000000c8328aabcd9b9e8e64fbc566c4385c3bdeb219d700000000" + ] + }, + { + "version": "0x0", + "cell_deps": [ + { + "out_point": { + "tx_hash": "0xace5ea83c478bb866edf122ff862085789158f5cbff155b7bb5f13058555b708", + "index": "0x0" + }, + "dep_type": "dep_group" + } + ], + "header_deps": [], + "inputs": [ + { + "since": "0x0", + "previous_output": { + "tx_hash": "0xa563884b3686078ec7e7677a5f86449b15cf2693f3c1241766c6996f206cc541", + "index": "0x7" + } + } + ], + "outputs": [ + { + "capacity": "0x2540be400", + "lock": { + "code_hash": "0x709f3fda12f561cfacf92273c57a98fede188a3f1a59b1f888d113f9cce08649", + "hash_type": "data", + "args": "0xc8328aabcd9b9e8e64fbc566c4385c3bdeb219d7" + }, + "type": null + }, + { + "capacity": "0x2540be400", + "lock": { + "code_hash": "0x9bd7e06f3ecf4be0f2fcd2188b23f1b9fcc88e5d4b65a8637b17723bbda3cce8", + "hash_type": "type", + "args": "0xc8328aabcd9b9e8e64fbc566c4385c3bdeb219d7" + }, + "type": null + }, + { + "capacity": "0x2540be400", + "lock": { + "code_hash": "0x709f3fda12f561cfacf92273c57a98fede188a3f1a59b1f888d113f9cce08649", + "hash_type": "data1", + "args": "0xc8328aabcd9b9e8e64fbc566c4385c3bdeb219d7" + }, + "type": null + } + ], + "outputs_data": [ + "0x", + "0x", + "0x" + ], + "witnesses": [ + "0x550000001000000055000000550000004100000070b823564f7d1f814cc135ddd56fd8e8931b3a7040eaf1fb828adae29736a3cb0bc7f65021135b293d10a22da61fcc64f7cb660bf2c3276ad63630dad0b6099001" + ] + } + ], + "proposals": [], + "extension": "0x626c6f636b202333" +} +``` + +
+ +The hashes: + +``` +Block Hash: +0xb93dad02d24e9d30c49023d08f84dd8ec34118c1bfec9ed432b75619964686c3 + +Transaction Hashes: +0x185d1c46fe3c4a0a1a5ae47203df2aeebbb97ac353abcf2c6a3fc2548ecd4eda +0x9110ca9266f89938f09ae6f93cc914b2c856cc842440d56fda6d16ee62543f5c +``` + ## Appendix ### ckbhash From 2e737d862b2b759b624e8aaf6f45bd80ef12f6d3 Mon Sep 17 00:00:00 2001 From: ian Date: Wed, 18 Aug 2021 11:58:54 +0800 Subject: [PATCH 3/6] apply suggestions from https://github.com/nervosnetwork/rfcs/pull/224#issuecomment-896594788 --- .../0020-ckb-consensus-protocol.md | 4 +++- .../0031-variable-length-header-field.md | 2 +- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/rfcs/0020-ckb-consensus-protocol/0020-ckb-consensus-protocol.md b/rfcs/0020-ckb-consensus-protocol/0020-ckb-consensus-protocol.md index 9f1e22def..e03528e4f 100644 --- a/rfcs/0020-ckb-consensus-protocol/0020-ckb-consensus-protocol.md +++ b/rfcs/0020-ckb-consensus-protocol/0020-ckb-consensus-protocol.md @@ -117,9 +117,11 @@ Similar to NC, in our protocol, a compact block replaces a block’s commitment Additional block structure rules: -- The total size of the first four fields should be no larger than the hard-coded **block size limit**. The main purpose of implementing a block size limit is to avoid overloading public nodes' bandwidth. The uncle blocks’ proposal zones do not count in the limit as they are usually already synchronized when the block is mined. +- The total size of the first four fields should be no larger than the hard-coded **block size limit**. The main purpose of implementing a block size limit is to avoid overloading public nodes' bandwidth. The uncle blocks’ proposal zones do not count in the limit as they are usually already synchronized when the block is mined. Since [RFC31], the new field `extension` is also counted in the total size. - The number of `txpid`s in a proposal zone also has a hard-coded upper bound. +[RFC31]: ../0031-variable-length-header-field/0031-variable-length-header-field.md + Two heuristic requirements may help practitioners choose the parameters. First, the upper bound number of `txpid`s in a proposal zone should be no smaller than the maximum number of committed transactions in a block, so that even if *wclose=wfar*, this bound is not the protocol's throughput bottleneck. Second, ideally the compact block should be no bigger than 80KB. According to [a 2016 study by Croman et al.](https://fc16.ifca.ai/bitcoin/papers/CDE+16.pdf), messages no larger than 80KB have similar propagation latency in the Bitcoin network; larger messages propagate slower as the network throughput becomes the bottleneck. This number may change as the network condition improves. #### Block Propagation Protocol diff --git a/rfcs/0031-variable-length-header-field/0031-variable-length-header-field.md b/rfcs/0031-variable-length-header-field/0031-variable-length-header-field.md index f8665a7f4..6c969752a 100644 --- a/rfcs/0031-variable-length-header-field/0031-variable-length-header-field.md +++ b/rfcs/0031-variable-length-header-field/0031-variable-length-header-field.md @@ -93,7 +93,7 @@ table CompactBlock { For blocks before the activation epoch A, `extension` must be absent. After activation, the node must verify that `extension` is absent or a binary with 1 to 96 bytes, and `uncles` and `extension` match the `extra_hash` in the header. -Pay attention that the `extension` field will occupy the block size. +Pay attention that the `extension` field will occupy the block size. See section [Block and Compact Block Structure](../0020-ckb-consensus-protocol/0020-ckb-consensus-protocol.md#block-and-compact-block-structure) in RFC20 for details. ### RPC Changes From 30fd31c3cde475cb7d8f70cbc0793ccc587722bf Mon Sep 17 00:00:00 2001 From: ian Date: Wed, 18 Aug 2021 11:59:32 +0800 Subject: [PATCH 4/6] apply suggestions from https://github.com/nervosnetwork/rfcs/pull/224#issuecomment-896627400 --- .../0031-variable-length-header-field.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/rfcs/0031-variable-length-header-field/0031-variable-length-header-field.md b/rfcs/0031-variable-length-header-field/0031-variable-length-header-field.md index 6c969752a..fab658e56 100644 --- a/rfcs/0031-variable-length-header-field/0031-variable-length-header-field.md +++ b/rfcs/0031-variable-length-header-field/0031-variable-length-header-field.md @@ -95,6 +95,8 @@ For blocks before the activation epoch A, `extension` must be absent. After acti Pay attention that the `extension` field will occupy the block size. See section [Block and Compact Block Structure](../0020-ckb-consensus-protocol/0020-ckb-consensus-protocol.md#block-and-compact-block-structure) in RFC20 for details. +The uncle blocks packaged in `uncles` will not add the `extension` field. + ### RPC Changes * The `uncles_hash` is renamed to `extra_hash`. From 155749b655825cbfc4e4f851955f996d765d3240 Mon Sep 17 00:00:00 2001 From: ian Date: Wed, 2 Mar 2022 11:49:22 +0800 Subject: [PATCH 5/6] Apply suggestions from code review Co-authored-by: busyforking <5958+janx@users.noreply.github.com> --- .../0031-variable-length-header-field.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0031-variable-length-header-field/0031-variable-length-header-field.md b/rfcs/0031-variable-length-header-field/0031-variable-length-header-field.md index fab658e56..655cc6695 100644 --- a/rfcs/0031-variable-length-header-field/0031-variable-length-header-field.md +++ b/rfcs/0031-variable-length-header-field/0031-variable-length-header-field.md @@ -95,7 +95,7 @@ For blocks before the activation epoch A, `extension` must be absent. After acti Pay attention that the `extension` field will occupy the block size. See section [Block and Compact Block Structure](../0020-ckb-consensus-protocol/0020-ckb-consensus-protocol.md#block-and-compact-block-structure) in RFC20 for details. -The uncle blocks packaged in `uncles` will not add the `extension` field. +The uncle blocks packaged in `uncles` will not include the `extension` field. ### RPC Changes From 7da4dea1e6f83caffa0c86a0442f6b8706ec4f99 Mon Sep 17 00:00:00 2001 From: ian Date: Mon, 14 Mar 2022 16:47:04 +0800 Subject: [PATCH 6/6] revise rfc31 according to the review comment --- .../0031-variable-length-header-field.md | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/rfcs/0031-variable-length-header-field/0031-variable-length-header-field.md b/rfcs/0031-variable-length-header-field/0031-variable-length-header-field.md index 655cc6695..38dafd36e 100644 --- a/rfcs/0031-variable-length-header-field/0031-variable-length-header-field.md +++ b/rfcs/0031-variable-length-header-field/0031-variable-length-header-field.md @@ -15,23 +15,22 @@ This document proposes adding an optional variable length field to the block. ## Motivation -Currently, the block header is a fixed length structure. Each header consists of 208 bytes. +In the consensus version before activating this RFC, the block header is a fixed length structure. Each header consists of 208 bytes. -Many extensions require adding new fields into the block. For example, PoA for testnet requires 65 bytes for each signature, and flyclient also needs to add a 64 bytes hash. +Many extensions require adding new fields into the block, but there’s no enough reserved bits for them. For example, flyclient requires a 64-byte hash in the header. +Workaround exists such as storing these data in the cellbase transaction, but it has a big overhead for clients which want to verify the chain using PoW only. Because they have to download the cellbase transaction and the merkle tree proof of the cellbase transaction, which can be larger than the block header itself. -There's no enough reserved bits in the header for these extensions. There's a workaround to store these data in the cellbase transaction, but this solution has a big overhead for clients which want to quickly verify the data using PoW only. If the data are stored in the cellbase transaction, the client has to download the cellbase transaction and the merkle tree proof of the cellbase transaction, which can be larger than the block header itself. - -This document proposes a solution to add a variable length field in the block. How the new field is interpreted is beyond the scope of this document and must be defined and deployed via a future soft fork. Although the field is added to the block body, nodes can synchronize the block header and this field together in the future version. +This document proposes a solution to add a variable length field in the block. How to interpret the new field is beyond the scope of this document and must be defined and deployed via a future soft fork. Although the field is added to the block body, nodes can synchronize the block header and this field together in the future version. ## Specification The block header is encoded as a molecule struct, which consists of fixed length fields. The header binary is just the concatenation of all the fields in sequence. -There are many different ways to add the variable length field to the block header. This RFC proposes to replace the `uncles_hash` in the header with the new field `extra_hash`, which is also a 32-bytes hash. The block will have a new field `extension`. +There are many ways to add the variable length field to the block header. This RFC proposes to replace the `uncles_hash` in the header with the new field `extra_hash`, which is also a 32-byte hash. The block will have a new field `extension`. There are two important time points to deploy this RFC, activation epoch A and extension application epoch B. -In blocks before epoch A, the `extension` must be absent. The value of `extra_hash` is the same as the original `uncles_hash` in these blocks, so this RFC will not change the serialized headers of existing blocks. The field `extra_hash` is all zeros when `uncles` is empty, or `ckbhash` on all the uncle header hashes concatenated together. +In blocks before epoch A, the `extension` must be absent. The value of `extra_hash` is the same as the original `uncles_hash` in these blocks, so this RFC will not change the serialized headers of existing blocks. The field `extra_hash` is all zeros when the `uncles` field is empty, or `ckbhash` on all the uncle header hashes concatenated together. ``` uncles_hash = 0 when uncles is empty, otherwise @@ -42,7 +41,7 @@ uncles_hash = ckbhash(U1 || U2 || ... || Un) See Appendix for the default hash function `ckbhash`. The annotation `||` means bytes concatenation. -In blocks generated since epoch A, `extension` can be absent, or any binary with 1 to 96 bytes. The upper limit 96 prevents abusing this field because there's no consensus rule to verify the content of `extension`. The 96 bytes limit allows storing the 64-bytes flyclient hash and extra 32-bytes as a hash on further extension bytes. +In blocks generated since epoch A, `extension` can be absent, or any binary with 1 to 96 bytes. The upper limit 96 prevents abusing this field because there's no consensus rule to verify the content of `extension`. The 96 bytes limit allows storing the 64-byte flyclient hash and an extra 32-byte hash on further extension bytes. The `extra_hash` is defined as: