ECRECOVER for extra_field signatures and sealer

[startgeek]

extra_field parameters are unknown and I cant ecrecover

Description

so this is my assumption:
there is a block generated with all of its header's ready to signed and sealed and included in chain
I believe there are manipulations to extra_field data before signing and sealing and after this stage a keccake256 of this half-block is used as MESSAGE for signing and verification (BTW I've done this before with BSC which is also is a POA and it worked smoothly. I don't know what I'm missing here)
these are steps I took.
1.get a block (remove zeros and RLP.decode the extra_field so you get these 3 parts screenshot 1)

fields of extradata:

2.arrange fields in order shown in picture 2 (also for zero values use "0x" and convert of the decimal values to hex);

3. add the manipulated extra_data as last field (tried all of combinations imaginable.)
4. take a keccake256 of this header and use the hash as message
5. as you can see in picture 3 we have 6 signer and 1 sealer using message generated in step 4 we shall be enable to call ECRECOVER on each signature (signers and sealer)
   

Your environment

• OS and version
  ubuntu 20
• version of the Polygon Edge
  tag:0.4.1
• branch that causes this issue
  tag:0.4.1

Steps to reproduce

just run a POA polygon edge and try to recover address of sealer or signers.

• Tell us how to reproduce this issue <br
  it's fairly generic in my case there is 9 validators which makes it 6 signers and 1 sealer and I cant recover any public address from their
  signatures/>
• Where the issue is, if you know <br
  I believe its either is 1.padding 0 's or 2.maybe some fields such as blooms should be "0x" instead of real value
  or 3.the way extra_data field gets nested (BTW I tested all of these) />
• Which commands triggered the issue, if any
  its not depend on any command

Expected behaviour

 it should be easy to get the message and verify all signatures but seems impossible

Actual behaviour

with all of combinations I tried always got the wrong public address

Logs

const arrayToBeRlpeed = [  //fields from aBlock is unmodified and other fields converted to hex
    aBlock.parentHash,
    aBlock.sha3Uncles,
    aBlock.miner, //(Im passing this param without modification ex: 0x0000000000000000000... is this ok?)
    aBlock.stateRoot,
    aBlock.transactionsRoot,
    aBlock.receiptsRoot,
    "0x", //difficulty againg 0x is ethreum way for displaying 0 in rlp is this ok?
    "0x020119",
    "0x500000",
    "0x", //gas 0 used in hex (ethereum way)
    "0x62DE5B5D",

    [
        "0xbe3c39725304e9ecb3341f45c90ca4d729dc0cc4",
        "0x0c142c3e47cb5bef3116dab444de22ac08ae4fcc",
        "0xbc978fe9ec3fe248aaca2c3950fc1c3942ad6f02",
        "0xabe63fe9492a3340e933b27160eb47b8598235b2",
        "0x33855298c8cdaafcb9192a179c32240d78807aae",
        "0xb5aaff8eb4535ded9a245e9906a7346351915bfa",
        "0x327fa1a264d965f0da61a721b42fd1677c0e5d8e",
        "0x43e37753b9dfdf86e50fbc84c03d4d67281f7246",
        "0xc00a0ffcba6bcbcbba638755296fe45bbdd5b00c"]
]
const rlpeed = rlp.encode(arrayToBeRlpeed);
// console.log(`rlpeed header selected: ${rlpeed}`);
const hashHeader = keccak256(rlpeed)
console.log(`firstField extra data decoded: `, rlp.decode(aBlock.extraData)[0]);
let messageSigned = rlp.decode(aBlock.extraData)[1].toString("hex")
console.log(`message signed by validator so if we recover the signature we understand who signed the message:`);
console.log(typeof messageSigned);
var two = rlp.decode(aBlock.extraData)[1];
const RSV = ethJsUtil.fromRpcSig(two)
console.log(`  V is: ${RSV.v}`);
console.log(`  R is: ${RSV.r.toString('hex')}`);
console.log(`  S is: ${RSV.s.toString('hex')}`);
const pubKey = ethJsUtil.ecrecover(hashHeader, RSV.v, RSV.r, RSV.s);

console.log(`public key is: ${pubKey.toString('hex')}`);
const addrBuf = ethJsUtil.pubToAddress(pubKey);
// console.log(addrBuf);

const addr = ethJsUtil.bufferToHex(addrBuf);
console.log(addr);

Proposed solution

A sudo-code or a web3.js (since its heavy standard in industry)example starting from a raw block header to signer and/or sealer public address would be awesome and worth million lines of documents

[zivkovicmilos]

Hey @startgeek,

Thank you for opening up an issue 🙏

I'm pretty sure you're not doing anything wrong, it's just that Edge does some additional steps apart from the ones you've mentioned, which is very misleading.

Please check out these methods that we use for creating committed seals and proposer seals (and for recovering them!):

polygon-edge/consensus/ibft/sign.go

Lines 15 to 89 in 8502125

	func commitMsg(b []byte) []byte {
	// message that the nodes need to sign to commit to a block
	// hash with COMMIT_MSG_CODE which is the same value used in quorum
	return crypto.Keccak256(b, []byte{byte(proto.MessageReq_Commit)})
	}
	func ecrecoverImpl(sig, msg []byte) (types.Address, error) {
	pub, err := crypto.RecoverPubkey(sig, crypto.Keccak256(msg))
	if err != nil {
	return types.Address{}, err
	}
	return crypto.PubKeyToAddress(pub), nil
	}
	func ecrecoverFromHeader(h *types.Header) (types.Address, error) {
	// get the extra part that contains the seal
	extra, err := getIbftExtra(h)
	if err != nil {
	return types.Address{}, err
	}
	// get the sig
	msg, err := calculateHeaderHash(h)
	if err != nil {
	return types.Address{}, err
	}
	return ecrecoverImpl(extra.Seal, msg)
	}
	func signSealImpl(prv *ecdsa.PrivateKey, h *types.Header, committed bool) ([]byte, error) {
	hash, err := calculateHeaderHash(h)
	if err != nil {
	return nil, err
	}
	// if we are singing the committed seals we need to do something more
	msg := hash
	if committed {
	msg = commitMsg(hash)
	}
	seal, err := crypto.Sign(prv, crypto.Keccak256(msg))
	if err != nil {
	return nil, err
	}
	return seal, nil
	}
	func writeSeal(prv *ecdsa.PrivateKey, h *types.Header) (*types.Header, error) {
	h = h.Copy()
	seal, err := signSealImpl(prv, h, false)
	if err != nil {
	return nil, err
	}
	extra, err := getIbftExtra(h)
	if err != nil {
	return nil, err
	}
	extra.Seal = seal
	if err := PutIbftExtra(h, extra); err != nil {
	return nil, err
	}
	return h, nil
	}
	func writeCommittedSeal(prv *ecdsa.PrivateKey, h *types.Header) ([]byte, error) {
	return signSealImpl(prv, h, true)
	}

I'll move this issue to a discussion since it's only Edge related in terms of an implementation that we sadly, cannot change at this point.

[zivkovicmilos]

I also want to point you to the way Edge calculates Header hashes:

polygon-edge/consensus/ibft/header_hash.go

Lines 11 to 45 in 8502125

	func istanbulHeaderHash(h *types.Header) types.Hash {
	// this function replaces extra so we need to make a copy
	h = h.Copy() // Remove later
	arena := fastrlp.DefaultArenaPool.Get()
	defer fastrlp.DefaultArenaPool.Put(arena)
	// when hashing the block for signing we have to remove from
	// the extra field the seal and committed seal items
	extra, err := getIbftExtra(h)
	if err != nil {
	return types.Hash{}
	}
	putIbftExtraValidators(h, extra.Validators)
	vv := arena.NewArray()
	vv.Set(arena.NewBytes(h.ParentHash.Bytes()))
	vv.Set(arena.NewBytes(h.Sha3Uncles.Bytes()))
	vv.Set(arena.NewBytes(h.Miner.Bytes()))
	vv.Set(arena.NewBytes(h.StateRoot.Bytes()))
	vv.Set(arena.NewBytes(h.TxRoot.Bytes()))
	vv.Set(arena.NewBytes(h.ReceiptsRoot.Bytes()))
	vv.Set(arena.NewBytes(h.LogsBloom[:]))
	vv.Set(arena.NewUint(h.Difficulty))
	vv.Set(arena.NewUint(h.Number))
	vv.Set(arena.NewUint(h.GasLimit))
	vv.Set(arena.NewUint(h.GasUsed))
	vv.Set(arena.NewUint(h.Timestamp))
	vv.Set(arena.NewCopyBytes(h.ExtraData))
	buf := keccak.Keccak256Rlp(nil, vv)
	return types.BytesToHash(buf)
	}

[zivkovicmilos]

Note that the hash of the header still takes into account the ExtraData field, but only the validator set and the proposer seal are in there, not the committed seals.

Committed seals are usually something unique for each node in the consensus network, where the validator set and the proposer seal need to be the same. This is why we're dropping the committed seals from the hash calculation - we want all nodes to generate the same block hash, regardless of which committed seals they got

[startgeek]

I ran this code with it's examples and debugger on and to be honest it's still doesnt work.
Also I asked question regarding other fields such as 0 fields and it shall get included in RLP as "0x" or not and fields with long 0000 zeros such as logBloom.

the most clear way for this question is I write a block here and you gave me the input of rlp encode for signing and recovering:

here's the example block:
const aBlock = {
"parentHash": "0x21f3360d46ee0be081392b41d99c286e97d6bd3e771217e276ef7be705b1e176",
"sha3Uncles": "0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
"miner": "0x0000000000000000000000000000000000000000",
"stateRoot": "0x38ff6c87a8ce4a7410e7238055c0c1f37edf45ed4487c729e50f99607573b798",
"transactionsRoot": "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",
"receiptsRoot": "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",
"logsBloom": "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
"difficulty": "131353",
"totalDifficulty": "131353",
"size": 1214,
"number": 131353,
"gasLimit": 5242880,
"gasUsed": 0,
"timestamp": 1658739549,
"extraData": "0x0000000000000000000000000000000000000000000000000000000000000000f90297f8bd94be3c39725304e9ecb3341f45c90ca4d729dc0cc4940c142c3e47cb5bef3116dab444de22ac08ae4fcc94bc978fe9ec3fe248aaca2c3950fc1c3942ad6f0294abe63fe9492a3340e933b27160eb47b8598235b29433855298c8cdaafcb9192a179c32240d78807aae94b5aaff8eb4535ded9a245e9906a7346351915bfa94327fa1a264d965f0da61a721b42fd1677c0e5d8e9443e37753b9dfdf86e50fbc84c03d4d67281f724694c00a0ffcba6bcbcbba638755296fe45bbdd5b00cb841bc49b91be421df8d37f12399592d8474d5ee8561fbf53e5279764d40346f0cd227fc82e099a11702b2fefdd28a19f511788cae81b297c83c3481211c71a5c63500f90192b84108c16ed3a4c039d4a77b1f275f55d788978b7f1aa812904a67d362104664ff0550126d7e6f187abb47df1f4ace6637e5cf89835e60dfbe9da09e9ea520adf7ce00b841be8c118984df049613423bbffbbce38fd17a204bd931f8d405c837101d7428944c0abc4f8cd7f229152efa3f2328e8afe0680bfc3609732c3a1bca50620c06d000b841790f24be65d93db95ace14fb4c06476ab3a052537ae6f614a64f8e2f2c7821bf604916ebd34612a4a0f08f709e3b4e97cea19f81cbc764bf1ae51449529f0a5d01b841d67a6c8be453f67292c54af63c441975a1162f4e2ddd63229149080eb9074ec86b496c1acaf7b96fbba347a5289f28208f737e0db4f1458c496e36d6ec254bbe01b841ab0eabfdce4ca40550c97cb7bcb6499b94f2077f67dd7846c752e8f59fec754c5d74cc75c039394285982e2b3b603c402eabf5ca4375078d5b879f3772bbaff200b841f33b9ed5503a5346a71b3d3dc6b7ea55c2c8589c46b9119f1012655709ca0d9a242f164cb657f9ebb0c7c584db90e47c9f69829bf1467e89d67a19fe92faf02901",
"mixHash": "0x63746963616c2062797a616e74696e65206661756c7420746f6c6572616e6365",
"nonce": "0x0000000000000000",
"hash": "0xadeed557460b4e72bd76003f6ada30df2df34c3471ca5c547b5e2a0a7433211d",
"transactions": [],
"uncles": []
}

[startgeek]

BTW ran your test files and used debug to visually see data in action. but its impossible to reproduce. spend 3 full working day for nothing. there is something buggy going on. if you guy please just make an example of this block it would be super clear