Implement multi-hop routing of consensus messages

.github/workflows/ubuntu-builder.yml

@@ -11,9 +11,9 @@ jobs:
       fail-fast: false
       matrix:
         include:
-          - builder-image: "ghcr.io/${{ github.repository_owner }}/psibase-builder-ubuntu-2004:b2a46a609b12f645aaeb03245afd0e3d2bf75340"
+          - builder-image: "ghcr.io/${{ github.repository_owner }}/psibase-builder-ubuntu-2004:efa3d81568eb2d4cdc3b3d8a343dd26de0fcba6c"
             ubuntu-version: "2004"
-          - builder-image: "ghcr.io/${{ github.repository_owner }}/psibase-builder-ubuntu-2204:7e3c6e5739df95ba33943789d2cdf5472f91111a"
+          - builder-image: "ghcr.io/${{ github.repository_owner }}/psibase-builder-ubuntu-2204:efa3d81568eb2d4cdc3b3d8a343dd26de0fcba6c"
             ubuntu-version: "2204"
     steps:
       - name: Timestamp

.gitignore

@@ -24,3 +24,4 @@ vite.config.ts.timestamp-*.mjs
 psibase.127.0.0.1.sslip.io*.pem
 psinode_db
 psinode_db_secure
+__pycache__

doc/psidk/src/consensus.md

@@ -3,6 +3,18 @@
 - [CFT](#cft)
 - [BFT](#bft)
 - [Joint Consensus](#joint-consensus)
+- [Special Cases](#special-cases)
+
+## Overview
+
+The blockchain forms a replicated state machine. Each block contains a cryptographic hash identifying the previous block and a bundle of transactions that can be applied to the chain state to create a new state. Block execution is strictly deterministic. Applying the same sequence of blocks to the same initial state will always result in the same final state. The process of consensus ensures that nodes agree on the sequence of blocks to apply, and thus agree on the state of the chain.
+
+New blocks are created by block producers designated in the chain state. After a block producer creates a block, the block is distributed to all other block producers which then agree to make the block irreversible. If there are conflicts, then the consensus algorithm will resolve them. Each consensus algorithm has its own constraints on the behavior of block producers. As long as these constraints are met, the consensus algorithm guarantees safety:
+- Only one fork can ever become irreversible
+- If two blocks are irreversible, one must be a descendant of the other
+- An irreversible block cannot be forked out
+
+All of these formulations are equivalent.
 
 ## CFT
 
@@ -57,24 +69,53 @@ BFT consensus has the following properties:
 
 ### Block Production
 
-Blocks are produced by the leader. Leader selection is round-robin. The producer should build off a chain that includes the best block that has been prepared by 2f+1 producers.
+Blocks are produced by the leader. Leader selection is round-robin.
 
 ### Irreversibility
 
 - Blocks are ordered first by term, then by block height.
 - Blocks go through two rounds of confirmation, prepare and commit. Each producer broadcasts its confirmations to all other producer nodes.
-- A producer can only commit a block that has been prepared by 2f+1 producers.
-- If a producer commits a block A, it can only prepare a conflicting block B that is better than A if at least 2f+1 nodes have already prepared an ancestor of B that is better than A.
 - A block is considered irreversible if it has been committed by 2f+1 nodes.
 
+Block producers must obey these restrictions:
+1. A producer shall only commit a block that has been prepared by 2f+1 producers.
+2. A producer shall not prepare or commit conflicting blocks in the same term
+3. A view change for view N names a block from any prior view that has been prepared by 2f+1 producers. This block must not be worse than any block that the producer has committed in any view before N.
+4. The first block of a term must be a descendant of a block which is the best block referenced by 2f+1 view changes from different producers, and is referenced by the leader's best view change. The leader MAY issue more than one view change and MUST issue a second view change if its original view change refers to a block that is too old.
+
 ### View Change
 
 Each producer maintains a view change timer to detect failure in the leader. The view change timer is restarted whenever irreversibility advances. When the view change timer expires, a producer broadcasts a `view_change_message` to all other producers and increments the current term, moving to the next leader.
 
-- After initiating a view change, a producer will not restart the view change timer until it sees 2f+1 view change messages.
-- Consecutive view changes during which the chain does not make progress, cause the view change timeout to increase by a flat amount for each view change after the first.
+- After initiating a view change, a producer will not restart the view change timer until it sees 2f+1 block producers are in the same term or later.
+- Consecutive view changes during which the chain does not make progress, cause the view change timeout to increase.
 - A view change will trigger immediately if the leader of the current term or at least f+1 other producers are in a later term. This view change is independent of the view change timer and may skip terms.
 
+View change messages are broadcast network-wide. Every node retains the best view change from each active producer and broadcasts it to its peers whenever it receives a better one. When a node changes its view of active producers, it requests the current list of view change messages from all its peers.
+
+The ordering of view change messages from a single node is defined as follows:
+- A higher term is always better than a lower term
+- The ordering of view changes with the same term depends on whether the node that issued the view change is the leader.
+  - If the source is not the leader, a view change is better if it refers to an older block
+  - If the source is the leader, a view change is better if it refers to a newer block
+
+Note: Since the condition for activating a view requires the leader's view change to refer to a block which not older than other nodes' view changes, this ordering guarantees that as long as a valid set of view changes exist, all connected, honest nodes will converge to the same valid set.
+
+### Safety
+
+Let block X and block Y be two conflicting blocks which are both irreversible
+- Without loss of generality, let X be better than Y
+- Y has been committed by 2f+1 producers, because that is the requirement for it to be considered irreversible
+- let Z be the earliest block that is an ancestor of X (inclusive of X), conflicts with Y, is better than Y, and is prepared by 2f+1 producers. Such a block is guaranteed to exist because X itself must be prepared by 2f+1 producers in order to be committed by any honest producer.
+
+The existence of Z implies that at least f+1 producers have violated the protocol
+- If Z and Y have the same term then at least f+1 producers have violated rule 2
+- Y does not have a higher term than Z because Z is better than Y
+- If Z is in a higher term than Y, then there is a valid view change set for the term of Z
+  - Let W be the block named by the leader's view change
+  - If W is worse than Y, then at least f+1 producers have violated rule 3
+  - Otherwise, W is in a lower term than Z and is prepared by 2f+1 producers (by the definition of a view change) and conflicts with Y and is better than Y (otherwise X would be a descendant of Y), which contradicts the definition of Z.
+
 ### Messages
 
 `BlockMessage`
@@ -92,12 +133,17 @@ The leader is selected from any of the old or new producers. Leader elections an
 
 ### CFT → BFT
 
-The leader is selected by election from the old producers. Irreversibility requires a majority of the old producers and a quorum of the new producers. Leader election requires a majority of the old producers and a quorum of the new producers.
+The leader is selected by election from the old producers. Irreversibility requires a majority of the old producers and a quorum of the new producers. Leader election requires a majority of the old producers and a quorum of the new producers. The new producers cannot advance the current term.
 
 ### BFT → CFT
 
 The leader is selected from the old producers. The new producers commit, but do not prepare blocks. Commit require prepares by a quorum of the old producers. Irreversibility requires commits from a quorum of the old producers and a majority of the new producers.
 
 ### BFT → BFT
 
-The leader is selected from the old producers. Committing a block and advancing irreversibility require quorums of both the old and the new producers. Switching forks after a commit is permitted if a quorum of either the old or the new producers have prepared the new fork.
+The leader is selected from the old producers. Activating a view, committing a block, and advancing irreversibility all require quorums of both the old and the new producers. View change auto triggers if f+1 of the old producers are ahead of the current term.
+
+## Special Cases
+
+- Blocks with a single producer are immediately irreversible. This is a degenerate case for both algorithms and they behave identically.
+- The genesis block is immediately irreversible. There is nothing to preserve pre-genesis.