Redesign of block-producer
#480
Replies: 7 comments 16 replies
-
Fault toleranceI think this requires having a dependency graph which associates transactions, batches and notes together in a way that they are reversible. This should be possible to implement without really affecting the design of anything else. In a way this could be an extension of the current inflight constraints system. Something that does bother me though - is it possible for a submitted transaction to fail after we've accepted it? e.g. a transaction/note that must be consumed within |
Beta Was this translation helpful? Give feedback.
-
Block building costs & latencyWhat parts are expensive to perform? For example, if we have a bundle of batches ready to submit as a block, do we expect the block sealing to be quick enough that we just do it inline before submitting it to the store? Or is that something to spawn as a task and await elsewhere before submitting it to the store? Similarly for the batch processing. If we want to minimize latency then we probably want to pipeline things as much as possible? |
Beta Was this translation helpful? Give feedback.
-
[Design] Stages, pipelines & channelsThis is an attempt at using separate tasks / processes connected via channels. These processes are long-lived and are spawned once-off i.e. these are not short futures but instead a stage/subsystem. This tends to have nice properties:
The biggest stumbling block for me is usually the error propagation. How are errors communicated, and what happens to any existing data within the pipelines. In our case these would all need to be reverted / unwound somehow, which if we're careful could be possible. The main issue is that stage N assumes that all of its previous outputs were processed successfully. I'll lay out a design below and we can see if it might hold water:
On 1st glance this looks quite nice, simple and is easy to configure each stage to our liking. Some concerns though:
Mostly I'm concerned that there is some state that needs locking between the different stages which would somewhat negate the benefits, but maybe its not the end of the world. |
Beta Was this translation helpful? Give feedback.
-
[Design] Event loopThese are often used in rust based systems where concurrency, flexibility and extensibility are required. E.g. Core of the system is looping over a receiver of the system events, processing one event at a time like a state machine. The benefits are that code becomes a simple matter of writing if this then that, and that you have Downsides are that the "processing" in the loop must be kept to a minimum which is hard to enforce. And that the event enum can grow to be pretty unwieldy if one isn't careful. Biggest downside imo is that it becomes quite difficult to debug or reason about the order of events without have good traces. I'll try describe the design using the event enum: enum Event {
/// Time for a new block to be sealed. Should check for weird conditions for blocks still being sealed/stored.
BlockTick,
/// Spawn task to store the block. Ensure that previous block was already stored.
BlockSealed(Block),
BlockStored(Block),
/// via RPC, spawn task to verify it.
TxnReceived(Txn),
/// txn inputs validated, add to mempool, maybe submit a set of txns for batching
TxnValidated(Txn),
/// Batch worker produced a new batch and is ready for another set.
BatchProduced(Batch),
} Something that might be painful here is the book keeping between different events. For example, knowing when to submit a set of transactions for batching might occur when a batch has been produced, when a block has been sealed or persisted or when a txn has been added to the mempool. Or all of the above. Additionally, back-pressure is also painful as you will have to manually check before submitting unless you design additional cleverness into the tasks somehow. |
Beta Was this translation helpful? Give feedback.
-
[Design] Central logic unitSimilar to the event based loop, but instead of smaller event transitions we rely more on mutex's and different collections of futures. This is probably the closest to what we currently have and described in #191. I don't have a clear picture in my head for this yet tbh, so this is just a verbal vomit of thoughts.. The transaction receiver should be a separate task with rpc submitting new txns via a channel. This simplifies rpc but also allows for easy back-pressure. It could still use the locks it currently holds to "communicate" with the main system. The main system will
Honestly though the more I fiddle with this idea at the moment, the more I seem to morph it into one of the other options. Might need some more time to percolate 🧠. |
Beta Was this translation helpful? Give feedback.
-
If we want to have all data stored in a central repository, we could have something like a impl TransactionPool {
/// Tries to add a transaction to the pool.
///
/// Internally, this will get all required data from the store to verify that the transaction
/// is valid against the current stored + in-flight state.
pub fn add_transaction(&mut self, tx: ProvenTransaction) -> Result<()> {
...
}
/// Returns a set of transactions to combine into a batch, or None if the transaction pool is
/// empty.
///
/// Internally, this will reach out to the store to get all the required data for building
/// the the proposed batch.
///
/// This will also mark all the selected transactions as being batched (so as not to put them
/// into another batch).
pub fn select_batch(&mut self) -> Option<ProposedBatch> {
...
}
/// Adds the constructed batch to the pool.
///
/// This will mark all the involved traction as having been batched. If this method is not
/// called within some interval after `select_batch()`, the transactions selected for this
/// batched will be released.
pub fn add_batch(&mut self, batch: TransactionBatch) -> Result<()> {
...
}
/// Returns a set of batches to combine into a block, or None if there are no ready batches.
///
/// Internally, this will reach out to the store to get all the required data for building
/// the block.
///
/// This will also mark all selected batches as in the process of being put into a block.
pub fn select_block(&mut self) -> Option<ProposedBlock> {
...
}
/// Marks all transactions in the provided block (and corresponding batches) as committed
/// and persists the block to the store.
pub fn apply_block(&mut self, block: Block) -> Result<()> {
...
}
} Things would be pretty simple if we could put a mutex around this struct but, unfortunately, many of its methods need to access the store and so we cannot block when calling them. But the basic idea is this:
A few thoughts on concurrency:
|
Beta Was this translation helpful? Give feedback.
-
I have a rough design that I think minimizes state lock contention. The design centers around decoupling incoming transaction state, both from each other and from the block/batch processing. Starting with incoming transactions, we use an optimistic/latest state view only for validating and updating inputs: /// The latest state of the chain, including inflight transactions.
///
/// Right now it is simplified by assuming all state can be stored in-memory.
/// If this becomes prohibitively large then it is possible to drop some of it
/// back to disk/cold-storage but this would require more care/complexity which
/// I suggest we defer.
struct RwLock<StateView> {
/// The inner mutex provides allows incoming txns to take a read-only lock of the entire
/// structure, allowing concurrent access by multiple txns at once.
account: Map<AccountId, Mutex<AccountState>>
/// Only active notes - does not include notes already nullified by a block
/// though we could also track those if we cared to for some reason.
notes: Map<NoteId, Mutex<NoteState>>,
}
enum AccountState {
Inflight(Arc<Txn>),
Storage(StateDigest)
}
enum NoteState {
Inflight(Arc<Txn>),
Storage,
/// Was from storage, but already consumed by an inflight txn.
/// Ideally we would remove this note from the pool instead, but
/// that would require write access to the mapping.
Consumed,
}
impl StateView {
/// A transaction is only accepted once it can lock and verify all of its inputs
/// at the same time.
///
/// The new transaction is also added as a child to all existing inflight transactions in
/// the required inputs.
fn add_txn(&self, tx: Txn) -> Result<()> {..}
/// Aquire write lock and update/remove the mappings. We need an exclusive lock because
/// we want all of this to happen at once.
fn block_update(&mut self, block: Block, dead_txns: Vec<Arc<Txn>> {..}
} Lastly we have the transaction pool which is where batch selection strategies can be performed on. I'm leaving out the stuff required for batch creation and management. struct TransactionPool {
/// Transactions which have no child dependencies i.e. all of its inputs are
/// on-disk/in-blocks already. The roots of the transaction graph.
roots: Mutex<Vec<Arc<Txns>>>,
}
impl TransactionPool {
/// Called by `add_txn` only when it creates a txn with no inflight deps.
fn add_root(&self, tx: Arc<Txn>) {..}
/// Write locks StateView and roots before mutating them with the block changes.
fn apply_block(&self) ...
} |
Beta Was this translation helpful? Give feedback.
-
Background
The
block-producer
component is responsible for receiving new transactions and sequencing them into blocks. The existing code gets the job done but was written to get up and going quickly. We want to redesign this more meticulously to allow for better abstractions and trying out different sequencing strategies.Transactions are grouped together into batches which in turn form a block. These batches enable concurrency and recursive proving which increases throughput.
Open issues #185, #191 and #196 would be superseded this new design.
Goals
Design a system where we can freely change transaction batching and batch selection strategies without causing the system to be reworked constantly.
As an example, if should be possible to go from a FIFO model to a fee-based strategy without worrying overly about correctness.
Additional considerations
block-producer
.Beta Was this translation helpful? Give feedback.
All reactions