feat: Simplified asynchronous block writers

[ata-nas]

Story:

AS A Block Node Operator
I WANT the Block Node to be as correct and performant as possible
SO THAT I can minimize errors and operating costs.

This task aims to improve the architecture and correctness of of the local persistence handler. As a side benefit we also expect to gain performance improvements.

DEPENDS ON: #309

Tech Notes:

Currently, the local block writer abstraction is neither async, nor stateless. It makes a lot of complex decisions about writing and storage based on a stateful design that is highly susceptible to several forms of race condition and state-status mismatch.
To address these concerns, we need to detach the writer from having any other logic than simply writing a batch of data to the appropriate place(s). The writer must not be bothered with checking any state, making calculations or decisions "should or shouldn't I write some data to some place". The block writer must simply be supplied some data to persist and once it is flagged that it has received everything, it should then complete it's work and be gone once it is done.

One important note is that we are constantly receiving the block stream. It is guaranteed that we have a batch of block items. Batches can start with a block header and can end with block proof. Batches absolutely cannot start with a block proof, and cannot end with a block header. Also, a batch absolutely cannot have a block header or proof somewhere in the middle. This allows us to very easily denote and distinguish where a block starts (we check that a received batch starts with a block header) and where it ends (we check that a received batch ends with a block proof), all batches of items (if any) received in between such "start"/"end" batches are always of items for that delimited block.

All that being said, In order to achieve correct and asynchronous writing of the block files, we need to do the following:

• simplify the block writer:
  • the block writer only knows where to write data that it is being supplied & how to write it
  • the block writer makes no decisions as to wether it must/must not write some data
  • each distinct block (collection of block items from a Header to a Proof) must be supplied to a separate block writer, which then knows how to handle that specific block
• persistence handler is now the decision maker
  • block writers are now managed by the block persistence
  • shift the responsibility of deciding wether to persist a block or continue to accept items from the block writer (current implementation) to the persistence handler, which is receiving the stream, so it can make such decisions and is able to supply a block writer with the correct collection(s) of items
  • each block writer has a transfer queue, where we will be pushing items from the persistence handler as they are received until we denote that a block has ended
  • after a block has ended, we flag the writer that it has received everything and it can continue, asynchronously, to proceed with writing all the items to the appropriate place
  • in the meantime, we are now pushing the following items for the next block in a new writer, managed by the persistence handler