add threading to add block

[darcys22]

This PR is to open the discussion on adding multithreading to wallet3 to improve performance. After putting wallet3 through perf it highlighted 2 functions where wallet3 spent most of its time to scan the chain. output_and_derivation_ours() and generate_key_derivations()

Making some rough changes to wallet3 and adding threads around add_block resulted in a significant increase in block processing speed. Increasing wallet3 from around 2.5k blocks/second up to 13k blocks/second.
Before:

After:

However processing blocks out of order in this form isn't viable with the wallet outputting incorrect final balances after it finished scanning (in fact every time it resulted in a different balance). I suspect this is due to it ignoring outputs spent because the input hadnt been received yet

I do note that the two bottleneck functions are both in transaction scanning for outputs received and not outputs spent, so we possibly could split the scanning into first stage multithreaded scan of the outputs received, followed by a linear scan of outputs spent.

This is still pretty early days for wallet3 so adding multithreading might not be appropriate yet. But is good to know what the bottlenecks are atm

[majestrate]

if we could std::move the block in this foreach into that lambda, we can capture [this, block=std::move(block)] we can avoid a capture all by reference (which is something i always try to avoid using). that or capture with [this, &block]

[majestrate]

also the insides of add_block sound possibly thread unsafe.

do we mark all functions that are pure with const or no?

[majestrate]

thinking here: what if we used a threadpool kind of entity where we put heavy but pure jobs onto. then have the these heavy functions (if they can be pure) return a std::future that will be fulfilled when the operation is done in the thread pool. we could have those fan out onto as many threads as we want.

the following idea could look something looking like:

auto ftr_derivations = wallet_keys->async_generate_key_derivation(tx_public_keys);
auto derivation = ftr_derivations.get();

[tewinget]

I do note that the two bottleneck functions are both in transaction scanning for outputs received and not outputs spent, so we possibly could split the scanning into first stage multithreaded scan of the outputs received, followed by a linear scan of outputs spent.

That was basically the original plan, at least conceptually, so given that you independently considered it as well, I'd say it's not a bad one.