Extend AccountProcedureIndexMap with foreign procedures

[Fumuran]

This PR reworks the AccountProcedureIndexMap to store the index map of the foreign accounts.

Closes: #934

[Fumuran]

Current state is extremely raw, but I managed to rework the test to make sure that current approach is working.
As far as I can see, now the main question is how to provide the account code commitments to the host properly, because current version is very straightforward and crude, we certainly should find a more elegant solution.

Edit: I noticed that usage of the executor.load_account_code() function is unnecessary, since I anyway obtain the foreign account code from the advice inputs, differs only the key used to obtain it — code commitment or account id.

[bobbinth]

Thank you! Not a full review, but I left some comments inline. The main thing is that I don't think we need to rely on the advice provider to build procedure maps for foreign accounts.

[Fumuran]

@bobbinth based on your comments I have questions how should we provide the information about foreign account (including foreign procedures).

Right now it works like this:

• I construct the advice inputs with all the foreign account data, including the code commitment mapped with the account procedures (in the get_mock_advice_inputs()) and provide it to the TransactionContext, which then is provided to the TransactionExecutor as a DataStore. That's why I created this get_advice_inputs() function in the DataStore — I had to get somehow the advice inputs which was provided to the TransactionContext.
• TransactionKernel::prepare_inputs() then combines all the inputs from the TransactionContext: it merges the advice inputs with the transaction inputs and arguments, getting the final advice inputs.
• This inputs then provided to host along with an array of foreign account code commitments (first I tried to use the account ID, but that turns out to be an excess information — we need to use code commitment to get the procedures anyway, so why not to use it as an "ID" of the account's code everywhere (especially since it is an ID of the code)). Using code commitments and advice recorder I can obtain all required information to construct the procedures map.

But I see that this scheme should be reworked, and based on your comments the main think we should define is how do we want to provide the information about entire foreign account to the executor. Now it passed through the advice inputs, but since we shouldn't provide the foreign account procedure data using them, should we change the way we do that for the entire foreign account? Or otherwise, will we have to provide foreign account data except the procedures through the advice inputs, and procedures data through, for example, transaction arguments separately?

For now I think we can provide the foreign account data through the transaction arguments instead of the advice inputs — that would allow us to roll back the changes in the DataStore but still provide the foreign account data to the final advice inputs. For that we just need to add the function in the TransactionContextBuilder to provide the data to the advice map (which then will become a transaction args).

---

What we need to do is replace the TODO on lines 101 - 102 above to keep track of all accounts that have been loaded into the executor. For this, we'll probably need to add a new field to the executor maybe something like BTreeMap<AccountId, AccountProcedureIndexMap>. Another option would be to modify AccountProcedureIndexMap to keep track of the maps for multiple accounts.

But this approach will allow us to keep track only of account procedure data. But can we use it to store the entire account? That will solve the issue about how to provide the account data not using the advice provider. In that case we will be able to add the account code to the mast store and simultaneously store the maps representing the whole account data in the data store

[bobbinth]

But I see that this scheme should be reworked, and based on your comments the main think we should define is how do we want to provide the information about entire foreign account to the executor. Now it passed through the advice inputs, but since we shouldn't provide the foreign account procedure data using them, should we change the way we do that for the entire foreign account? Or otherwise, will we have to provide foreign account data except the procedures through the advice inputs, and procedures data through, for example, transaction arguments separately?

I've created #938 to solve this long-term, but for now we can go with a simpler solution.

But this approach will allow us to keep track only of account procedure data. But can we use it to store the entire account? That will solve the issue about how to provide the account data not using the advice provider. In that case we will be able to add the account code to the mast store and simultaneously store the maps representing the whole account data in the data store

We should still use the current approach for passing account data to the transaction executor (i.e., via the advice inputs). But this approach does not include passing full account code - and for this I created load_account_code() method.

One of the things we can do in this method is to track all code commitments for loaded account code and then pass it to the host the way you do now. So a big part of what you have now should still work - but we do need to get rid of DataStore::get_advice_inputs() method (the DataStore interface does not need to change in this PR).

[bobbinth]

Thank you! Looks good! I left some comments inline - but the overall structure seems correct.

[Fumuran]

I'm not sure what to do with tests: on one hand our previous version was able to check the memory status after the foreign procedure invocation, but on the other it was exceptionally useless in terms of checking the fact whether the correct foreign procedure was executed.

Probably the best option will be to leave the test_load_foreign_account_basic() and test_load_foreign_account_twice() tests as they were before this PR just to check the correctness of the memory layout after FPI, and add a new test which essentially will be the same as the current test_load_foreign_account_twice() test (perhaps only the transaction code could be simplified to execute only one foreign procedure).

[bobbinth]

Looks good! Thank you! I made a small commit to clean up a few things - but otherwise, I think it is good to go.

Probably the best option will be to leave the test_load_foreign_account_basic() and test_load_foreign_account_twice() tests as they were before this PR just to check the correctness of the memory layout after FPI, and add a new test which essentially will be the same as the current test_load_foreign_account_twice() test (perhaps only the transaction code could be simplified to execute only one foreign procedure).

Could we not add extra checks to the same test? I'd do some basic checks in this PR, but leave more robust testing for #917.

[igamigo]

Looks good! Wasn't too familiar with the context of this code but the changes make sense. I tested them and it seems the FPI test on the client now passes.

[Fumuran]

In this last commit I roll back the changes made to the test, since it makes sense to move them to the other PR, which will close the #917.