pindexer: implement batch processing API #4913
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This restructures the AppView interface to allow processing events in a batch. Previously, app views had to index one event at a time. This PR changes things so that app views get a batch of several blocks worth of events, with a guarantee to have all of the events in any block in the batch.
Making App Views Easier to Write
By having access to all the events in a block, app views are more ergonomic to write. For example, the dex explorer app view wants to know the time of the candlestick events it processes, but to do this, it needs to wait for the block root event later in the block, which provides this timestamp. Currently, because we don't have access to any context, we need to manually implement a queuing system in the database, which is very annoying, and a performance hit.
Making App Views More Performant
We can make app views more performant by processing both an entire block, and multiple blocks, since:
Additional Performance Improvements
Now the app views are run in parallel, which provides additional improvements when syncing up.
Testing
Pindexer should work as usual, after wiping the database.
Checklist
I have added guiding text to explain how a reviewer should test these changes.
If this code contains consensus-breaking changes, I have added the "consensus-breaking" label. Otherwise, I declare my belief that there are not consensus-breaking changes, for the following reason: