Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pindexer: implement batch processing API #4913

Merged
merged 3 commits into from
Nov 7, 2024
Merged

Conversation

cronokirby
Copy link
Contributor

This restructures the AppView interface to allow processing events in a batch. Previously, app views had to index one event at a time. This PR changes things so that app views get a batch of several blocks worth of events, with a guarantee to have all of the events in any block in the batch.

Making App Views Easier to Write

By having access to all the events in a block, app views are more ergonomic to write. For example, the dex explorer app view wants to know the time of the candlestick events it processes, but to do this, it needs to wait for the block root event later in the block, which provides this timestamp. Currently, because we don't have access to any context, we need to manually implement a queuing system in the database, which is very annoying, and a performance hit.

Making App Views More Performant

We can make app views more performant by processing both an entire block, and multiple blocks, since:

  • we don't need to write an update more than once per block to the database
  • we may be able to write updates less frequently, depending on the app view (e.g. when we need only the current value)
  • we can keep transient state in memory, instead of on the database, reducing writes and reads in all cases

Additional Performance Improvements

Now the app views are run in parallel, which provides additional improvements when syncing up.

Testing

Pindexer should work as usual, after wiping the database.

Checklist

  • I have added guiding text to explain how a reviewer should test these changes.

  • If this code contains consensus-breaking changes, I have added the "consensus-breaking" label. Otherwise, I declare my belief that there are not consensus-breaking changes, for the following reason:

    pindexer only

This would only sync up to the last block - 1, instead of the last
block.
@hdevalence
Copy link
Member

Processing each event individually (the status quo) definitely feels suboptimal. I designed it like that originally just for convenience to get something working.

One high-level design question though is whether the AppView should process blocks individually or in batches of blocks (like this PR). Does pushing the batching into the AppView increase complexity or is it manageable?

@zbuc zbuc assigned zbuc and unassigned zbuc Nov 6, 2024
@zbuc zbuc self-requested a review November 6, 2024 20:33
@cronokirby
Copy link
Contributor Author

I think the complexity offloaded onto the AppViews is actually minimal, because they can "opt out" by:

  • iterating over all events, processing them one-by-one (this is what I've done in the PR for minimal changes)
  • iterating over all blocks, processing them block by block

So this allows us to only ever need to tinker with this for app views that are particular causes of slow syncing.

@cronokirby
Copy link
Contributor Author

Furthermore, a lot of app view workloads look like:

  1. read current state from the database
  2. process event
  3. write information to the database
  4. write state to the database
    in which case batch processing is pretty easy to implement to reduce write loads substantially, by just moving 1 and 4 outside of the event processing loop to only be done once per batch

@cronokirby cronokirby merged commit 7533cc6 into main Nov 7, 2024
14 checks passed
@cronokirby cronokirby deleted the pindexer-batching branch November 7, 2024 01:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants