Skip to content

Speed up appending changes to a batch #6025

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Conversation

lutter
Copy link
Collaborator

@lutter lutter commented May 21, 2025

When we finish processing a block, we put all the changes into a Batch; that batch in turn is then added to the write queue where it might be combined with already existing batches for earlier blocks. An important aspect of speeding up writes is that block ranges for entities that have changes in several batches in the queue are properly closed before changes are written to the database, making it unnecessary to do that with much slower SQL queries.

The way this was done inside the Batch was fairly slow: we kept all changes for entities of a given type in a list, and when we needed to add a new change, we would search the entire list to find the last change for the same entity so we could manipulate the block range of that appropriately.

With this change, we keep a hash map from entity id to most recent change so that looking up those changes doesn't require traversing a list anymore.

The speedup can be seen with the append_row example. For example, which is fairly realistic,

GRAPH_STORE_WRITE_BATCH_MEMOIZE=false cargo run --release --example append_row -- --ids 4000 -r 9000

reports a time of about 110ms to construct the batch, whereas running the same with GRAPH_STORE_WRITE_BATCH_MEMOIZE=true takes only about 1.8ms.

@lutter lutter requested a review from zorancv May 21, 2025 19:25
Copy link
Contributor

@zorancv zorancv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants