Fix the perf issue in building nodes from splits. #10766
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Create the `relationships` object only once. Otherwise, it recomputes the whole text's hash for every node. It is very inefficient for long text. An alternative approach would be to cache the hash property. However, it wasn't so straightforward as `Document` isn't a cacheable type. I also do not know Python very well, maybe it would be enough to store a simple null and if it isn't null, then don't recompute? However, the most important reason is I'm not sure about the side effects and the existing assumption that the node is mutable and the hash always reflects the state during the call (unless we modify the object in multiple threads). This change doesn't break any assumptions. If the document was modified while we were creating nodes extracted from it, something would be very wrong. Benchmarks taken on a document attached to the bug: Before: Execution time for build_nodes_from_splits: 53.69 seconds After: Execution time for build_nodes_from_splits: 0.18 seconds
dosubot
bot
added
the
size:S
hatianzhang
approved these changes
Feb 15, 2024
logan-markewich
approved these changes
Feb 15, 2024
preemoDez
deleted the
do_not_recalculate_document_hash_for_every_node_bug_10554
branch
Dominastorm
pushed a commit
to uptrain-ai/llama_index
that referenced
this pull request
Feb 28, 2024
* Fix the perf issue in building nodes from splits. Create the `relationships` object only once. Otherwise, it recomputes the whole text's hash for every node. It is very inefficient for long text. An alternative approach would be to cache the hash property. However, it wasn't so straightforward as `Document` isn't a cacheable type. I also do not know Python very well, maybe it would be enough to store a simple null and if it isn't null, then don't recompute? However, the most important reason is I'm not sure about the side effects and the existing assumption that the node is mutable and the hash always reflects the state during the call (unless we modify the object in multiple threads). This change doesn't break any assumptions. If the document was modified while we were creating nodes extracted from it, something would be very wrong. Benchmarks taken on a document attached to the bug: Before: Execution time for build_nodes_from_splits: 53.69 seconds After: Execution time for build_nodes_from_splits: 0.18 seconds * Fix the formatting
Izukimat
pushed a commit
to Izukimat/llama_index
that referenced
this pull request
Mar 29, 2024
* Fix the perf issue in building nodes from splits. Create the `relationships` object only once. Otherwise, it recomputes the whole text's hash for every node. It is very inefficient for long text. An alternative approach would be to cache the hash property. However, it wasn't so straightforward as `Document` isn't a cacheable type. I also do not know Python very well, maybe it would be enough to store a simple null and if it isn't null, then don't recompute? However, the most important reason is I'm not sure about the side effects and the existing assumption that the node is mutable and the hash always reflects the state during the call (unless we modify the object in multiple threads). This change doesn't break any assumptions. If the document was modified while we were creating nodes extracted from it, something would be very wrong. Benchmarks taken on a document attached to the bug: Before: Execution time for build_nodes_from_splits: 53.69 seconds After: Execution time for build_nodes_from_splits: 0.18 seconds * Fix the formatting
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Create the
relationshipsobject only once. Otherwise, it recomputes the whole text's hash for every node. It is very inefficient for long text.An alternative approach would be to cache the hash property. However, it wasn't so straightforward as
Documentisn't a cacheable type. I also do not know Python very well, maybe it would be enough to store a simple null and if it isn't null, then don't recompute? However, the most important reason is I'm not sure about the side effects and the existing assumption that the node is mutable and the hash always reflects the state during the call (unless we modify the object in multiple threads). This change doesn't break any assumptions. If the document was modified while we were creating nodes extracted from it, something would be very wrong.Fixes #10554
Type of Change
Please delete options that are not relevant.
How Has This Been Tested?
Local benchmarks taken on a document attached to the bug:
Before: Execution time for build_nodes_from_splits: 53.69 seconds
After: Execution time for build_nodes_from_splits: 0.18 seconds
Suggested Checklist:
make format; make lintto appease the lint gods