Improve performance of grapheme frequency calculation #133
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…and replacing `filter_map` with `filter` and `map` in two places.
…equency calculation by parallelizing with `rayon` and switching unicode segmentation/clustering crates.
alilleybrinker
approved these changes
Jun 20, 2024
|
Thanks @vcfxb! |
j-lanson
reviewed
Jun 20, 2024
| continue; | ||
| } | ||
|
|
||
| // Count the number of graphemes in this patch, add it to the tortal, |
| let mut total_graphemes: usize = 0; | ||
|
|
||
| // Iterate over the file diffs by reference. | ||
| for file_diff in &commit_diff.diff.file_diffs { |
There was a problem hiding this comment.
I was going to ask if it be possible to parallelize over the file iterator as well, but I figured I should figure out the answer myself before suggesting anything. Have a PR up in #134 to demonstrate how this could be done. I don't have/know the benchmarking setup so I haven't tested it locally myself to see what the savings are, if any.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR improves the performance of grapheme frequency calculation by replacing several single threaded operations with parallel operations using
rayon. Additionally we switch from usingunicode_segmentationtofinl_unicodeto improve performance of grapheme cluster identification and iteration. On my machine, this all results in about a 50% total runtime reduction when checkingnpm/express.