-
Notifications
You must be signed in to change notification settings - Fork 7
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement GPBFT message compression using zstd #793
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #793 +/- ##
==========================================
- Coverage 69.64% 69.31% -0.33%
==========================================
Files 77 78 +1
Lines 7900 7953 +53
==========================================
+ Hits 5502 5513 +11
- Misses 1949 1979 +30
- Partials 449 461 +12
|
Lol, those decode benchmarks. Zstd is fast. |
We can reduce allocation significantly by pooling the buffers too across both CBOR and ZSTD. but baby steps/small PRs. |
} | ||
|
||
func (c *zstdGMessageEncoding) Decode(v []byte) (*gpbft.GMessage, error) { | ||
cborEncoded, err := c.decompressor.DecodeAll(v, make([]byte, 0, len(v))) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
out of interest - have you tried variations of the dst
argument here? nil
or oversizing by a % given your knowledge of the likely compression ratio?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have. My plan is to introduce pooling of buffers in a separate PR as it is an optimisation. There are a few places in encoding pipeline that we can benefit from it to reduce allocations.
In terms of impact, this probably doesn't matter as much considering the heap allocation numbers from passive testing. It is just wasteful.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
neat
ea3ad13
to
6093105
Compare
Add the ability to compress GPBFT messages controllable via manifest. Implement benchmarks to compare vanilla CBOR and ZSTD encoding. Basic local run: ``` BenchmarkCborEncoding-12 47173 25491 ns/op 135409 B/op 87 allocs/op BenchmarkCborDecoding-12 64550 18078 ns/op 61728 B/op 209 allocs/op BenchmarkZstdEncoding-12 29061 41489 ns/op 193455 B/op 88 allocs/op BenchmarkZstdDecoding-12 66172 17924 ns/op 176517 B/op 211 allocs/op ``` Fixes #786
6093105
to
1578660
Compare
Add the ability to compress GPBFT messages controllable via manifest. Implement benchmarks to compare vanilla CBOR and ZSTD encoding.
Basic local run:
Fixes #786