transport: Remove buffer copies while writing HTTP/2 Data frames #8667
Conversation
arjan-bal
added
Type: Performance
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #8667 +/- ##
==========================================
- Coverage 83.20% 82.85% -0.36%
==========================================
Files 417 415 -2
Lines 32308 32200 -108
==========================================
- Hits 26883 26678 -205
+ Misses 4057 4033 -24
- Partials 1368 1489 +121
🚀 New features to boost your workflow:
|
|
@dfawley : Again moving to your plate if you feel like having a second look. |
| } | ||
| if dSize > 0 { | ||
| var err error | ||
| l.writeBuf, err = reader.Peek(dSize, l.writeBuf) |
There was a problem hiding this comment.
It seems like this buffer can only grow and never shrinks.
- What happens if a slice holds a pointer to a huge amount of data? I believe it isn't possible to free it, but am not certain. E.g.
l.writeBuf = [][]byte{nil, nil, nil, nil, nil, nil, make([]byte, 10GB)}
l.writeBuf = l.writeBuf[:0]- What happens if
cap(l.writeBuf)grows to a large value and then we never need it to be that large ever again?
I think we need to have some way to scale this buffer back down.
There was a problem hiding this comment.
For point 1, I've updated the code to clear the buffer after calling Write. This releases references to all the slices and allows them to be GCed.
With respect to point 2, I've now set a limit of 64 on the buffer's length. If a buffer is longer than that, it's immediately freed after use instead of being cached.
Background on the 64-element limit: The BufferSlice from the proto codec is 1 element. With a potential gRPC header, the length is almost always 2. While custom codecs might produce larger slices, 64 is a generous limit that covers common cases without caching excessive memory.
This change also mitigates a worst-case memory scenario. Since Peek() filters empty slices, a 16KB http2 Data frame (the max size) could theoretically be split into 16K (16,384) distinct 1-byte slices. In that case, the memory overhead for the slice headers alone would be 24 bytes * 16 * 1024 (approx. 393KB), with the 64 size limit, the max held memory is approx 1.5KB. Also note that the framer already has a data buffer that grows up to 16KB, and after this change, that buffer should no longer be used for Data frames.
internal/transport/http_util.go
Outdated
| if len(d) == 0 { | ||
| continue | ||
| } |
There was a problem hiding this comment.
Would this be a bug if it were zero? I would have expected it to be.
If it is, then we should delete it. Write should handle a zero-length buffer as a nop already anyway.
There was a problem hiding this comment.
Removed. There should not be any empty buffers in the list, since Peek() filters them out. This was an artifact from the time I spent root-causing unexpected behavior on the local benchmarks with large payloads
internal/transport/controlbuf.go
Outdated
| // This must never happen since the reader must have at least dSize | ||
| // bytes. | ||
| clear(l.writeBuf) | ||
| l.writeBuf = nil |
There was a problem hiding this comment.
If this is impossible then:
logger.Errorseems like a good idea, unless the caller already does that with what we return..- We probably don't need to bother with the clear/nil (and surely don't want to do both?)?
3 participants
This PR removes 2 buffer copies while writing data frames to the underlying net.Conn: one within gRPC and the other in the framer. Care is taken to avoid any extra heap allocations which can affect performance for smaller payloads.
A CL is out for review which allows using the framer to write frame headers. This PR duplicates the header writing code as a temporary workaround. This PR will be merged only after the CL is merged.
Results
Small payloads
Performance for small payloads increases slightly due to the reduction of a
deferredstatement.Large payloads
Local benchmarks show a ~5-10% regression with 1 MB payloads on my dev machine. The profiles show increased time spent in the copy operation inside the buffered writer. Counterintuitively, copying the grpc header and message data into a larger buffer increased the performance by 4% (compared to master).
To validate this behaviour (extra copy increasing performance) I ran the k8s benchmark for 1MB payloads and 100 concurrent streams which showed ~5% increase in QPS without the copies across multiple runs. Adding a copy reduced the performance.
Load test config file: loadtest.yaml
RELEASE NOTES: