Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix performance degradation in sinks due to slow writer in hot path #12227

Open
pablosichert opened this issue Apr 14, 2022 · 1 comment
Open
Labels
domain: codecs Anything related to Vector's codecs (encoding/decoding) domain: performance Anything related to Vector's performance sink: socket Anything `socket` sink related

Comments

@pablosichert
Copy link
Contributor

pablosichert commented Apr 14, 2022

After integrating encoding::Encoder with sinks, performance dropped by roughly 20% when using JSON encoding: (e.g. for socket #10684 (comment), or for loki #12558 (comment)).

As evident by #10684 (comment) this can be explained by poor writer performance in the encoding hot path. Related issues: #11341 and tokio-rs/bytes#531.

@pablosichert pablosichert added sink: socket Anything `socket` sink related domain: performance Anything related to Vector's performance domain: codecs Anything related to Vector's codecs (encoding/decoding) labels Apr 14, 2022
@pablosichert pablosichert changed the title Fix performance degradation in socket sink due to slow writer in hot path Fix performance degradation in sinks due to slow writer in hot path May 18, 2022
@pablosichert
Copy link
Contributor Author

Related to this, some improvements could be made by reducing the allocations in encode_input

for mut event in events {
self.0.transform(&mut event);
let mut bytes = BytesMut::new();
encoder
.encode(event, &mut bytes)
.map_err(|error| io::Error::new(io::ErrorKind::InvalidData, error))?;
writer.write_all(&bytes)?;
bytes_written += bytes.len();
}
let mut event = last;
self.0.transform(&mut event);
let mut bytes = BytesMut::new();

as mentioned by @tobz in #13518 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
domain: codecs Anything related to Vector's codecs (encoding/decoding) domain: performance Anything related to Vector's performance sink: socket Anything `socket` sink related
Projects
None yet
Development

No branches or pull requests

1 participant