-
Notifications
You must be signed in to change notification settings - Fork 591
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add record_multiplexer
microbenchmarks
#24155
base: dev
Are you sure you want to change the base?
Conversation
record_multiplexer
microbenchmarks
3712bfa
to
da6f4e6
Compare
da6f4e6
to
2332bcb
Compare
record_multiplexer
microbenchmarksrecord_multiplexer
microbenchmarks
a77d575
to
5f7e229
Compare
co_return std::nullopt; | ||
} | ||
|
||
iobuf encode_protobuf_message_index(const std::vector<int32_t>& message_index) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there any existing serializer for this message index format anywhere in our code-base? There is a de-serializer; get_proto_offsets
in src/v/datalake/schema_registry.h
. Happy to move this serializer to a more general location if there is any use for it outside of the record generator.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Closest thing is
redpanda/src/v/datalake/tests/record_schema_resolver_test.cc
Lines 63 to 72 in 514f1df
iobuf encode_pb_offsets(const std::vector<int32_t>& offsets) { | |
auto cnt_bytes = vint::to_bytes(offsets.size()); | |
iobuf buf; | |
buf.append(cnt_bytes.data(), cnt_bytes.size()); | |
for (auto o : offsets) { | |
auto bytes = vint::to_bytes(o); | |
buf.append(bytes.data(), bytes.size()); | |
} | |
return buf; | |
} |
I don't have strong feelings about code placement, I think leaving it in the record generate seems reasonable
1a07259
to
d3cc7ab
Compare
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/58542#019352d8-d1a4-477d-92c3-9735aa6242dc |
d3cc7ab
to
fa6789a
Compare
This data writer is for tests. It doesn't write to any files, however, it does go through the process of converting the ostream to parqueat.
fa6789a
to
a151b5e
Compare
This PR adds benchmarks for
record_multiplexer
along with the following that was needed to support that;record_generator
.Backports Required
Release Notes