-
Notifications
You must be signed in to change notification settings - Fork 28
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Replace CAF deserializers with Broker decoders #424
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM Dominik, thanks. I've run Zeek's testsuites with it and they ran fine. I tried to eyeball performance via broker-throughput
and it runs fine on this branch, but for master the server is currently throwing lots of deserialization_failed("failed to parse data")
errors — I tried
$ ./broker-throughput --verbose --server :8888
and
$ ./broker-throughput --verbose -t 3 -r 1000 localhost:8888
... I can't recall whether this is known — do you know? We'll see performance numbers also via Zeek's CI setup when we bump Broker.
I can merge this too, if you're out of time.
libbroker/broker/format/bin.test.cc
Outdated
CHECK_EQ(cpy.sender, cmd.sender); | ||
CHECK_EQ(cpy.receiver, cmd.receiver); | ||
if (CHECK(std::holds_alternative<cumulative_ack_command>(cpy.content))) { | ||
CHECK_EQ(std::get<cumulative_ack_command>(cpy.content).seq, 42); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a quick suggestion here since GCC 14.2 warns:
CHECK_EQ(std::get<cumulative_ack_command>(cpy.content).seq, 42); | |
CHECK_EQ(std::get<cumulative_ack_command>(cpy.content).seq, sequence_number_type{42}); |
In file included from /home/christian/devel/zeek/zeek/auxil/broker/caf/libcaf_test/caf/test/dsl.hpp:15,
from /home/christian/devel/zeek/zeek/auxil/broker/caf/libcaf_test/caf/test/bdd_dsl.hpp:3,
from /home/christian/devel/zeek/zeek/auxil/broker/libbroker/broker/broker-test.test.hh:7,
from /home/christian/devel/zeek/zeek/auxil/broker/libbroker/broker/format/bin.test.cc:3:
/home/christian/devel/zeek/zeek/auxil/broker/libbroker/broker/format/bin.test.cc: In instantiation of ‘{anonymous}::test396::run_test_impl()::<lambda(const auto:264&, const auto:265&)> [with auto:264 = long unsigned int; auto:265 = int]’:
/home/christian/devel/zeek/zeek/auxil/broker/libbroker/broker/format/bin.test.cc:434:5: required from here
596 | [](const auto& x_val, const auto& y_val) { \
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
597 | return ::caf::test::detail::check_bin( \
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
598 | x_val == y_val, __FILE__, __LINE__, #x_expr " == " #y_expr, \
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
599 | caf::detail::stringification_inspector::render(x_val), \
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
600 | caf::detail::stringification_inspector::render(y_val)); \
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
601 | }(x_expr, y_expr)
| ~^~~~~~~~~~~~~~~~
/home/christian/devel/zeek/zeek/auxil/broker/caf/libcaf_test/caf/test/unit_test.hpp:598:13: warning: comparison of integer expressions of different signedness: ‘const long unsigned int’ and ‘const int’ [-Wsign-compare]
598 | x_val == y_val, __FILE__, __LINE__, #x_expr " == " #y_expr, \
| ~~~~~~^~~~~~~~
/home/christian/devel/zeek/zeek/auxil/broker/caf/libcaf_test/caf/test/bdd_dsl.hpp:41:28: note: in expansion of macro ‘CAF_CHECK_EQUAL’
41 | #define CHECK_EQ(lhs, rhs) CAF_CHECK_EQUAL(lhs, rhs)
| ^~~~~~~~~~~~~~~
/home/christian/devel/zeek/zeek/auxil/broker/libbroker/broker/format/bin.test.cc:434:5: note: in expansion of macro ‘CHECK_EQ’
434 | CHECK_EQ(std::get<cumulative_ack_command>(cpy.content).seq, 42);
| ^~~~~~~~
libbroker/broker/format/bin.hh
Outdated
auto proto = uint8_t{0}; | ||
if (!read(pos, end, proto)) | ||
return {false, pos}; | ||
if (proto > 3) // 3 is the highest protocol number we support (ICMP). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Total nit, but if we know this is about ICMP we could just say broker::port::protocol::icmp
.
Hm, that news to me. I don't think I'll be able to dig into that today. I'd like to look into that before we merge the PR, even if it means letting this sit for a while until I'm back. Would you be OK with that? |
Yeah no problem, I doubt we'll make other changes that would invalidate your PR. Have a great trip! |
Ok, so there's been two things. First, type 3 in |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks Dominik — apologies as always for taking a while. Merging now.
The new decoding logic lacked a consistency check that allows sets and tables to verify that items/keys are unique during decoding.
c058909
to
bf9ad0f
Compare
Just rebasing here since time has passed. I've tested Zeek's testsuite with this and it's all looking good. |
This pulls in zeek/broker#424, zeek/broker#430, zeek/broker#431.
When implementing the new networking layer and
broker::variant
, we have implemented encoders for the binary serialization format. However, the decoding has only been implemented forbroker::variant
.This set of changes implements the decoding as well, meaning that Broker is now fully "self-hosted" and no longer relies on the binary serialization primitives in CAF which make no guarantees on the stability of the format.
There are two "flavors" for the decoding:
decode
function with a SAX-like interface. This function is tailored towards Broker's type system and can be used for decodingbroker::variant
andbroker::data
.decoder
class that is designed to work with theinspect
API. We use this as a drop-in replacement forcaf::deserializer
, e.g., when decoding store commands.Closes #161.