Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: Update C/Pico/Cpp migration guide for serialization #73

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
161 changes: 88 additions & 73 deletions content/docs/migration_1.0/C++.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,80 +65,95 @@ try {

All returned and `std::move`'d-in objects are guaranteed to be left in an “empty” state in case of function call failure.

## Serialization and Deserialization
## Payload

In version 0.11.0 it was only possible to send `std::string`/ `const char*` or `std::vector<uint8_t>` / `uint8_t*` using the `BytesView` class:

```cpp
publisher.put("my_payload");
```

In 1.0.0, the `BytesView` class is gone and we introduced the `Bytes` object which represents a serialized payload.


In 1.0.0, the `BytesView` class is gone and we introduced the `Bytes` object which represents a (serialized) payload.
Similarly to 0.11.0 it can be used to store raw bytes or strings:

```cpp
void publish_data(const Publisher& publisher, const MyData& data) {
publisher.put(Bytes<MyCodec>::serialize(data));
void publish_string(const Publisher& publisher, const std::string& data) {
publisher.put(Bytes(data));
}

void publish_string_without_copy(const Publisher& publisher, std::string&& data) {
publisher.put(Bytes(data));
}

void receive_data(const Sample &sample) {
void receive_string(const Sample &sample) {
std::cout <<"Received: "
<< sample.get_payload().deserialize<MyData, MyCodec>()
<< sample.get_payload().as_string()
<< "\n";
};

void publish_bytes(const Publisher& publisher, const std::vector<uint8_t>& data) {
publisher.put(Bytes(data));
}

void publish_bytes_without_copy(const Publisher& publisher, std::vector<uint8_t>&& data) {
publisher.put(Bytes(data));
}

void receive_bytes(const Sample &sample) {
std::vector<uint8_t> = sample.get_payload().as_vector();
};
```

We added a default `ZenohCodec`, which provides default serialization / deserialization for common numerical types, strings, and containers:
Additionaly `zenoh::ext` namespace provides support for serialization/deserialziation of typed data to/into `Bytes`:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additionally


```cpp
// stream of bytes serialization
std::vector<uint8_t> data = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9};
Bytes b = Bytes::serialize(data);
assert(b.deserialize<std::vector<uint8_t>>() == data);

// arithmetic type serialization
// arithmetic types
double pi = 3.1415926;
Bytes b = Bytes::serialize(pi);
assert(b.deserialize<TYPE>() == pi);
Bytes b = ext::serialize(pi);
assert(ext::deserialize<doulbe>(b) == pi);

// Composite types serialization
std::vector<float> v = {0.1f, .2f, 0.3f};
auto b = Bytes::serialize(v);
assert(b.deserialize<decltype(v)>() == v);
// Composite types
std::vector<float> v = {0.1f, 0.2f, 0.3f};
b = ext::serialize(v);
assert(ext::deserialize<decltype(v)>(b) == v);

std::map<std::string, std::deque<double>> m = {
std::unordered_map<std::string, std::deque<double>> m = {
{"a", {0.5, 0.2}},
{"b", {-123.45, 0.4}},
{"abc", {3.1415926, -1.0} }
};

b = Bytes::serialize(m);
assert(b.deserialize<decltype(m2)>() == m);

// alternatively serialize via move
// the string keys will not be copied in this case, but rather move into Bytes
b_move = Bytes::serialize(std::move(m));
b = ext::serialize(m);
assert(ext::deserialize<decltype(m)>(b) == m);
```

Please note that this serialization functionality is only provided for prototyping and demonstration purposes and, as such, might be less efficient than custom-written serialization methods.

Users can easily define their own serialization/deserialization functions by providing a custom codec:
Users can easily define serialization/deserialization for their own custom types by using `ext::Serializer` and `ext::Deserializer` classes:

```cpp
MyType my_data(...);
MyCodec my_codec(...);
Bytes b = Bytes::serialize(my_data, my_codec);
// or Bytes::serialize<MyCodec>(my_data) if MyCodec is a stateless codec with an empty constructor
assert(b.deserialize<std::vector<MyType>>(my_codec) == my_data);
// or assert(b.deserialize<std::vector<MyType, MyCodec>>() == my_data); if MyCodec is a stateless with an empty constructor
```
struct CustomStruct {
std::vector<double> vd;
int32_t i;
std::string s;
};

For finer control on serialization / deserialization implementation `Bytes::Iterator`, `Bytes::Writer` and `Bytes::Reader` classes are also introduced.
// One needs to implement __zenoh_serialize_with_serializer in the same namespace as CustomStruct
void __zenoh_serialize_with_serializer(ext::Serializer& serializer, const CustomStruct& s) {
serializer.serialize(s.vd);
serializer.serialize(s.i);
serializer.serialize(s.s);
}

A working example with custom-defined serialization / deserialization can be found here:
void serialize_custom() {
CustomStruct s = {{0.1, 0.2, -1000.55}, 32, "test"};
Bytes b = ext::serialize(s);
CustomStruct s_out = ext::deserialize<CustomStruct>(b);
assert(s.vd == s_out.vd);
assert(s.i == s_out.i);
assert(s.s == s_out.s);
}
```

https://github.com/eclipse-zenoh/zenoh-cpp/blob/dev/1.0.0/examples/simple/universal/z_simple.cxx
For lower-level access to the `Bytes` content `Bytes::Reader`, `Bytes::Writer` and `Bytes::SliceIterator` classes can be used.

## Stream Handlers and Callbacks

Expand All @@ -155,9 +170,9 @@ session.get(keyexpr, "", std::move(send), opts);
Reply reply(nullptr);
// blocking
for (recv(reply); reply.check(); recv(reply)) {
auto sample = expect<Sample>(reply.get());
std::cout << "Received ('" << sample.get_keyexpr().as_string_view() << "' : '"
<< sample.get_payload().as_string_view() << "')\n";
auto sample = expect<Sample>(reply.get());
std::cout << "Received ('" << sample.get_keyexpr().as_string_view() << "' : '"
<< sample.get_payload().as_string() << "')\n";
}

// non-blocking
Expand All @@ -169,7 +184,7 @@ for (bool call_success = recv(reply); !call_success || reply.check(); call_succe
}
auto sample = expect<Sample>(reply.get());
std::cout << "\nReceived ('" << sample.get_keyexpr().as_string_view() << "' : '"
<< sample.get_payload().as_string_view() << "')";
<< sample.get_payload().as_string() << "')";
}
```

Expand All @@ -194,15 +209,15 @@ auto replies = session.get(
for (auto res = replies.recv(); std::has_alternative<Reply>(res); res = replies.recv()) {
const auto& sample = std::get<Reply>(res).get_ok();
std::cout << "Received ('" << sample.get_keyexpr().as_string_view() << "' : '"
<< sample.get_payload().deserialize<std::string>() << "')\n";
<< sample.get_payload().as_string() << "')\n";
}
// non-blocking
while (true) {
auto res = replies.try_recv();
if (std::has_alternative<Reply>(res)) {
const auto& sample = std::get<Reply>(res).get_ok();
std::cout << "Received ('" << sample.get_keyexpr().as_string_view() << "' : '"
<< sample.get_payload().deserialize<std::string>() << "')\n";
<< sample.get_payload().as_string() << "')\n";
} else if (std::get<channels::RecvError>(res) == channels::RecvError::Z_NODATA) {
// try_recv is non-blocking call, so may fail to return a reply if the Fifo buffer is empty
std::cout << ".";
Expand All @@ -223,7 +238,7 @@ The same works for `Subscriber` and `Queryable`:
auto data_callback = [](const Sample &sample) {
std::cout << ">> [Subscriber] Received ('"
<< sample.get_keyexpr().as_string_view()
<< "' : '" << sample.get_payload().deserialize<std::string>()
<< "' : '" << sample.get_payload().as_string()
<< "')\n";
};

Expand All @@ -241,19 +256,19 @@ auto subscriber = session.declare_subscriber(keyexpr, channels::FifoChannel(16))
const auto& messages = subscriber.handler();
//blocking
for (auto res = messages.recv(); std::has_alternative<Sample>(res); res = messages.recv()) {
// recv will block until there is at least one sample in the Fifo buffer
// it will return an empty sample and alive=false once subscriber gets disconnected
const Sample& sample = std::get<Sample>(res);
std::cout << "Received ('" << sample.get_keyexpr().as_string_view() << "' : '"
<< sample.get_payload().deserialize<std::string>() << "')\n";
// recv will block until there is at least one sample in the Fifo buffer
// it will return an empty sample and alive=false once subscriber gets disconnected
const Sample& sample = std::get<Sample>(res);
std::cout << "Received ('" << sample.get_keyexpr().as_string_view() << "' : '"
<< sample.get_payload().as_string() << "')\n";
}
// non-blocking
while (true) {
auto res = messages.try_recv();
if (std::has_alternative<Sample>(res)) {
const auto& sample = std::get<Sample>(res);
std::cout << "Received ('" << sample.get_keyexpr().as_string_view() << "' : '"
<< sample.get_payload().deserialize<std::string>() << "')\n";
<< sample.get_payload().as_string() << "')\n";
} else if (std::get<channels::RecvError>(res) == channels::RecvError::Z_NODATA) {
// try_recv is non-blocking call, so may fail to return a sample if the Fifo buffer is empty
std::cout << ".";
Expand Down Expand Up @@ -281,24 +296,24 @@ options.set_attachment(attachment_map);
pub.put(s, options);

// subscriber callback function receiving message with attachment
data_handler(const Sample &sample) {
std::cout << ">> [Subscriber] Received " ('"
void data_handler(const Sample &sample) {
std::cout << ">> [Subscriber] Received \" ('"
<< sample.get_keyexpr().as_string_view()
<< "' : '"
<< sample.get_payload().as_string_view()
<< "')\n";
if (sample.get_attachment().check()) {
// reads full attachment
sample.get_attachment().iterate([](const BytesView &key, const BytesView &value) -> bool {
std::cout << " attachment: " << key.as_string_view() << ": '" << value.as_string_view() << "'\n";
return true;
});

// or read particular attachment item
auto index = sample.get_attachment().get("index");
if (index != "") {
std::cout << " message number: " << index.as_string_view() << std::endl;
}
// reads full attachment
sample.get_attachment().iterate([](const BytesView &key, const BytesView &value) -> bool {
std::cout << " attachment: " << key.as_string_view() << ": '" << value.as_string_view() << "'\n";
return true;
});

// or read particular attachment item
auto index = sample.get_attachment().get("index");
if (index != "") {
std::cout << " message number: " << index.as_string_view() << std::endl;
}
}
};
```
Expand All @@ -317,8 +332,8 @@ std::unordered_map<std::string, std::string> attachment_map = {
{"index", "0"}
};
pub.put(
Bytes::serialize("my_payload"),
{.encoding = Encoding("text/plain"), .attachment = std::move(attachment_map)}
Bytes("my_payload"),
{.encoding = Encoding("text/plain"), .attachment = ext::serialize(attachment_map)}
);


Expand All @@ -327,13 +342,13 @@ void data_handler(const Sample &sample) {
std::cout << ">> [Subscriber] Received ('"
<< sample.get_keyexpr().as_string_view()
<< "' : '"
<< sample.get_payload().deserialize<std::string>()
<< sample.get_payload().as_string()
<< "')\n";
auto attachment = sample.get_attachment();
if (!attachment.has_value()) return;
// we expect attachment in the form of key-value pairs
auto attachment_deserialized = attachment->get().deserialize<std::unordered_map<std::string, std::string>>();
for (auto&& [key, value]: attachment) {
auto attachment_deserialized = ext::deserialize<std::unordered_map<std::string, std::string>>(attachment->get());
for (auto&& [key, value]: attachment_deserialized) {
std::cout << " attachment: " << key << ": '" << value << "'\n";
}
};
Expand All @@ -359,5 +374,5 @@ session.get(keyexpr, "", {on_reply, on_done}, opts);
In 1.0.0:

```cpp
session.get(keyexpr, "", on_reply, on_done, {.target = Z_QUERY_TARGET_ALL, .payload = Bytes::serialize(value)});
session.get(keyexpr, "", on_reply, on_done, {.target = Z_QUERY_TARGET_ALL, .payload = ext::serialize(value)});
```
Loading