Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[meta] Automatic opaque serialization #1409

Open
wants to merge 13 commits into
base: master
Choose a base branch
from

Conversation

jpeletier
Copy link
Contributor

@jpeletier jpeletier commented Oct 21, 2024

Note: must merge #1405 first

This PR builds on the previous two to add automatic opaque serialization for those opaques defining the appropriate get_* callbacks.

When defining serialization for an opaque, you have to define a serialize callback. If this callback is not defined, serialization will fail.

With this change, when a serialize callback is not defined, the meta module will try to find a way to serialize the opaque if the opaque has defined another callback that can be used.

This is useful if you have already defined a get_ callback that does the work and don't want to define a serializer.

If you do define a serializer, the serializer takes precedence and it is used as before.

For example:

  struct IntOpaque {
    int value;
  };

// BEFORE:
  ecs.component<IntOpaque>()
      .opaque(flecs::I32)
      .serialize([](const flecs::serializer *ser, const IntOpaque *data) -> int { 
        ser->value(data->value);
        return 0; 
    });

// Now: 
  ecs.component<IntOpaque>()
    .opaque(flecs::I32)
    .get_int([](const IntOpaque *data) -> int64_t { 
      return data->value; 
    });

It is still recommended to write a serializer if what you are doing with the type is to fully serialize it to JSON or something similar.

If, however, you are using reflection to access portions of opaque data randomly, the serializer could be slower/inefficient because it dumps the whole content, whereas the get_ callbacks provide the opaque with a way to go straight to the requested data.

For example, the following will now work without a serializer:

  using IntVector = std::vector<int>;

  ecs.component<IntVector>()
      .opaque(ecs.vector<int>())
      .get_element([](const IntVector *src, size_t i) -> const void * { return src->data() + i; })
      .count([](const IntVector *src) -> size_t { return src->size(); });

  IntVector vec = {1, 2, 3};

  // work with IntVector accessing a random element:
  flecs::cursor cur(ecs, ecs.component<IntVector>(), &vec);
  cur.push();
  cur.elem(2);
  std::cout << "vec[2] = " 
            << cur.get_int()  // will call opaque's get_element()
            << std::endl;

  // Can also serialize the whole vector to JSON, without having to write a serializer,
  // at a small performance cost:
  std::cout << "JSON: " << ecs.to_json(&vec) << std::endl;

  // output:
  // vec[2] = 3
  // JSON: [1, 2, 3]

@SanderMertens
Copy link
Owner

Similar to my comment on the other PR- I'm not sure how this works if you have a type with multiple members. I don't think the edge case of a type with a single member warrants extending the interface.

@jpeletier
Copy link
Contributor Author

jpeletier commented Dec 1, 2024

I think perhaps my example here is misunderstood. You should not be looking at the fact that the opaque implementation happens to be a struct with just one member, because it is an opaque. It could be anything. I put a struct to make it a simple example. This opaque is saying, as per the example, that wants to be considered as a flecs::I32.

The same way, the std::string example in Examples defines it as an opaque that wants to be considered a flecs::String (a primitive type in Flecs). To complete the std::string example, one would need the get_string handler, which is introduced in #1403, so the opaque can also be read, not only written. Yes, serialize can be used for reading but it conveys, from my point of view, a different purpose which is to output the whole content, and it is inefficient if the as_type is a struct and you just want to read one of the members.

Think of cursors. You should be able to read and write with them, not only write. When the cursor is pointing to an opaque, you should be able to read and write to it too as if it wasn't an opaque, because for what it is worth, that opaque has to behave as the as_type it committed to behave as.

When a type has multiple members, you mean the as_type is got multiple members? in that case #1405 deals with it via get_member and get_element. These are the counterpart of ensure_member and ensure_element, but pledge they won't modify the underlying instance and only require (and receive) a const pointer.

This PR is not as important as the other two; I would consider it optional. I wrote this one because I found interesting that using reflection itself and the new get_ methods it was possible to infer a serializer automatically.

Again, the key background to properly understand the context of this set of contributions is here:
https://discord.com/channels/633826290415435777/1295740423293108298

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants