Support serialize_packed?

[jteplitz]

It looks like this only supports using serialize right now. Is there a way to use serialize_packed? If not, I'm happy to help out with the implementation.

[dwrensha]

Right, async packed serialization is not implemented yet. It would probably be a fascinating project involving some rather hairy state machines, and I'd be thrilled if you wanted to attack it.

Note that the reference C++ implementation also does not have async packed serialization yet, so if we get this done first we'll have some bragging rights!

[jteplitz]

Haha, I guess I didn't realize what I might be getting into. I may have some time to take a look in a week or so. Mind giving me a quick overview of why it's so challenging?

[dwrensha]

I'm not sure that it's necessarily super challenging. It's just that nobody has needed it enough to bother yet.

The goal is to have versions of read() and write() that can be paused at arbitrary points in the byte streams. I can imagine two general strategies.

1. We could define enums that implement Future and explicitly represent the in-progress state. So basically every piece of control flow in the synchronous versions turns into data in some branch of these enums. This is how the unpacked async methods work in capnp-futures-rs today. (e.g. https://github.com/dwrensha/capnp-futures-rs/blob/27d39d18dd2bcc38c555a36ff3ec7899726ade2d/src/serialize.rs#L97-L125)
2. Or we could try the higher-level approach of using async control flow like and_then() and loop_fn(), and not writing a custom Future impl.

I have a vague worry that (2) might not work for some reason, but I might misremembering something.

[DemiMarie]

This will be nigh-trivial once Rust gets generators and corresponding async/await.

In the mean time, it is already possible to implement this, at the cost of an extra copy, by serializing to an in-memory buffer (synchronously) and then writing out the buffer.