Natively support Memory<byte>

[verdie-g]

There are many PRs open aiming to extend the API to allow low allocation produce but they all have some caveats:

• Binary Producer with Span<T> #1603 tries to support Span but given the constraints of that type, it means creating a whole new set of API.
• Improve serialization memory management #1725 suggests to use IBufferWriter instead, which unlike byte[] or Span, supports non-contiguous memory, kind of like Stream but it enables a lof of ways to reduce allocations. I don't think IBufferWriter is ideal here because in the end, a continuous memory buffer is expected for both the key and value. Also, the PR is introducing several breaking changes.
• Add a new type of Serializer for supporting zero allocation scenarios #2177, Add ArraySegment<byte> as native supported type #2219, and Support using array segments for serializers #1782 suggests adding support for ArraySegment propose to support ArraySegment which does not break the API and fixes the original problem. Though it would prevent the use of Memory<byte> which is the preferred type when dealing with contiguous memory when Span<byte> can't be used. This type is for example used in Stream.WriteAsync, Socket.SendAsync, and also in MemoryPool.Rent.

Instead of adding a third type of serializer (after ISerializer and IAsyncSerializer), this PR makes a special case for Memory<byte> (and ReadOnlyMemory<byte>) to by pass the serializers and directly use Message.Value or Message.Key.

Closes #1238.

[cla-assistant]

All committers have signed the CLA.

[cla-assistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[bmesetovic]

There seems to be no progress on either of the several approaches to reduce allocations on the publishing side of Kafka. All of them are stuck at the review phase.

We are affected by the issue as well and would really like to see how we can push this topic so that it finally gets the attention it deserves.

@verdie-g Did you end up forking the project and adding your changes?

[verdie-g]

We used a fork with #2219 just to confirm the important performance benefits but we were not ready to maintain a fork.

We are still waiting for a review on that PR.

[confluent-cla-assistant]

🎉 All Contributor License Agreements have been signed. Ready to merge.
✅ verdie-g
_{Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.}