Question and API proposal regarding DotNext.IO

[thenameless314159]

Hello, first of all I would first like to express my sincere appreciation for this amazing library !

Question

For the sake of my personal understanding, I am curious about the motivations behind your choice to design a non-generic SequenceReader, rather than enhancing the existing System.Buffers.SequenceReader (aside from the async method overloads).

API proposal

Furthermore, I would like to make a proposal for new APIs and changes to existing ones for pipe-related scenarios which could be beneficial for users willing to directly read from a sequence that was obtained from a PipeReader via a SequenceReader<byte>.

I have largely completed the necessary work and testing within a private repository of my own, achieving close to 90% coverage. If required, I can ensure this coverage reaches 100%. It would be my pleasure to introduce these modifications to DotNext via a pull request.

Background and motivation

The IBinaryFormattable<TSelf>, from my perspective, is somewhat lacking in terms of extensibility. It could be beneficial to divide it into separate read-only and write-only interfaces. Additionally, new interfaces could be introduced for binary readable types that expose settable members, which would not necessarily require the implementation of a static abstract interface method.

In relation to pipe-related scenarios, users might desire to implement an IBinaryFormattable<TSelf>, and as such, may not be inclined to implement parsing logic based on a SpanReader<byte> to process input data from a ReadOnlySequence<byte>. Instead, they could directly depend on a SequenceReader<byte>, which would eliminate the need to copy the input data into a single stack-allocated or rented span.

I believe that implementing these modifications would pave the way for APIs that are prepared for source-generator integration which is a feature I intend to develop in the future.

Proposal

Consequently, my proposal could take several directions depending on the scenarios we aim to address:

If we primarily want to address pipe-related scenarios:

• Divide IBinaryFormattable<TSelf> into three interfaces:
  • IBinaryWritable with a public void Serialize(ref SpanWriter<T> output) and potentially an instance int GetSizeOf() method, or another interface enabling the addition of extension methods to write into an IBufferWriter<byte> or various memory owners.
  • IBinaryReadable with a public void Deserialize(ref SpanReader<T> input) for objects with settable members.
  • IBinaryReadable<TSelf> with a static abstract Read method for objects with init-only members.
• Introduce IBinaryFormattable : IBinaryWritable, IBinaryReadable for objects exposing setting members.
• Refactor IBinaryFormattable<TSelf> to IBinaryFormattable<TSelf> : IBinaryReadable<TSelf>, IBinaryWritable.
• Introduce IBufferReadable and IBufferReadable<TSelf> that mirror the IBinaryReadable interfaces but use a SequenceReader<byte> instead.
• Provide IBinaryFormattable-like interfaces for the sequence-based IBufferReadable interfaces.

If we aim to cover a broader range of scenarios (e.g., spans of chars):

• We would include all of the above, except the new interfaces would then inherit from additional ISpanReadable<T>, ISpanReadable<TSelf, T> and ISpanWritable<T> interfaces, where T represents the span element type.

Source code

I have made some of the suggested APIs available in a gist, which can be found here: https://gist.github.com/thenameless314159/a2a6487ce19c0da24d08a1cf328ddc8e

[sakno]

I am curious about the motivations behind your choice to design a non-generic SequenceReader, rather than enhancing the existing System.Buffers.SequenceReader (aside from the async method overloads).

DotNext.IO.SequenceReader is an implementation of IAsyncBinaryReader for unstructured block of data loaded to the memory (as contiguous or sparse memory block). Other implementations exist for Stream and PipeReader. This design allows to abstract IDataTransferObject from the actual source of raw data. Another significant reason: System.Buffers.SequenceReader is ref-type, while DotNext.IO.SequenceReader is not. This fact allows to use DotNext.IO.SequenceReader in async methods. But it sounds reasonable to have extension methods for System.Buffers.SequenceReader just like StreamExtensions doing the same for Stream.

In relation to pipe-related scenarios, users might desire to implement an IBinaryFormattable, and as such, may not be inclined to implement parsing logic based on a SpanReader to process input data from a ReadOnlySequence

IBinaryFormattable is a lightweight counterpart of ISerializable<T> interface that derives from IDataTransferObject. The last one uses IAsyncBinaryReader and IAsyncBinaryWriter interfaces as abstractions over pipes, streams, contiguous buffers, sparse buffers. Therefore, it is better choice than IBinaryFormattable interface. IBinaryFormattable is designed to abstract some contiguous and relatively small binary objects, such as protocol frames, file headers, etc. You can find a few types in DotNext implementing this interface: ClusterMemberId, LogEntryMetadata.

for objects with settable members.

It is not mandatory to have settable members on types implementing IBinaryFormattable. This fact provides a room for some low-level optimizations, especially in DotNext itself, e.g.

dotNext/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/PersistentState.Internal.cs

Lines 115 to 138 in 39b7492

	public void Format(Span<byte> output)
	{
	Debug.Assert(output.Length >= Size);
	// fast path without any overhead for LE byte order
	ref var ptr = ref MemoryMarshal.GetReference(output);
	if (!BitConverter.IsLittleEndian)
	{
	// BE case
	FormatSlow(output);
	}
	else if (IntPtr.Size is sizeof(long))
	{
	// 64-bit LE case, the pointer is always aligned to 8 bytes
	Debug.Assert(Intrinsics.AddressOf(in ptr) % IntPtr.Size is 0);
	Unsafe.As<byte, LogEntryMetadata>(ref ptr) = this;
	}
	else
	{
	// 32-bit LE case, the pointer may not be aligned to 8 bytes
	Unsafe.WriteUnaligned<LogEntryMetadata>(ref ptr, this);
	}
	}

These optimizations are possible only of the object contents is in contiguous memory (while ReadOnlySequence may represent non-contiguous buffer).

implement parsing logic based on a SpanReader to process input data from a ReadOnlySequence.

This reduces a surface for potential reuse. For instance, stackalloc'ed memory cannot be represented as ReadOnlySequence<byte> (without unsafe and allocating workarounds). Same true for unmanaged memory pointer. In the same time, the opposite situation is true. It is possible to copy T.Size portion of necessary data from ReadOnlySequence<byte> to contiguous buffer (pooled array, stack memory, unmanaged memory, or whatever you want) and then create SpanReader for that Span<byte>, then parse IBinaryFormattable.

[thenameless314159]

I'd like to apologize for initially posting my proposal in the wrong location, and thank you for your detailed and comprehensive response. Your insights have significantly enhanced my understanding of the logic behind your design choices and helped clarify why my proposal may not be in alignment with your current design.

However, I find that I am still grappling with some aspects of binary transformations, particularly within certain contexts. For instance, I see that the IBinaryFormattable interface presupposes a static length. How would this accommodate variable data structures such as 7 bit encoded integers or variable length arrays? I suppose it doesn't and the IDataTransferObject interface is what should be implemented instead with the instance member long? Length for size computation ?

Additionally, for network-related scenarios, like a TCP protocol with a prefixed length and ID followed by a payload, would it be appropriate to consider the message headers as IBufferFormattable and implement the typed messages as IDataTransferObjects ? This approach seems logical to me, but I would appreciate your guidance and recommendations.

[sakno]

IBinaryFormattable interface presupposes a static length. How would this accommodate variable data structures such as 7 bit encoded integers or variable length arrays? I suppose it doesn't

Your suspicions are correct. The interface is not designed for that. Typically, it should be implemented by structs where the size of the data is statically known. In other words, the implementing type provides structured representation of some contiguous sequence of bytes. You can checkout the whole project and select Find All Implementations in VS or VSCode to find all internal implementations across DotNext, e.g. https://github.com/dotnet/dotNext/blob/39b749229721eecfb47179329434b41996cdf587/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/TransportServices/LogEntryMetadata.cs, which is a fixed-length portion of Raft wire format for TCP/UDP.

IDataTransferObject interface is what should be implemented instead with the instance member long? Length property of size computation?

Correct, but this property is not mandatory and may return null. It is used by underlying library infrastructure as optimization hint. If length is known, good, library can perform some optimizations (e.g. pre-allocate file on disk using PreallocationSize or informs another side of network socket about expected size of the payload, initialize initial capacity of the buffers, etc.).

Both interfaces are used by Raft wire protocol shipped with the library. When transforming IDataTransferObject, you can utilize some special methods exposed by IAsyncBinaryReader and perform synchronous decoding if underlying source of raw data is fully loaded to the memory. For instance, bool IAsyncBinaryReader.TryGetSequence(out ReadOnlySequence<byte>) gives you this opportunity. In this case, you can use System.Buffers.SequenceReader<T> for fast decoding. If not, use fallback code path using await async invocations.