Describe the bug

The format guarantees that each IPC file embeds a valid IPC stream in order to allow readers to ignore the Footer, skip the file's leading magic, and reuse a stream reader.

However, when writing IPC files arrow-rs aligns the encapsulated flatbuffers Messages to 64 byte boundaries instead of 8 bytes. This can leave gaps of padding bytes between the Messages which a stream reader would not know to skip.

To Reproduce

apache/arrow#43834 adds validation of the embedded stream to arrow-c++'s arrow-json-integration-test. Running this against IPC files written by arrow-rs raises an error

$ arrow-json-integration-test -arrow datetime.arrow -json datetime.json -integration -mode VALIDATE
Error message: Invalid: Tried reading schema message, was null or length 0

Expected behavior

A stream reader should be able to read an IPC file by skipping the first 8 bytes.

Additional context

This was originally introduced in eddef43 which added alignment to address new requirements around i128. However the alignment should not be applied to flatbuffers Messages; apart from the above issue I think there's no SIMD or other advantage to aligning those to more than 8 bytes. Body buffers can of course still be padded and aligned freely.

This was discovered while adding IPC file reading to nanoarrow; we were trying to defer reading Footers for a follow up and discovered that the go, rust, and javascript implementations don't embed a valid stream. Most readers have not noticed because offsets and schemas are more efficiently read from a Footer, and once acquired obviate sequential stream-style reading.