Async Arrow Reader

[tustvold]

This is largely a placeholder and would need further fleshing out if someone wishes to work on this

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Current we only support reading IPC files from a synchronous buffer. This limits the ability to read parts of files.

Describe the solution you'd like

I high-level implementation inspired by parquet might look like

• A low-level mechanism to decode a footer from bytes - https://docs.rs/parquet/latest/parquet/file/footer/index.html
• A higher-level mechanism to fetch and decode a footer from an async source - https://docs.rs/parquet/latest/parquet/arrow/async_reader/fn.fetch_parquet_metadata.html
• An API to decode a RecordBatch from a data block, with an optional projection

This would then naturally lend itself to being wrapped up in an async interface similar to ParquetRecordBatchStream

Describe alternatives you've considered

Additional context

[alamb]

There is related commentary on https://github.com/apache/arrow-datafusion/pull/7962/files#r1379219105 about how to look at the footer of arrow files to get metadata, etc

[Jefffrey]

FYI a duplicate/related issue exists: #1207