Open an Arrow IPC file and determine the schema

[dmbates]

I am a beginner with Rust but have used Arrow in other languages. Please feel free to refer me to documentation that I may have missed. I am reading "the book" but it can be heavy going at times.

If I have a file "data/foo.arrow". How do I open the file with, I assume, an arrow::ipc::reader::FileReader and extract the schema and the lengths of individual columns.

My attempts look like

    let file = std::fs:File::open("./data/foo.arrow")? ;
    let rdr = arrow::ipc::reader::FileReader::try_new(file, ???);

but I don't know what the type of the second argument Option<Vec<usize>> means and what one would pass as an actual argument.

Of course it is possible that I have completely missed the point and am looking in the wrong place of the documentation entirely. A small snippet of code that I can use as a model would be much appreciated.

[dmbates]

In the end I read the documentation for the arrow2 crate which did show an example of using arrow2 to do what I wanted. And, for anyone who encounters this discussion in the future, as I understand it an Option<Vec<usize>> would be a Vec containing values like Some(1u64) or None. I found this confusing coming from the Julia programming language where the type would be written like Vector{Union{UInt64,Missing}}, which reverses the order of the Option concept and the Vec.

[dmbates]

I was wrong about the structure Option<Vec<usize>>. The structure I described, a vector whose elements can be like Some(1usize) or None would be a Vec<Option<usize>>. So I guess one could pass None as the second argument. I'm still not sure what it is used for.

[tustvold]

It is a list of column indices to project, so if you had columns a,b,c then a projection of Some(vec![1,2]) would just give you the data for columns b and c. This is an important optimisation to avoid reading data that a query doesn't need.

If you want all the columns, just pass in None. We could definitely document this better