GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface

[kou]

Rationale for this change

Statistics are useful for fast query processing. Many query engines
use statistics to optimize their query plan.

Apache Arrow format doesn't have statistics but other formats that can
be read as Apache Arrow data may have statistics. For example, Apache
Parquet C++ can read Apache Parquet file as Apache Arrow data and
Apache Parquet file may have statistics.

One of the Arrow C data interface use cases is the following:

1. Module A reads Apache Parquet file as Apache Arrow data
2. Module A passes the read Apache Arrow data to module B through the
   Arrow C data interface
3. Module B processes the passed Apache Arrow data

If module A can pass the statistics associated with the Apache Parquet
file to module B through the Arrow C data interface, module B can use
the statistics to optimize its query plan.

What changes are included in this PR?

Add the specification to pass statistics through the Arrow C data interface based on the discussion on the dev@ mailing list: https://lists.apache.org/thread/z0jz2bnv61j7c6lbk7lympdrs49f69cx

Are these changes tested?

Yes.

Are there any user-facing changes?

Yes.

• GitHub Issue: [Format] Passing column statistics through Arrow C data interface #38837

[kou]

@github-actions crossbow submit preview-docs

[github-actions]

⚠️ GitHub issue #38837 has been automatically assigned in GitHub to PR creator.

[kou]

I'm not a native English speaker. Wording suggestions are very welcome.

I'll add examples after I implement a convenient API to C++.

[github-actions]

Revision: 22336f4

Submitted crossbow builds: ursacomputing/crossbow @ actions-28c2a45b3d

Task	Status
preview-docs

[ianmcook]

What about the Arrow IPC format? Can you add a sentence here that explains why we do not recommend using this to pass statistics over Arrow IPC?

[kou]

Good point. This may fit the Arrow IPC format. (A producer sends data and statistics as 2 separated the Arrow IPC format data.)

But the Arrow IPC format can use more approaches. For example, the Arrow IPC format can have metadata for each record batch data: https://arrow.apache.org/docs/format/Columnar.html#encapsulated-message-format (The Arrow C data can't have metadata for ArrowArray.)
The Arrow IPC format can be used with other mechanisms such as Arrow Flight and ADBC.

So this may not be the best approach for the Arrow IPC format. We should discuss this use case with the Arrow IPC format separately.

I'll add something to here.

[ianmcook]

Maybe link to https://arrow.apache.org/adbc/current/format/specification.html#statistics

[kou]

Thanks. I should have done it.

[ianmcook]

@pdet please take a look and add comments if you have any, thanks!

[pdet]

The format and contents LGTM! I was just slightly confused for one second that the second mapping is the value items in the first mapping.

@Tmonster I think the proposed Statistics keys already cover the table statistics we use/produce. Could you double-check?

[kou]

@ianmcook Thanks for your suggestions! I've merged all of them!

[kou]

Good point. This may fit the Arrow IPC format. (A producer sends data and statistics as 2 separated the Arrow IPC format data.)

But the Arrow IPC format can use more approaches. For example, the Arrow IPC format can have metadata for each record batch data: https://arrow.apache.org/docs/format/Columnar.html#encapsulated-message-format (The Arrow C data can't have metadata for ArrowArray.)
The Arrow IPC format can be used with other mechanisms such as Arrow Flight and ADBC.

So this may not be the best approach for the Arrow IPC format. We should discuss this use case with the Arrow IPC format separately.

I'll add something to here.

[kou]

Thanks. I should have done it.

[kou]

They are the same as ADBC's one: https://github.com/apache/arrow-adbc/blob/05fa60d643c66b572d426ab28aa78fc52e9520e8/c/include/arrow-adbc/adbc.h#L524-L570

[kou]

I was just slightly confused for one second that the second mapping is the value items in the first mapping.

Ah, it makes sense. I'll improve it. Thanks.

[kou]

@github-actions crossbow submit preview-docs

[kou]

I think the C Data Interface framing came from the original use case (which I would like to not lose sight of); allowing engines like DuckDB to have a way to get statistics when given a C Data Interface stream, so that they can properly do query planning. Even if we reframe this as just a definition for the format of the statistics it might be good to mention the original use case so there's proper context, to help commenters who aren't familiar with query planning.

I've added the original DuckDB use case.

DuckDB may be able to get statistics without ArrowArrayStream (DuckDB may be able to call separate API to get statistics) because duckdb::TableFunction has table_function_cardinality_t cardinality and table_statistics_t statistics.
See also: https://github.com/duckdb/duckdb/blob/v1.1.3/src/function/table/arrow.cpp#L525-L527

I'll start a discussion on the mailing list tomorrow.
(I wanted to do it today...)

[kou]

The discussion thread: https://lists.apache.org/thread/b6chzlyn95rztoybs39b6olz907g12gj

[kou]

Based on the mailing list discussion, we don't limit this specification to only the Arrow C data interface.
We standardize only statistics schema.

I close this in favor of GH-45058.

[alamb]

Sorry for being (very) late to respond / review this proposal

[alamb]

We have had trouble in the arrow-rs implementation or other places where schema's contain names but they aren't standardized

For example, we have an embedded field name for Lists and some implementations use "item" and some "element" which causes spurious schema mismatch errors

Therefore I also recommend removing field names unless there is some good reason for doing so (it isn't clear to me from the text why there are arbitrary field namds)

[alamb]

Maybe we could be more explicitly about what the key means (it seems like it is the "statistics type" )?

[alamb]

Nit is I would find the term "statistics type" rather than "statistics key" easier to understand

[alamb]

I didn't understand this example. I thought the statistics were structs, so I would have expected the data to look something like this (perhaps we could give the "logical contents" and then the specific array encoding):

[ 
  // first struct element
  { 
    column: null, # record batch
     statistics: {
        "ARROW:row_count:exact": 0
     }
   },
  { 
    column: 0, # vendor_id
     statistics: {
        "ARROW:null_count:exact": 0,
        "ARROW:distinct_count:exact": 2,
        "ARROW:max_value:exact": 5,
        "ARROW:min_value:exact": 1,
     }
   },
...
]

I can help work out the example if people think this is a good idea

[alamb]

It is probably too late, but I figured I would point out another format we use for statistics in DataFusion is a columnar form rather than a nested structure

https://docs.rs/datafusion/latest/datafusion/physical_optimizer/pruning/trait.PruningStatistics.html

For example to represent this data in DataFusion it would be

vendor_id::null_count	vendor_id::min	vendor_id::max	...	passenger_count::max
0	1	5	...	2

The benefit of this encoding is that it can be used to quickly evaluate predicates on ranges (e.g. figure out if vendor_id = 6 could ever be true)

We use this format to read statistics from the parquet files (see ParquetStatisticsConverter to rule out row groups)

It does result in potentially very wide schemas however