From 5308f9fe96458961a5aad416ab49f8e566a34c2a Mon Sep 17 00:00:00 2001 From: Sutou Kouhei Date: Wed, 18 Dec 2024 15:43:42 +0900 Subject: [PATCH 01/10] GH-43553: [Format] Add the specification for statistics schema --- docs/source/format/Columnar.rst | 1 + docs/source/format/StatisticsSchema.rst | 600 ++++++++++++++++++++++++ docs/source/format/index.rst | 1 + 3 files changed, 602 insertions(+) create mode 100644 docs/source/format/StatisticsSchema.rst diff --git a/docs/source/format/Columnar.rst b/docs/source/format/Columnar.rst index 33c937ea34820..9ef6a933528f8 100644 --- a/docs/source/format/Columnar.rst +++ b/docs/source/format/Columnar.rst @@ -1619,6 +1619,7 @@ example as above, an alternate encoding could be: :: 0 EOS +.. _format_metadata: Custom Application Metadata --------------------------- diff --git a/docs/source/format/StatisticsSchema.rst b/docs/source/format/StatisticsSchema.rst new file mode 100644 index 0000000000000..c595c10420fa7 --- /dev/null +++ b/docs/source/format/StatisticsSchema.rst @@ -0,0 +1,600 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. _statistics-schema: + +================= +Statistics schema +================= + +.. warning:: This specification should be considered experimental. + +Rationale +========= + +Statistics are useful for fast query processing. Many query engines +use statistics to optimize their query plan. + +Apache Arrow format doesn't have statistics but other formats that can +be read as Apache Arrow data may have statistics. For example, the +Apache Parquet C++ implementation can read an Apache Parquet file as +Apache Arrow data and the Apache Parquet file may have statistics. + +We standardize how to represent statistics as an Apache Arrow array +for easy to exchange. + +Use case +-------- + +One of :ref:`c-stream-interface` use cases is the following: + +1. Module A reads Apache Parquet file as Apache Arrow data. +2. Module A passes the read Apache Arrow data to module B through the + Arrow C stream interface. +3. Module B processes the passed Apache Arrow data. + +If module A can pass the statistics associated with the Apache Parquet +file to module B, module B can use the statistics to optimize its +query plan. + +For example, DuckDB uses this approach but DuckDB couldn't use +statistics because there wasn't the standardized way to represent +statistics for an Apache Arrow data. + +.. seealso:: + + `duckdb::ArrowTableFunction::ArrowScanBind() in DuckDB 1.1.3 + `_ + +Goals +----- + +* Establish a standard way to represent statistics as an Apache Arrow + array. + +Non-goals +--------- + +* Establish a standard way to pass an Apache Arrow array that + represents statistics. +* Establish a standard way to embed statistics to an Apache Arrow + array. + +Schema +====== + +This specification provides only the schema for statistics. This is +the canonical schema to represent statistics about an Apache Arrow +dataset as Apache Arrow data. + +Here is the outline of the schema for statistics:: + + struct< + column: int32, + statistics: map< + key: dictionary< + indices: int32, + dictionary: utf8 + >, + items: dense_union<...all needed types...> + > + > + +Here is the details of top-level ``struct``: + +.. list-table:: + :header-rows: 1 + + * - Name + - Data type + - Nullable + - Notes + * - ``column`` + - ``int32`` + - ``true`` + - The zero-based column index, or null if the statistics + describe the whole table or record batch. + + The column index is computed as the same rule used by + :ref:`ipc-recordbatch-message`. + * - ``statistics`` + - ``map`` + - ``false`` + - Statistics for the target column, table or record batch. See + the separate table below for details. + +Here is the details of the ``map`` of the ``statistics``: + +.. list-table:: + :header-rows: 1 + + * - Key or items + - Data type + - Nullable + - Notes + * - key + - ``dictionary`` + - ``false`` + - Statistics key is string. Dictionary is used for + efficiency. Different keys are assigned for exact value and + approximate value. Also see the separate description below for + statistics key. + * - items + - ``dense_union`` + - ``false`` + - Statistics value is dense union. It has at least all needed + types based on statistics kinds in the keys. For example, you + need at least ``int64`` and ``float64`` types when you have a + ``int64`` distinct count statistic and a ``float64`` average + byte width statistic. Also see the separate description below + for statistics key. + + We don't standardize field names for the dense union because we + can access to proper field by type code not name. So we can use + any valid name for fields. + + TODO: Should we standardize field names? + +.. _statistics-schema-key: + +Statistics key +-------------- + +Statistics key is string. ``dictionary`` is used for +efficiency. + +We assign different statistics keys for individual statistics instead +of using flags. For example, we assign different statistics keys for +exact value and approximate value. + +The colon symbol ``:`` is to be used as a namespace separator like +:ref:`format_metadata`. It can be used multiple times in a key. + +The ``ARROW`` pattern is a reserved namespace for pre-defined +statistics keys. User-defined statistics must not use it. +For example, you can use your product name as namespace +such as ``MY_PRODUCT:my_statistics:exact``. + +Here are pre-defined statistics keys: + +.. list-table:: + :header-rows: 1 + + * - Key + - Data type + - Notes + * - ``ARROW:average_byte_width:exact`` + - ``float64``: TODO: Should we use ``int64`` instead? + - The average size in bytes of a row in the target + column. (exact) + * - ``ARROW:average_byte_width:approximate`` + - ``float64``: TODO: Should we use ``int64`` instead? + - The average size in bytes of a row in the target + column. (approximate) + * - ``ARROW:distinct_count:exact`` + - ``int64`` + - The number of distinct values in the target column. (exact) + * - ``ARROW:distinct_count:approximate`` + - ``float64`` + - The number of distinct values in the target + column. (approximate) + * - ``ARROW:max_byte_width:exact`` + - ``int64`` + - The maximum size in bytes of a row in the target + column. (exact) + * - ``ARROW:max_byte_width:approximate`` + - ``float64``: TODO: Should we use ``int64`` instead? + - The maximum size in bytes of a row in the target + column. (approximate) + * - ``ARROW:max_value:exact`` + - Target dependent + - The maximum value in the target column. (exact) + * - ``ARROW:max_value:approximate`` + - Target dependent + - The maximum value in the target column. (approximate) + * - ``ARROW:min_value:exact`` + - Target dependent + - The minimum value in the target column. (exact) + * - ``ARROW:min_value:approximate`` + - Target dependent + - The minimum value in the target column. (approximate) + * - ``ARROW:null_count:exact`` + - ``int64`` + - The number of nulls in the target column. (exact) + * - ``ARROW:null_count:approximate`` + - ``float64`` + - The number of nulls in the target column. (approximate) + * - ``ARROW:row_count:exact`` + - ``int64`` + - The number of rows in the target table or record batch. (exact) + * - ``ARROW:row_count:approximate`` + - ``float64`` + - The number of rows in the target table or record + batch. (approximate) + +If you find a missing statistics key that is usable for multiple +systems, please propose it on the `Apache Arrow development +mailing-list `__. + +.. _statistics-schema-examples: + +Examples +======== + +Here are some examples to help you understand. + +Simple record batch +------------------- + +Schema:: + + vendor_id: int32 + passenger_count: int64 + +Data:: + + vendor_id: [5, 1, 5, 1, 5] + passenger_count: [1, 1, 2, 0, null] + +Statistics: + +.. list-table:: + :header-rows: 1 + + * - Target + - Key + - Value + * - Record batch + - The number of rows + - ``5`` + * - ``vendor_id`` + - The number of nulls + - ``0`` + * - ``vendor_id`` + - The number of distinct values + - ``2`` + * - ``vendor_id`` + - The max value + - ``5`` + * - ``vendor_id`` + - The min value + - ``1`` + * - ``passenger_count`` + - The number of nulls + - ``1`` + * - ``passenger_count`` + - The number of distinct values + - ``3`` + * - ``passenger_count`` + - The max value + - ``2`` + * - ``passenger_count`` + - The min value + - ``0`` + +Column indexes: + +.. list-table:: + :header-rows: 1 + + * - Index + - Target + * - ``0`` + - ``vendor_id`` + * - ``1`` + - ``passenger_count`` + +Statistics schema:: + + struct< + column: int32, + statistics: map< + key: dictionary< + indices: int32, + dictionary: utf8 + >, + items: dense_union<0: int64> + > + > + +Statistics array:: + + column: [ + null, # record batch + 0, # vendor_id + 0, # vendor_id + 0, # vendor_id + 0, # vendor_id + 1, # passenger_count + 1, # passenger_count + 1, # passenger_count + 1, # passenger_count + ] + statistics: + key: + indices: [ + 0, # "ARROW:row_count:exact" + 1, # "ARROW:null_count:exact" + 2, # "ARROW:distinct_count:exact" + 3, # "ARROW:max_value:exact" + 4, # "ARROW:min_value:exact" + 1, # "ARROW:null_count:exact" + 2, # "ARROW:distinct_count:exact" + 3, # "ARROW:max_value:exact" + 4, # "ARROW:min_value:exact" + ] + dictionary: [ + "ARROW:row_count:exact", + "ARROW:null_count:exact", + "ARROW:distinct_count:exact", + "ARROW:max_value:exact", + "ARROW:min_value:exact", + ], + items: + children: + 0: [ # int64 + 5, # record batch: "ARROW:row_count:exact" + 0, # vendor_id: "ARROW:null_count:exact" + 2, # vendor_id: "ARROW:distinct_count:exact" + 5, # vendor_id: "ARROW:max_value:exact" + 1, # vendor_id: "ARROW:min_value:exact" + 1, # passenger_count: "ARROW:null_count:exact" + 3, # passenger_count: "ARROW:distinct_count:exact" + 2, # passenger_count: "ARROW:max_value:exact" + 0, # passenger_count: "ARROW:min_value:exact" + ] + types: [ # all values are int64 + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + ] + offsets: [ + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + ] + +Complex record batch +-------------------- + +This uses nested types. + +Schema:: + + col1: struct, c: float64> + col2: utf8 + +Data:: + + col1: [ + {a: 1, b: [20, 30, 40], c: 2.9}, + {a: 2, b: null, c: -2.9}, + {a: 3, b: [99], c: null}, + ] + col2: ["x", null, "z"] + +Statistics: + +.. list-table:: + :header-rows: 1 + + * - Target + - Key + - Value + * - Record batch + - The number of rows + - ``3`` + * - ``col1`` + - The number of nulls + - ``0`` + * - ``col1.a`` + - The number of nulls + - ``0`` + * - ``col1.a`` + - The number of distinct values + - ``3`` + * - ``col1.a`` + - The approximate max value + - ``5`` + * - ``col1.a`` + - The approximate min value + - ``0`` + * - ``col1.b`` + - The number of nulls + - ``1`` + * - ``col1.b.item`` + - The max value + - ``99`` + * - ``col1.b.item`` + - The min value + - ``20`` + * - ``col1.c`` + - The number of nulls + - ``1`` + * - ``col1.c`` + - The approximate max value + - ``3.0`` + * - ``col1.c`` + - The approximate min value + - ``-3.0`` + * - ``col2`` + - The number of nulls + - ``1`` + * - ``col2`` + - The number of distinct values + - ``2`` + +Column indexes: + +.. list-table:: + :header-rows: 1 + + * - Index + - Target + * - ``0`` + - ``col1`` + * - ``1`` + - ``col1.a`` + * - ``2`` + - ``col1.b`` + * - ``3`` + - ``col1.b.item`` + * - ``4`` + - ``col1.c`` + * - ``5`` + - ``col2`` + +See also :ref:`ipc-recordbatch-message` how to compute column indexes. + +Statistics schema:: + + struct< + column: int32, + statistics: map< + key: dictionary< + indices: int32, + dictionary: utf8 + >, + items: dense_union< + # For the number of rows, the number of nulls and so on + 0: int64, + # For the max/min values of col1.c. + 1: float64 + > + > + > + +Statistics array:: + + column: [ + null, # record batch + 0, # col1 + 1, # col1.a + 1, # col1.a + 1, # col1.a + 1, # col1.a + 2, # col1.b + 3, # col1.b.item + 3, # col1.b.item + 4, # col1.c + 4, # col1.c + 4, # col1.c + 5, # col2 + 5, # col2 + ] + statistics: + key: + indices: [ + 0, # "ARROW:row_count:exact" + 1, # "ARROW:null_count:exact" + 1, # "ARROW:null_count:exact" + 2, # "ARROW:distinct_count:exact" + 3, # "ARROW:max_value:approximate" + 4, # "ARROW:min_value:approximate" + 1, # "ARROW:null_count:exact" + 5, # "ARROW:max_value:exact" + 6, # "ARROW:min_value:exact" + 1, # "ARROW:null_count:exact" + 3, # "ARROW:max_value:approximate" + 4, # "ARROW:min_value:approximate" + 1, # "ARROW:null_count:exact" + 2, # "ARROW:distinct_count:exact" + ] + dictionary: [ + "ARROW:row_count:exact", + "ARROW:null_count:exact", + "ARROW:distinct_count:exact", + "ARROW:max_value:approximate", + "ARROW:min_value:approximate", + "ARROW:max_value:exact", + "ARROW:min_value:exact", + ], + items: + children: + 0: [ # int64 + 3, # record batch: "ARROW:row_count:exact" + 0, # col1: "ARROW:null_count:exact" + 0, # col1.a: "ARROW:null_count:exact" + 3, # col1.a: "ARROW:distinct_count:exact" + 5, # col1.a: "ARROW:max_value:approximate" + 0, # col1.a: "ARROW:min_value:approximate" + 1, # col1.b: "ARROW:null_count:exact" + 99, # col1.b.item: "ARROW:max_value:exact" + 20, # col1.b.item: "ARROW:min_value:exact" + 1, # col1.c: "ARROW:null_count:exact" + 1, # col2: "ARROW:null_count:exact" + 2, # col2: "ARROW:distinct_count:exact" + ] + 1: [ # float64 + 3.0, # col1.c: "ARROW:max_value:approximate" + -3.0, # col1.c: "ARROW:min_value:approximate" + ] + types: [ + 0, # int64: record batch: "ARROW:row_count:exact" + 0, # int64: col1: "ARROW:null_count:exact" + 0, # int64: col1.a: "ARROW:null_count:exact" + 0, # int64: col1.a: "ARROW:distinct_count:exact" + 0, # int64: col1.a: "ARROW:max_value:approximate" + 0, # int64: col1.a: "ARROW:min_value:approximate" + 0, # int64: col1.b: "ARROW:null_count:exact" + 0, # int64: col1.b.item: "ARROW:max_value:exact" + 0, # int64: col1.b.item: "ARROW:min_value:exact" + 0, # int64: col1.c: "ARROW:null_count:exact" + 1, # float64: col1.c: "ARROW:max_value:approximate" + 1, # float64: col1.c: "ARROW:min_value:approximate" + 0, # int64: col2: "ARROW:null_count:exact" + 0, # int64: col2: "ARROW:distinct_count:exact" + ] + offsets: [ + 0, # int64: record batch: "ARROW:row_count:exact" + 1, # int64: col1: "ARROW:null_count:exact" + 2, # int64: col1.a: "ARROW:null_count:exact" + 3, # int64: col1.a: "ARROW:distinct_count:exact" + 4, # int64: col1.a: "ARROW:max_value:approximate" + 5, # int64: col1.a: "ARROW:min_value:approximate" + 6, # int64: col1.b: "ARROW:null_count:exact" + 7, # int64: col1.b.item: "ARROW:max_value:exact" + 8, # int64: col1.b.item: "ARROW:min_value:exact" + 9, # int64: col1.c: "ARROW:null_count:exact" + 0, # float64: col1.c: "ARROW:max_value:approximate" + 1, # float64: col1.c: "ARROW:min_value:approximate" + 10, # int64: col2: "ARROW:null_count:exact" + 11, # int64: col2: "ARROW:distinct_count:exact" + ] + + +Simple array +------------ + +TODO + +Complex array +------------- + +TODO: It uses nested type. diff --git a/docs/source/format/index.rst b/docs/source/format/index.rst index ce31a15a1f36a..91912a5325d52 100644 --- a/docs/source/format/index.rst +++ b/docs/source/format/index.rst @@ -32,6 +32,7 @@ Specifications CDataInterface CStreamInterface CDeviceDataInterface + StatisticsSchema DissociatedIPC Flight FlightSql From b0104047c9ecc56f8c70bb0ba4c916366cd413d3 Mon Sep 17 00:00:00 2001 From: Sutou Kouhei Date: Wed, 18 Dec 2024 16:56:12 +0900 Subject: [PATCH 02/10] Remove needless TODOs --- docs/source/format/StatisticsSchema.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/format/StatisticsSchema.rst b/docs/source/format/StatisticsSchema.rst index c595c10420fa7..c45b34122cd84 100644 --- a/docs/source/format/StatisticsSchema.rst +++ b/docs/source/format/StatisticsSchema.rst @@ -178,11 +178,11 @@ Here are pre-defined statistics keys: - Data type - Notes * - ``ARROW:average_byte_width:exact`` - - ``float64``: TODO: Should we use ``int64`` instead? + - ``float64`` - The average size in bytes of a row in the target column. (exact) * - ``ARROW:average_byte_width:approximate`` - - ``float64``: TODO: Should we use ``int64`` instead? + - ``float64`` - The average size in bytes of a row in the target column. (approximate) * - ``ARROW:distinct_count:exact`` From ea687510d958e81210062bbe6b3fca0b63dc89fe Mon Sep 17 00:00:00 2001 From: Sutou Kouhei Date: Mon, 23 Dec 2024 14:17:11 +0900 Subject: [PATCH 03/10] Add array examples --- docs/source/format/StatisticsSchema.rst | 325 ++++++++++++++++++++++-- 1 file changed, 305 insertions(+), 20 deletions(-) diff --git a/docs/source/format/StatisticsSchema.rst b/docs/source/format/StatisticsSchema.rst index c45b34122cd84..7927a7892508c 100644 --- a/docs/source/format/StatisticsSchema.rst +++ b/docs/source/format/StatisticsSchema.rst @@ -220,11 +220,12 @@ Here are pre-defined statistics keys: - The number of nulls in the target column. (approximate) * - ``ARROW:row_count:exact`` - ``int64`` - - The number of rows in the target table or record batch. (exact) + - The number of rows in the target table, record batch or + array. (exact) * - ``ARROW:row_count:approximate`` - ``float64`` - - The number of rows in the target table or record - batch. (approximate) + - The number of rows in the target table, record batch or + array. (approximate) If you find a missing statistics key that is usable for multiple systems, please propose it on the `Apache Arrow development @@ -482,7 +483,7 @@ Statistics schema:: dictionary: utf8 >, items: dense_union< - # For the number of rows, the number of nulls and so on + # For the number of rows, the number of nulls and so on. 0: int64, # For the max/min values of col1.c. 1: float64 @@ -538,18 +539,18 @@ Statistics array:: items: children: 0: [ # int64 - 3, # record batch: "ARROW:row_count:exact" - 0, # col1: "ARROW:null_count:exact" - 0, # col1.a: "ARROW:null_count:exact" - 3, # col1.a: "ARROW:distinct_count:exact" - 5, # col1.a: "ARROW:max_value:approximate" - 0, # col1.a: "ARROW:min_value:approximate" - 1, # col1.b: "ARROW:null_count:exact" - 99, # col1.b.item: "ARROW:max_value:exact" - 20, # col1.b.item: "ARROW:min_value:exact" - 1, # col1.c: "ARROW:null_count:exact" - 1, # col2: "ARROW:null_count:exact" - 2, # col2: "ARROW:distinct_count:exact" + 3, # record batch: "ARROW:row_count:exact" + 0, # col1: "ARROW:null_count:exact" + 0, # col1.a: "ARROW:null_count:exact" + 3, # col1.a: "ARROW:distinct_count:exact" + 5, # col1.a: "ARROW:max_value:approximate" + 0, # col1.a: "ARROW:min_value:approximate" + 1, # col1.b: "ARROW:null_count:exact" + 99, # col1.b.item: "ARROW:max_value:exact" + 20, # col1.b.item: "ARROW:min_value:exact" + 1, # col1.c: "ARROW:null_count:exact" + 1, # col2: "ARROW:null_count:exact" + 2, # col2: "ARROW:distinct_count:exact" ] 1: [ # float64 3.0, # col1.c: "ARROW:max_value:approximate" @@ -581,20 +582,304 @@ Statistics array:: 6, # int64: col1.b: "ARROW:null_count:exact" 7, # int64: col1.b.item: "ARROW:max_value:exact" 8, # int64: col1.b.item: "ARROW:min_value:exact" - 9, # int64: col1.c: "ARROW:null_count:exact" + 9, # int64: col1.c: "ARROW:null_count:exact" 0, # float64: col1.c: "ARROW:max_value:approximate" 1, # float64: col1.c: "ARROW:min_value:approximate" 10, # int64: col2: "ARROW:null_count:exact" 11, # int64: col2: "ARROW:distinct_count:exact" ] - Simple array ------------ -TODO +Schema:: + + int64 + +Data:: + + [1, 1, 2, 0, null] + +Statistics: + +.. list-table:: + :header-rows: 1 + + * - Target + - Key + - Value + * - Array + - The number of rows + - ``5`` + * - Array + - The number of nulls + - ``1`` + * - Array + - The number of distinct values + - ``3`` + * - Array + - The max value + - ``2`` + * - Array + - The min value + - ``0`` + +Column indexes: + +.. list-table:: + :header-rows: 1 + + * - Index + - Target + * - ``0`` + - Array + +Statistics schema:: + + struct< + column: int32, + statistics: map< + key: dictionary< + indices: int32, + dictionary: utf8 + >, + items: dense_union<0: int64> + > + > + +Statistics array:: + + column: [ + 0, # array + 0, # array + 0, # array + 0, # array + 0, # array + ] + statistics: + key: + indices: [ + 0, # "ARROW:row_count:exact" + 1, # "ARROW:null_count:exact" + 2, # "ARROW:distinct_count:exact" + 3, # "ARROW:max_value:exact" + 4, # "ARROW:min_value:exact" + ] + dictionary: [ + "ARROW:row_count:exact", + "ARROW:null_count:exact", + "ARROW:distinct_count:exact", + "ARROW:max_value:exact", + "ARROW:min_value:exact", + ], + items: + children: + 0: [ # int64 + 5, # array: "ARROW:row_count:exact" + 1, # array: "ARROW:null_count:exact" + 3, # array: "ARROW:distinct_count:exact" + 2, # array: "ARROW:max_value:exact" + 0, # array: "ARROW:min_value:exact" + ] + types: [ # all values are int64 + 0, + 0, + 0, + 0, + 0, + ] + offsets: [ + 0, + 1, + 2, + 3, + 4, + ] Complex array ------------- -TODO: It uses nested type. +This uses nested types. + +Schema:: + + struct, c: float64> + +Data:: + + [ + {a: 1, b: [20, 30, 40], c: 2.9}, + {a: 2, b: null, c: -2.9}, + {a: 3, b: [99], c: null}, + ] + +Statistics: + +.. list-table:: + :header-rows: 1 + + * - Target + - Key + - Value + * - Array + - The number of rows + - ``3`` + * - Array + - The number of nulls + - ``0`` + * - ``a`` + - The number of nulls + - ``0`` + * - ``a`` + - The number of distinct values + - ``3`` + * - ``a`` + - The approximate max value + - ``5`` + * - ``a`` + - The approximate min value + - ``0`` + * - ``b`` + - The number of nulls + - ``1`` + * - ``b.item`` + - The max value + - ``99`` + * - ``b.item`` + - The min value + - ``20`` + * - ``c`` + - The number of nulls + - ``1`` + * - ``c`` + - The approximate max value + - ``3.0`` + * - ``c`` + - The approximate min value + - ``-3.0`` + +Column indexes: + +.. list-table:: + :header-rows: 1 + + * - Index + - Target + * - ``0`` + - Array + * - ``1`` + - ``a`` + * - ``2`` + - ``b`` + * - ``3`` + - ``b.item`` + * - ``4`` + - ``c`` + +See also :ref:`ipc-recordbatch-message` how to compute column indexes. + +Statistics schema:: + + struct< + column: int32, + statistics: map< + key: dictionary< + indices: int32, + dictionary: utf8 + >, + items: dense_union< + # For the number of rows, the number of nulls and so on. + 0: int64, + # For the max/min values of c. + 1: float64 + > + > + > + +Statistics array:: + + column: [ + 0, # array + 0, # array + 1, # a + 1, # a + 1, # a + 1, # a + 2, # b + 3, # b.item + 3, # b.item + 4, # c + 4, # c + 4, # c + ] + statistics: + key: + indices: [ + 0, # "ARROW:row_count:exact" + 1, # "ARROW:null_count:exact" + 1, # "ARROW:null_count:exact" + 2, # "ARROW:distinct_count:exact" + 3, # "ARROW:max_value:approximate" + 4, # "ARROW:min_value:approximate" + 1, # "ARROW:null_count:exact" + 5, # "ARROW:max_value:exact" + 6, # "ARROW:min_value:exact" + 1, # "ARROW:null_count:exact" + 3, # "ARROW:max_value:approximate" + 4, # "ARROW:min_value:approximate" + ] + dictionary: [ + "ARROW:row_count:exact", + "ARROW:null_count:exact", + "ARROW:distinct_count:exact", + "ARROW:max_value:approximate", + "ARROW:min_value:approximate", + "ARROW:max_value:exact", + "ARROW:min_value:exact", + ], + items: + children: + 0: [ # int64 + 3, # array: "ARROW:row_count:exact" + 0, # array: "ARROW:null_count:exact" + 0, # a: "ARROW:null_count:exact" + 3, # a: "ARROW:distinct_count:exact" + 5, # a: "ARROW:max_value:approximate" + 0, # a: "ARROW:min_value:approximate" + 1, # b: "ARROW:null_count:exact" + 99, # b.item: "ARROW:max_value:exact" + 20, # b.item: "ARROW:min_value:exact" + 1, # c: "ARROW:null_count:exact" + ] + 1: [ # float64 + 3.0, # c: "ARROW:max_value:approximate" + -3.0, # c: "ARROW:min_value:approximate" + ] + types: [ + 0, # int64: array: "ARROW:row_count:exact" + 0, # int64: array: "ARROW:null_count:exact" + 0, # int64: a: "ARROW:null_count:exact" + 0, # int64: a: "ARROW:distinct_count:exact" + 0, # int64: a: "ARROW:max_value:approximate" + 0, # int64: a: "ARROW:min_value:approximate" + 0, # int64: b: "ARROW:null_count:exact" + 0, # int64: b.item: "ARROW:max_value:exact" + 0, # int64: b.item: "ARROW:min_value:exact" + 0, # int64: c: "ARROW:null_count:exact" + 1, # float64: c: "ARROW:max_value:approximate" + 1, # float64: c: "ARROW:min_value:approximate" + ] + offsets: [ + 0, # int64: array: "ARROW:row_count:exact" + 1, # int64: array: "ARROW:null_count:exact" + 2, # int64: a: "ARROW:null_count:exact" + 3, # int64: a: "ARROW:distinct_count:exact" + 4, # int64: a: "ARROW:max_value:approximate" + 5, # int64: a: "ARROW:min_value:approximate" + 6, # int64: b: "ARROW:null_count:exact" + 7, # int64: b.item: "ARROW:max_value:exact" + 8, # int64: b.item: "ARROW:min_value:exact" + 9, # int64: c: "ARROW:null_count:exact" + 0, # float64: c: "ARROW:max_value:approximate" + 1, # float64: c: "ARROW:min_value:approximate" + ] From 195c3742a9d0b09248a67da8ac41dd60014fefdb Mon Sep 17 00:00:00 2001 From: Sutou Kouhei Date: Mon, 23 Dec 2024 14:17:28 +0900 Subject: [PATCH 04/10] Remove TODOs --- docs/source/format/StatisticsSchema.rst | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/source/format/StatisticsSchema.rst b/docs/source/format/StatisticsSchema.rst index 7927a7892508c..9ff652452fc5f 100644 --- a/docs/source/format/StatisticsSchema.rst +++ b/docs/source/format/StatisticsSchema.rst @@ -147,8 +147,6 @@ Here is the details of the ``map`` of the ``statistics``: can access to proper field by type code not name. So we can use any valid name for fields. - TODO: Should we standardize field names? - .. _statistics-schema-key: Statistics key @@ -197,7 +195,7 @@ Here are pre-defined statistics keys: - The maximum size in bytes of a row in the target column. (exact) * - ``ARROW:max_byte_width:approximate`` - - ``float64``: TODO: Should we use ``int64`` instead? + - ``float64`` - The maximum size in bytes of a row in the target column. (approximate) * - ``ARROW:max_value:exact`` From cf18851eed348052552be4d28f52489aff35ff62 Mon Sep 17 00:00:00 2001 From: Sutou Kouhei Date: Tue, 24 Dec 2024 09:00:06 +0900 Subject: [PATCH 05/10] Improve English Co-authored-by: Felipe Oliveira Carvalho --- docs/source/format/StatisticsSchema.rst | 49 ++++++++++++++----------- 1 file changed, 27 insertions(+), 22 deletions(-) diff --git a/docs/source/format/StatisticsSchema.rst b/docs/source/format/StatisticsSchema.rst index 9ff652452fc5f..eddd4109cc402 100644 --- a/docs/source/format/StatisticsSchema.rst +++ b/docs/source/format/StatisticsSchema.rst @@ -34,8 +34,8 @@ be read as Apache Arrow data may have statistics. For example, the Apache Parquet C++ implementation can read an Apache Parquet file as Apache Arrow data and the Apache Parquet file may have statistics. -We standardize how to represent statistics as an Apache Arrow array -for easy to exchange. +We standardize the representation of statistics as an Apache Arrow array +for ease of exchange. Use case -------- @@ -52,8 +52,8 @@ file to module B, module B can use the statistics to optimize its query plan. For example, DuckDB uses this approach but DuckDB couldn't use -statistics because there wasn't the standardized way to represent -statistics for an Apache Arrow data. +statistics because there wasn't a standardized way to represent +statistics for the Apache Arrow data. .. seealso:: @@ -71,8 +71,8 @@ Non-goals * Establish a standard way to pass an Apache Arrow array that represents statistics. -* Establish a standard way to embed statistics to an Apache Arrow - array. +* Establish a standard way to embed statistics into an Apache Arrow + array itself. Schema ====== @@ -129,10 +129,10 @@ Here is the details of the ``map`` of the ``statistics``: * - key - ``dictionary`` - ``false`` - - Statistics key is string. Dictionary is used for - efficiency. Different keys are assigned for exact value and - approximate value. Also see the separate description below for - statistics key. + - The string key is the name of the statistic. Dictionary-encoding is used for + efficiency as the same statistic may be repeated for different columns. + Different keys are assigned for exact and + approximate statistic values. Each statistic has their own description below. * - items - ``dense_union`` - ``false`` @@ -140,8 +140,7 @@ Here is the details of the ``map`` of the ``statistics``: types based on statistics kinds in the keys. For example, you need at least ``int64`` and ``float64`` types when you have a ``int64`` distinct count statistic and a ``float64`` average - byte width statistic. Also see the separate description below - for statistics key. + byte width statistic. See the description of each statistic below. We don't standardize field names for the dense union because we can access to proper field by type code not name. So we can use @@ -149,21 +148,23 @@ Here is the details of the ``map`` of the ``statistics``: .. _statistics-schema-key: -Statistics key --------------- +Standard statistics +------------------- -Statistics key is string. ``dictionary`` is used for -efficiency. +Each statistic kind has a name that appears as a key in the statistics map +for each column or entire table. ``dictionary`` +is used to encode the key for space-efficiency. -We assign different statistics keys for individual statistics instead -of using flags. For example, we assign different statistics keys for -exact value and approximate value. +We assign different names for variations of the same statistic instead +of using flags. For example, we assign different statistic names for +exact and approximate values of the "distinct_count" statistic. The colon symbol ``:`` is to be used as a namespace separator like :ref:`format_metadata`. It can be used multiple times in a key. -The ``ARROW`` pattern is a reserved namespace for pre-defined -statistics keys. User-defined statistics must not use it. +The ``ARROW`` prefix is a reserved namespace for pre-defined +statistic names in current and future versions of this specification. +User-defined statistics must not use it. For example, you can use your product name as namespace such as ``MY_PRODUCT:my_statistics:exact``. @@ -225,10 +226,14 @@ Here are pre-defined statistics keys: - The number of rows in the target table, record batch or array. (approximate) -If you find a missing statistics key that is usable for multiple +If you find a statistic that might be useful to multiple systems, please propose it on the `Apache Arrow development mailing-list `__. +Interoperability improves when producers and consumers of +statistics follow a previously agreed upon statistic +specification. + .. _statistics-schema-examples: Examples From 96872e72e046e219dfa0b943c8c9ee980f90e2ad Mon Sep 17 00:00:00 2001 From: Sutou Kouhei Date: Tue, 24 Dec 2024 09:07:45 +0900 Subject: [PATCH 06/10] Use "values" not "dictionary" Co-authored-by: Felipe Oliveira Carvalho --- docs/source/format/StatisticsSchema.rst | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/docs/source/format/StatisticsSchema.rst b/docs/source/format/StatisticsSchema.rst index eddd4109cc402..4c3b0f7e5b4e1 100644 --- a/docs/source/format/StatisticsSchema.rst +++ b/docs/source/format/StatisticsSchema.rst @@ -86,10 +86,7 @@ Here is the outline of the schema for statistics:: struct< column: int32, statistics: map< - key: dictionary< - indices: int32, - dictionary: utf8 - >, + key: dictionary, items: dense_union<...all needed types...> > > @@ -127,7 +124,7 @@ Here is the details of the ``map`` of the ``statistics``: - Nullable - Notes * - key - - ``dictionary`` + - ``dictionary`` - ``false`` - The string key is the name of the statistic. Dictionary-encoding is used for efficiency as the same statistic may be repeated for different columns. From 44c48bfb50f2d81534bf78def28828405eac512f Mon Sep 17 00:00:00 2001 From: Sutou Kouhei Date: Tue, 24 Dec 2024 09:16:59 +0900 Subject: [PATCH 07/10] Add a note about dense union field name --- docs/source/format/StatisticsSchema.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/format/StatisticsSchema.rst b/docs/source/format/StatisticsSchema.rst index 4c3b0f7e5b4e1..728b411f6ce96 100644 --- a/docs/source/format/StatisticsSchema.rst +++ b/docs/source/format/StatisticsSchema.rst @@ -139,7 +139,7 @@ Here is the details of the ``map`` of the ``statistics``: ``int64`` distinct count statistic and a ``float64`` average byte width statistic. See the description of each statistic below. - We don't standardize field names for the dense union because we + Dense union has name for each field but we don't standardize field names for the dense union because we can access to proper field by type code not name. So we can use any valid name for fields. From ad4937186a7382edb2ad7ef430451d46220a3c86 Mon Sep 17 00:00:00 2001 From: Sutou Kouhei Date: Tue, 24 Dec 2024 09:21:15 +0900 Subject: [PATCH 08/10] Adjust --- docs/source/format/StatisticsSchema.rst | 136 +++++++++++------------- 1 file changed, 62 insertions(+), 74 deletions(-) diff --git a/docs/source/format/StatisticsSchema.rst b/docs/source/format/StatisticsSchema.rst index 728b411f6ce96..e5c3563c709f8 100644 --- a/docs/source/format/StatisticsSchema.rst +++ b/docs/source/format/StatisticsSchema.rst @@ -34,8 +34,8 @@ be read as Apache Arrow data may have statistics. For example, the Apache Parquet C++ implementation can read an Apache Parquet file as Apache Arrow data and the Apache Parquet file may have statistics. -We standardize the representation of statistics as an Apache Arrow array -for ease of exchange. +We standardize the representation of statistics as an Apache Arrow +array for ease of exchange. Use case -------- @@ -86,7 +86,7 @@ Here is the outline of the schema for statistics:: struct< column: int32, statistics: map< - key: dictionary, + key: dictionary, items: dense_union<...all needed types...> > > @@ -124,12 +124,13 @@ Here is the details of the ``map`` of the ``statistics``: - Nullable - Notes * - key - - ``dictionary`` + - ``dictionary`` - ``false`` - - The string key is the name of the statistic. Dictionary-encoding is used for - efficiency as the same statistic may be repeated for different columns. - Different keys are assigned for exact and - approximate statistic values. Each statistic has their own description below. + - The string key is the name of the + statistic. Dictionary-encoding is used for efficiency as the + same statistic may be repeated for different columns. + Different keys are assigned for exact and approximate statistic + values. Each statistic has their own description below. * - items - ``dense_union`` - ``false`` @@ -139,31 +140,31 @@ Here is the details of the ``map`` of the ``statistics``: ``int64`` distinct count statistic and a ``float64`` average byte width statistic. See the description of each statistic below. - Dense union has name for each field but we don't standardize field names for the dense union because we - can access to proper field by type code not name. So we can use - any valid name for fields. + Dense union has name for each field but we don't standardize + field names for the dense union because we can access to proper + field by type code not name. So we can use any valid name for + fields. .. _statistics-schema-key: Standard statistics ------------------- -Each statistic kind has a name that appears as a key in the statistics map -for each column or entire table. ``dictionary`` -is used to encode the key for space-efficiency. +Each statistic kind has a name that appears as a key in the statistics +map for each column or entire table. ``dictionary`` is used to encode the key for space-efficiency. We assign different names for variations of the same statistic instead of using flags. For example, we assign different statistic names for -exact and approximate values of the "distinct_count" statistic. +exact and approximate values of the "distinct count" statistic. The colon symbol ``:`` is to be used as a namespace separator like :ref:`format_metadata`. It can be used multiple times in a key. -The ``ARROW`` prefix is a reserved namespace for pre-defined -statistic names in current and future versions of this specification. -User-defined statistics must not use it. -For example, you can use your product name as namespace -such as ``MY_PRODUCT:my_statistics:exact``. +The ``ARROW`` prefix is a reserved namespace for pre-defined statistic +names in current and future versions of this specification. +User-defined statistics must not use it. For example, you can use your +product name as namespace such as ``MY_PRODUCT:my_statistics:exact``. Here are pre-defined statistics keys: @@ -223,13 +224,12 @@ Here are pre-defined statistics keys: - The number of rows in the target table, record batch or array. (approximate) -If you find a statistic that might be useful to multiple -systems, please propose it on the `Apache Arrow development -mailing-list `__. +If you find a statistic that might be useful to multiple systems, +please propose it on the `Apache Arrow development mailing-list +`__. -Interoperability improves when producers and consumers of -statistics follow a previously agreed upon statistic -specification. +Interoperability improves when producers and consumers of statistics +follow a previously agreed upon statistic specification. .. _statistics-schema-examples: @@ -304,10 +304,7 @@ Statistics schema:: struct< column: int32, statistics: map< - key: dictionary< - indices: int32, - dictionary: utf8 - >, + key: dictionary, items: dense_union<0: int64> > > @@ -327,6 +324,13 @@ Statistics array:: ] statistics: key: + values: [ + "ARROW:row_count:exact", + "ARROW:null_count:exact", + "ARROW:distinct_count:exact", + "ARROW:max_value:exact", + "ARROW:min_value:exact", + ], indices: [ 0, # "ARROW:row_count:exact" 1, # "ARROW:null_count:exact" @@ -338,13 +342,6 @@ Statistics array:: 3, # "ARROW:max_value:exact" 4, # "ARROW:min_value:exact" ] - dictionary: [ - "ARROW:row_count:exact", - "ARROW:null_count:exact", - "ARROW:distinct_count:exact", - "ARROW:max_value:exact", - "ARROW:min_value:exact", - ], items: children: 0: [ # int64 @@ -478,10 +475,7 @@ Statistics schema:: struct< column: int32, statistics: map< - key: dictionary< - indices: int32, - dictionary: utf8 - >, + key: dictionary, items: dense_union< # For the number of rows, the number of nulls and so on. 0: int64, @@ -511,6 +505,15 @@ Statistics array:: ] statistics: key: + values: [ + "ARROW:row_count:exact", + "ARROW:null_count:exact", + "ARROW:distinct_count:exact", + "ARROW:max_value:approximate", + "ARROW:min_value:approximate", + "ARROW:max_value:exact", + "ARROW:min_value:exact", + ] indices: [ 0, # "ARROW:row_count:exact" 1, # "ARROW:null_count:exact" @@ -527,15 +530,6 @@ Statistics array:: 1, # "ARROW:null_count:exact" 2, # "ARROW:distinct_count:exact" ] - dictionary: [ - "ARROW:row_count:exact", - "ARROW:null_count:exact", - "ARROW:distinct_count:exact", - "ARROW:max_value:approximate", - "ARROW:min_value:approximate", - "ARROW:max_value:exact", - "ARROW:min_value:exact", - ], items: children: 0: [ # int64 @@ -639,10 +633,7 @@ Statistics schema:: struct< column: int32, statistics: map< - key: dictionary< - indices: int32, - dictionary: utf8 - >, + key: dictionary, items: dense_union<0: int64> > > @@ -658,6 +649,13 @@ Statistics array:: ] statistics: key: + values: [ + "ARROW:row_count:exact", + "ARROW:null_count:exact", + "ARROW:distinct_count:exact", + "ARROW:max_value:exact", + "ARROW:min_value:exact", + ] indices: [ 0, # "ARROW:row_count:exact" 1, # "ARROW:null_count:exact" @@ -665,13 +663,6 @@ Statistics array:: 3, # "ARROW:max_value:exact" 4, # "ARROW:min_value:exact" ] - dictionary: [ - "ARROW:row_count:exact", - "ARROW:null_count:exact", - "ARROW:distinct_count:exact", - "ARROW:max_value:exact", - "ARROW:min_value:exact", - ], items: children: 0: [ # int64 @@ -783,10 +774,7 @@ Statistics schema:: struct< column: int32, statistics: map< - key: dictionary< - indices: int32, - dictionary: utf8 - >, + key: dictionary, items: dense_union< # For the number of rows, the number of nulls and so on. 0: int64, @@ -814,6 +802,15 @@ Statistics array:: ] statistics: key: + values: [ + "ARROW:row_count:exact", + "ARROW:null_count:exact", + "ARROW:distinct_count:exact", + "ARROW:max_value:approximate", + "ARROW:min_value:approximate", + "ARROW:max_value:exact", + "ARROW:min_value:exact", + ] indices: [ 0, # "ARROW:row_count:exact" 1, # "ARROW:null_count:exact" @@ -828,15 +825,6 @@ Statistics array:: 3, # "ARROW:max_value:approximate" 4, # "ARROW:min_value:approximate" ] - dictionary: [ - "ARROW:row_count:exact", - "ARROW:null_count:exact", - "ARROW:distinct_count:exact", - "ARROW:max_value:approximate", - "ARROW:min_value:approximate", - "ARROW:max_value:exact", - "ARROW:min_value:exact", - ], items: children: 0: [ # int64 From 3b9c918a0cd271166fa9862a4a61a2dc84df7b57 Mon Sep 17 00:00:00 2001 From: Sutou Kouhei Date: Tue, 24 Dec 2024 09:27:12 +0900 Subject: [PATCH 09/10] Use "name" not "key" --- docs/source/format/StatisticsSchema.rst | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/source/format/StatisticsSchema.rst b/docs/source/format/StatisticsSchema.rst index e5c3563c709f8..18b132ae6cdf5 100644 --- a/docs/source/format/StatisticsSchema.rst +++ b/docs/source/format/StatisticsSchema.rst @@ -145,33 +145,33 @@ Here is the details of the ``map`` of the ``statistics``: field by type code not name. So we can use any valid name for fields. -.. _statistics-schema-key: +.. _statistics-schema-name: Standard statistics ------------------- Each statistic kind has a name that appears as a key in the statistics map for each column or entire table. ``dictionary`` is used to encode the key for space-efficiency. +indices: int32>`` is used to encode the name for space-efficiency. We assign different names for variations of the same statistic instead of using flags. For example, we assign different statistic names for exact and approximate values of the "distinct count" statistic. The colon symbol ``:`` is to be used as a namespace separator like -:ref:`format_metadata`. It can be used multiple times in a key. +:ref:`format_metadata`. It can be used multiple times in a name. The ``ARROW`` prefix is a reserved namespace for pre-defined statistic names in current and future versions of this specification. User-defined statistics must not use it. For example, you can use your product name as namespace such as ``MY_PRODUCT:my_statistics:exact``. -Here are pre-defined statistics keys: +Here are pre-defined statistics names: .. list-table:: :header-rows: 1 - * - Key + * - Name - Data type - Notes * - ``ARROW:average_byte_width:exact`` @@ -257,7 +257,7 @@ Statistics: :header-rows: 1 * - Target - - Key + - Name - Value * - Record batch - The number of rows @@ -403,7 +403,7 @@ Statistics: :header-rows: 1 * - Target - - Key + - Name - Value * - Record batch - The number of rows @@ -600,7 +600,7 @@ Statistics: :header-rows: 1 * - Target - - Key + - Name - Value * - Array - The number of rows @@ -710,7 +710,7 @@ Statistics: :header-rows: 1 * - Target - - Key + - Name - Value * - Array - The number of rows From cb0251637607be60b0e8fd0c9bd0bc8acc3c4ca1 Mon Sep 17 00:00:00 2001 From: Sutou Kouhei Date: Wed, 25 Dec 2024 06:20:37 +0900 Subject: [PATCH 10/10] Fix English Co-authored-by: Felipe Oliveira Carvalho --- docs/source/format/StatisticsSchema.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/source/format/StatisticsSchema.rst b/docs/source/format/StatisticsSchema.rst index 18b132ae6cdf5..01cc0da7c4eb5 100644 --- a/docs/source/format/StatisticsSchema.rst +++ b/docs/source/format/StatisticsSchema.rst @@ -140,10 +140,10 @@ Here is the details of the ``map`` of the ``statistics``: ``int64`` distinct count statistic and a ``float64`` average byte width statistic. See the description of each statistic below. - Dense union has name for each field but we don't standardize - field names for the dense union because we can access to proper - field by type code not name. So we can use any valid name for - fields. + Dense union arrays have names for each field but we don't standardize + field names for these because we can access the proper + field by type code instead. So we can use any valid name for + the fields. .. _statistics-schema-name: