Improve stats convert performance for Binary/String/Boolean arrays

[Rachelint]

Which issue does this PR close?

Closes #11281

Rationale for this change

Improve the stats convert performance as possible.

What changes are included in this PR?

• Use string builder directly to eliminate some tmp Vecs and Strings(for example, Utf8 and LargeUtf case in both row group or page stats).
• Eliminate some unnecessary collect to avoid tmp Vec creation

Are these changes tested?

Test by exist tests.

Are there any user-facing changes?

No

[alamb]

Thank you @Rachelint -- this code looks great. 🚀

I do think this will conflict with #11295 from @efredine

I am running the benchmarks and will post my results shortly

[alamb]

Rather than using expect it is likely possible to check the value of builder.append_value and if it is an error append null. But perhaps that is overly confusing. This formulation is quite clear and explicit

[Rachelint]

Rather than using expect it is likely possible to check the value of builder.append_value and if it is an error append null. But perhaps that is overly confusing. This formulation is quite clear and explicit

🤔 Yes, I append_null before. And I read the codes of append_value after, I found the only reason to return Err is to append a value with different width that has been checked and ensured actually.

[alamb]

👍

[efredine]

We could use BooleanBuilder instead? That would likely be more efficient.

[Rachelint]

We could use BooleanBuilder instead? That would likely be more efficient.

Seems reasonable! Maybe it can work to remove the collect in BooleanArray case, I try it soon.

[alamb]

I suspect the remaining cases where we are using collect could be made more efficient using the Builder pattern?

I think the reason the Builder is faster for Strings / Binary is that due to how the references worked out, we can avoid copying the strings via value.to_string()

I don't think the Builder pattern is fundamentally better/worse than the from_iter (under the covers they all end up doing the same thing in arrow-rs I think)

[alamb]

I ran the benchmarks and this looks like a 2x performance improvement for strings and dictionaries 👍

group                                                                                                      improve-page-stats-convert              main
-----                                                                                                      --------------------------              ----
Extract data page statistics for Dictionary(Int32, String)/extract_statistics/Dictionary(Int32, String)    1.00     39.6±3.96µs        ? ?/sec     2.28     90.3±0.20µs        ? ?/sec
Extract data page statistics for F64/extract_statistics/F64                                                1.05     16.7±0.03µs        ? ?/sec     1.00     15.9±0.04µs        ? ?/sec
Extract data page statistics for Int64/extract_statistics/Int64                                            1.00     14.8±0.05µs        ? ?/sec     1.02     15.1±0.47µs        ? ?/sec
Extract data page statistics for String/extract_statistics/String                                          1.00     39.3±4.71µs        ? ?/sec     2.30     90.2±0.29µs        ? ?/sec
Extract data page statistics for UInt64/extract_statistics/UInt64                                          1.01     15.5±0.03µs        ? ?/sec     1.00     15.4±0.04µs        ? ?/sec
Extract row group statistics for Dictionary(Int32, String)/extract_statistics/Dictionary(Int32, String)    1.00   1962.4±3.95ns        ? ?/sec     1.06      2.1±0.01µs        ? ?/sec
Extract row group statistics for F64/extract_statistics/F64                                                1.02   1193.2±7.54ns        ? ?/sec     1.00   1174.4±5.82ns        ? ?/sec
Extract row group statistics for Int64/extract_statistics/Int64                                            1.00   1090.8±3.08ns        ? ?/sec     1.01   1099.2±1.98ns        ? ?/sec
Extract row group statistics for String/extract_statistics/String                                          1.00  1918.0±235.67ns        ? ?/sec    1.07      2.1±0.00µs        ? ?/sec
Extract row group statistics for UInt64/extract_statistics/UInt64                                          1.00   1166.0±2.87ns        ? ?/sec     1.01   1176.4±2.77ns        ? ?/sec

[Rachelint]

I suspect the remaining cases where we are using collect could be made more efficient using the Builder pattern?

I think the reason the Builder is faster for Strings / Binary is that due to how the references worked out, we can avoid copying the strings via value.to_string()

I don't think the Builder pattern is fundamentally better/worse than the from_iter (under the covers they all end up doing the same thing in arrow-rs I think)

Yes, I tested the Uint64Array case in my POC, use Builder directly seems not obviously faster than use from_iter actually.
https://github.com/Rachelint/arrow-datafusion/blob/70b9f05e737e81c514259e70a8bd2e9f0ad8e725/datafusion/core/src/datasource/physical_plan/parquet/statistics.rs#L790

[Rachelint]

@alamb Have merged with main, seem ready now.
@efredine @alamb And I made an poc about using Builder instead of collect, performance improved!
see https://github.com/Rachelint/arrow-datafusion/blob/a6f748d0a12559e8c243f2a76ce2c2757bf7c93c/datafusion/core/src/datasource/physical_plan/parquet/statistics.rs#L748-L784

• use collect

Extract data page statistics for Boolean/extract_statistics/Boolean
                        time:   [11.392 µs 11.403 µs 11.411 µs]
                        change: [-0.2092% -0.0651% +0.1005%] (p = 0.42 > 0.05)

• use Builder(the builder v2 arm)

Extract data page statistics for Boolean/extract_statistics/Boolean
                        time:   [10.530 µs 10.542 µs 10.553 µs]
                        change: [-18.333% -18.201% -18.087%] (p = 0.00 < 0.05)

I have added such codes about BooleanArray to this pr.

[alamb]

Looks great -- thanks again @Rachelint and @efredine