GH-42102: [C++][Parquet] Add binary that extracts a footer from a parquet file

[alkis]

Rationale for this change

This binary will make it a lot easier for customers to share their parquet metadata with the community so that we can build a repository of footers that can be used for advancing the state of metadata in parquet.

What changes are included in this PR?

Usage from the file binary itself:

Usage: parquet-dump-footer
  -h|--help    Print help and exit
  --no-scrub   Do not scrub potentially confidential metadata
  --debug      Output text represenation of footer for inspection
  --in <uri>   Input file (required): must be an URI or an absolute local path
  --out <path> Output file (optional, default stdout)

  Dump the footer of a Parquet file to stdout or to a file, optionally with
  potentially confidential metadata scrubbed.

Are these changes tested?

Manually on existing parquet files.

Are there any user-facing changes?

No.

• GitHub Issue: [C++][Parquet] Add tool to extract parquet footers #42102

[github-actions]

⚠️ GitHub issue #42102 has been automatically assigned in GitHub to PR creator.

[alkis]

Windows build keeps failing. Could you advice? It seems it can't find library arrow_static to link to.

[emkornfield]

Just a high-level comment on usability. I don't think any binaries are currently packaged in release channels so if we want this to be easier to use we should consider embedding it in the library with python bindings

[alkis]

Just a high-level comment on usability. I don't think any binaries are currently packaged in release channels so if we want this to be easier to use we should consider embedding it in the library with python bindings

You mean writing this in python and shipping as a python script?

[kou]

I don't think any binaries are currently packaged in release channels

FYI: deb and RPM include binaries:

• https://github.com/apache/arrow/blob/main/dev/tasks/linux-packages/apache-arrow/debian/parquet-tools.install

• arrow/dev/tasks/linux-packages/apache-arrow/yum/arrow.spec.in

  Line 531 in a01fe03

  %{_bindir}/parquet-*

[emkornfield]

I don't think any binaries are currently packaged in release channels

FYI: deb and RPM include binaries:

• https://github.com/apache/arrow/blob/main/dev/tasks/linux-packages/apache-arrow/debian/parquet-tools.install

• arrow/dev/tasks/linux-packages/apache-arrow/yum/arrow.spec.in

  Line 531 in a01fe03

  %{_bindir}/parquet-*

I didn't realize this. I guess I was thinking of the pyarrow/R libraries as the primary distribution source.

You mean writing this in python and shipping as a python script?

I was thinking of just include the functionality in the pyarrow arrow library by including C++ as part of the library and wrapping in python. given that there is some packaging that does include the binaries this might be less of an issue.

[alkis]

Given that tools are packages in deb and rpm this covers the majority of linux.

[alkis]

Is this ready for merge?

[mapleFU]

General looks ok to me, so this patch would extract a footer with "scrub" might be used?

[alkis]

General looks ok to me, so this patch would extract a footer with "scrub" might be used?

Yes, the purpose is to build a repository of large/slow to parse footers and guide the design of better parquet metadata representation.

[alkis]

Friendly ping. Can this be merged?

[pitrou]

@github-actions crossbow submit -g cpp

[pitrou]

LGTM. @alkis could you review the changes I've made to make sure they're ok to you?

[alkis]

LGTM. @alkis could you review the changes I've made to make sure they're ok to you?

Thanks a ton for the fixes! I reverted the JSON fix and named the optional properly as --debug. I wanted that option for folks to be able to inspect what's going to be the output of the command before donating a footer to us for benchmarking use.

[pitrou]

Ah, fair enough.

[pitrou]

@github-actions crossbow submit -g cpp

[github-actions]

Revision: 44f9768

Submitted crossbow builds: ursacomputing/crossbow @ actions-366196b72e

Task	Status
test-alpine-linux-cpp
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-cuda-cpp
test-debian-12-cpp-amd64
test-debian-12-cpp-i386
test-fedora-39-cpp
test-ubuntu-20.04-cpp
test-ubuntu-20.04-cpp-bundled
test-ubuntu-20.04-cpp-minimal-with-formats
test-ubuntu-20.04-cpp-thread-sanitizer
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-20
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-gcc-14

[alkis]

I have one more nit coming in.

[pitrou]

Thanks @alkis . I'm waiting for CI and will merge if ok.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit d21a924.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 4 possible false positives for unstable benchmarks that are known to sometimes produce them.