Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Add basic benchmark suite to C library #393

Merged
merged 6 commits into from
Mar 7, 2024

Conversation

paleolimbot
Copy link
Member

@paleolimbot paleolimbot commented Feb 28, 2024

This PR adds an initial set of benchmarks covering some realistic usage patterns. The general approach is to use doxygen comments to document the benchmarks, which will run against the released version and the previous version. I'm not sure exactly what the output format will be but I'd like the benchmarks to be written in such a way that there's a path to programatically generating a report (maybe using conbench, maybe just a Quarto document).

Work in progress!

@pitrou
Copy link
Member

pitrou commented Feb 28, 2024

Can you show what the results look like?

@paleolimbot
Copy link
Member Author

This is from the CI run (so timings are maybe meaningless), but the output looks like:

  2024-02-28T17:43:48+00:00
  ::group::array_benchmark

  Run on (4 X 3139.35 MHz CPU s)
  CPU Caches:
    L1 Data 32 KiB (x2)
    L1 Instruction 32 KiB (x2)
    L2 Unified 512 KiB (x2)
    L3 Unified 32768 KiB (x1)
  Load Average: 0.89, 0.34, 0.12
  -------------------------------------------------------------------------------------------------
  Benchmark                                       Time             CPU   Iterations UserCounters...
  -------------------------------------------------------------------------------------------------
  BM_ArrayViewGetIntUnsafeInt8              1576584 ns      1576545 ns          449 items_per_second=634.298M/s
  BM_ArrayViewGetIntUnsafeInt16              936609 ns       936540 ns          749 items_per_second=1.06776G/s
  BM_ArrayViewGetIntUnsafeInt32             1244619 ns      1244574 ns          562 items_per_second=803.488M/s
  BM_ArrayViewGetIntUnsafeInt64              945470 ns       945435 ns          745 items_per_second=1.05771G/s
  BM_ArrayViewGetIntUnsafeInt64CheckNull    1751277 ns      1751243 ns          396 items_per_second=571.023M/s

::group::schema_benchmark
  2024-02-28T17:43:52+00:00
  Running ./schema_benchmark
  Run on (4 X 3241.55 MHz CPU s)
  CPU Caches:
    L1 Data 32 KiB (x2)
    L1 Instruction 32 KiB (x2)
    L2 Unified 512 KiB (x2)
    L3 Unified 32768 KiB (x1)
  Load Average: 0.90, 0.35, 0.13
  --------------------------------------------------------------------------------------
  Benchmark                            Time             CPU   Iterations UserCounters...
  --------------------------------------------------------------------------------------
  BM_SchemaInitWideStruct         768760 ns       768689 ns          911 items_per_second=13.0092M/s
  BM_SchemaViewInitWideStruct     175154 ns       175138 ns         4202 items_per_second=57.0978M/s

BENCHMARK(BM_ArrayViewGetIntUnsafeInt16);
BENCHMARK(BM_ArrayViewGetIntUnsafeInt32);
BENCHMARK(BM_ArrayViewGetIntUnsafeInt64);
BENCHMARK(BM_ArrayViewGetIntUnsafeInt64CheckNull);

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm in Arrow C++ we don't start the benchmarks with BM_. Is that necessary?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! I think it was rather contagious copy/paste from the first example in the benchmark library README 😬

@pitrou
Copy link
Member

pitrou commented Feb 29, 2024

This is from the CI run (so timings are maybe meaningless), but the output looks like:

Thank you! This looks fine to me.

@paleolimbot paleolimbot marked this pull request as ready for review March 7, 2024 18:46
@paleolimbot paleolimbot merged commit 5756b76 into apache:main Mar 7, 2024
32 checks passed
@paleolimbot paleolimbot deleted the c-benchmarks branch March 7, 2024 19:19
eddelbuettel pushed a commit to eddelbuettel/arrow-nanoarrow that referenced this pull request Apr 10, 2024
This PR adds an initial set of benchmarks covering some realistic usage
patterns. The general approach is to use doxygen comments to document
the benchmarks, which will run against the released version and the
previous version. I'm not sure exactly what the output format will be
but I'd like the benchmarks to be written in such a way that there's a
path to programatically generating a report (maybe using conbench, maybe
just a Quarto document).

Work in progress!
@paleolimbot paleolimbot added this to the nanoarrow 0.5.0 milestone May 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants