Skip to content

Some benchmarks of different languages

License

Notifications You must be signed in to change notification settings

ColourGrey/benchmarks

 
 

Repository files navigation

Table of Content

Overview

The benchmarks follow the criteria:

  • They are written as the average software developer would write them, i.e.

    • The algorithms are implemented as cited in public sources;
    • The libraries are used as described in the tutorials, documentation and examples;
    • The used data structures are idiomatic.
  • The used algorithms are similar between the languages (as the reference implementations), variants are acceptable if the reference implementation exists.

  • All final binaries are releases (optimized for performance if possible) as debug performance may vary too much depending on the compiler.

My other benchmarks: jit-benchmarks, crystal-benchmarks-game

Measurements

The measured values are:

  • time spent for the benchmark execution (loading required data and code self-testing are not measured);
  • memory consumption of the benchmark process, reported as base + increase, where base is the RSS before the benchmark and increase is the peak increase of the RSS during the benchmark;
  • energy consumption of the CPU package during the benchmark: PP0 (cores) + PP1 (uncores like GPU) + DRAM. Currently, only Intel CPU are supported via the powercap interface.

All values are presented as: median±median absolute deviation.

UPDATE: 2022-05-30

Test Cases

Brainfuck

Testing brainfuck implementations using two code samples (bench.b and mandel.b). Supports two mode:

  • Verbose (default). Prints the output immediately.
  • Quiet (if QUIET environment variable is set). Accumulates the output using Fletcher-16 checksum, and prints it out after the benchmark.

Brainfuck

bench.b

Language Time, s Memory, MiB Energy, J
Racket (Syntax Objects) 1.294±0.001 104.83±00.98 + 0.00±00.00 47.02±00.02
C++/g++ 1.317±0.000 1.65±00.00 + 0.00±00.00 52.77±00.08
Rust 1.589±0.001 2.12±00.03 + 0.00±00.00 63.57±00.08
D/ldc2 1.655±0.000 3.21±00.03 + 0.00±00.00 66.25±00.07
Java 1.655±0.001 37.33±00.46 + 0.90±00.37 64.58±00.05
D/gdc 1.660±0.000 7.49±00.02 + 0.00±00.00 67.55±00.71
C/clang 1.663±0.000 0.69±00.03 + 0.00±00.00 64.54±00.06
C/gcc 1.667±0.000 0.71±00.00 + 0.00±00.00 64.71±00.06
Kotlin/JVM 1.677±0.006 40.74±00.11 + 0.42±00.16 65.50±00.18
C++/clang++ 1.691±0.000 1.48±00.01 + 0.00±00.00 64.32±00.73
Zig 1.747±0.000 0.70±00.00 + 0.00±00.00 68.20±00.38
F#/.NET Core 1.896±0.000 37.86±00.09 + 0.54±00.00 76.35±00.41
C#/.NET Core 1.902±0.000 33.85±00.29 + 0.00±00.00 76.27±00.19
Go 1.904±0.000 2.79±00.08 + 0.00±00.00 72.13±00.49
OCaml 1.912±0.000 2.71±00.02 + 2.30±00.03 85.13±00.36
Chez Scheme 1.923±0.001 24.91±00.08 + 4.40±00.02 79.48±00.05
Nim/gcc 1.976±0.000 2.02±00.02 + 0.00±00.00 78.82±00.74
Racket 2.012±0.010 93.21±00.51 + 0.00±00.00 81.10±00.77
Vala/gcc 2.090±0.001 4.25±00.05 + 0.00±00.00 77.06±00.85
Nim/clang 2.100±0.002 2.43±00.05 + 0.00±00.00 81.52±00.15
Go/gccgo 2.172±0.000 24.55±00.10 + 0.00±00.00 86.43±00.27
Vala/clang 2.262±0.000 4.26±00.07 + 0.00±00.00 82.35±00.19
V/gcc 2.340±0.000 0.66±00.00 + 0.00±00.00 87.35±00.63
Crystal 2.384±0.000 3.46±00.01 + 0.00±00.00 95.11±00.44
MLton 2.608±0.000 0.80±00.03 + 1.01±00.03 102.87±00.28
C#/Mono 2.685±0.001 24.99±00.13 + 0.00±00.00 107.58±00.08
V/clang 2.967±0.019 0.66±00.00 + 0.00±00.00 115.67±00.92
Haskell (MArray) 3.165±0.001 4.18±00.04 + 3.97±00.00 131.78±00.77
Scala 3.222±0.023 66.15±00.25 + 141.06±00.32 133.08±00.82
D/dmd 3.267±0.000 3.76±00.01 + 0.00±00.00 122.89±00.09
Julia 3.565±0.001 208.86±00.13 + 0.99±00.03 134.35±00.18
Node.js 4.016±0.002 37.68±00.02 + 0.85±00.08 161.03±00.35
Swift 5.415±0.000 13.76±00.14 + 0.00±00.00 200.89±02.89
Lua/luajit 6.492±0.008 2.41±00.04 + 0.00±00.00 259.77±00.54
Ruby/truffleruby 7.352±0.037 291.61±00.77 + 396.22±08.35 347.59±02.47
Ruby/truffleruby (--jvm) 8.767±0.078 376.87±16.30 + 458.33±29.58 430.85±06.14
Python/pypy 12.207±0.065 65.14±00.02 + 29.65±00.03 512.74±00.94
Haskell 13.368±0.019 4.31±00.04 + 3.97±00.00 595.87±04.15
Ruby (--jit) 39.635±0.156 270.95±00.02 + 0.02±00.00 1618.92±08.59
Elixir 43.913±0.156 70.50±00.34 + 0.00±00.00 2011.98±10.43
Lua 50.617±0.073 2.30±00.02 + 0.00±00.00 1941.25±23.16
Ruby/jruby 78.965±0.799 183.82±00.75 + 83.58±00.83 3500.74±39.03
Ruby 85.918±0.486 14.41±00.03 + 0.00±00.00 3615.58±32.78
Python 185.860±1.077 10.63±00.00 + 0.00±00.00 7415.67±57.06
Tcl (FP) 239.705±1.971 4.56±00.03 + 0.00±00.00 10232.99±120.09
Perl 324.279±3.019 7.06±00.05 + 0.00±00.00 12967.03±121.22
Tcl (OOP) 492.812±2.162 4.55±00.08 + 0.00±00.00 20801.83±165.07

mandel.b

Mandel in Brainfuck

Language Time, s Memory, MiB Energy, J
C++/g++ 10.461±0.014 1.65±00.00 + 2.34±00.05 429.24±01.97
Racket (Syntax Objects) 14.090±0.116 105.24±00.45 + 72.57±00.64 563.20±06.84
C/gcc 14.203±0.011 0.68±00.03 + 1.05±00.02 561.03±02.60
Kotlin/JVM 14.337±0.030 40.77±00.09 + 1.00±00.14 580.64±03.91
V/gcc 14.519±0.017 0.72±00.00 + 2.08±00.05 589.11±05.55
C++/clang++ 14.656±0.009 1.50±00.02 + 2.09±00.02 588.07±01.37
Rust 14.823±0.008 2.17±00.04 + 0.25±00.00 593.32±00.71
D/ldc2 14.864±0.019 3.20±00.03 + 0.77±00.00 598.33±02.87
D/gdc 15.138±0.015 7.47±00.03 + 0.77±00.00 640.93±01.15
C/clang 15.431±0.010 0.69±00.03 + 1.05±00.04 644.66±01.57
Zig 16.222±0.009 0.70±00.00 + 1.61±00.01 669.85±04.37
Go 17.065±0.259 3.34±00.11 + 1.26±00.00 674.90±08.80
Swift 18.068±0.088 14.62±00.06 + 0.00±00.00 734.16±06.70
Java 18.283±0.012 37.81±00.06 + 1.29±00.13 732.18±00.83
Vala/gcc 18.498±0.004 4.09±00.05 + 2.04±00.02 694.48±01.06
C#/.NET Core 18.529±0.008 34.13±00.07 + 0.83±00.01 751.34±04.79
Crystal 19.010±0.215 3.46±00.01 + 0.38±00.02 778.36±08.01
Nim/clang 19.309±0.305 2.43±00.04 + 0.57±00.00 784.15±16.13
Vala/clang 19.913±0.003 4.08±00.02 + 2.05±00.06 770.85±00.76
Go/gccgo 20.039±0.013 24.59±00.05 + 1.27±00.00 845.96±02.65
Nim/gcc 21.108±0.569 1.95±00.01 + 0.51±00.00 854.87±21.62
Scala 21.395±0.016 66.24±00.05 + 141.19±00.08 883.73±05.55
V/clang 22.592±0.064 0.66±00.00 + 1.89±00.03 931.66±02.36
OCaml 26.203±0.017 3.48±00.02 + 8.08±03.91 1247.53±12.44
Chez Scheme 26.997±0.052 25.38±00.05 + 3.92±00.01 1180.81±03.40
C#/Mono 31.357±0.018 25.04±00.06 + 0.83±00.00 1331.72±01.53
Node.js 31.704±0.292 37.65±00.06 + 5.37±00.02 1317.45±17.77
Lua/luajit 35.127±0.069 2.42±00.01 + 0.50±00.00 1408.43±04.07
Racket 36.927±0.067 92.66±00.27 + 0.00±00.00 1636.94±08.74
D/dmd 37.471±0.015 3.76±00.01 + 0.77±00.00 1377.33±08.82
F#/.NET Core 38.358±0.052 37.99±00.04 + 2.08±00.00 1587.43±03.20
MLton 44.833±0.049 1.59±00.04 + 4.11±00.00 1945.45±09.66
Python/pypy 48.373±0.180 64.76±00.06 + 30.07±00.03 2068.90±24.04
Haskell (MArray) 49.631±0.622 4.12±00.02 + 5.83±00.00 2120.11±20.32
Julia 64.532±0.078 209.49±00.10 + 0.97±00.08 2486.31±13.12
Ruby/truffleruby (--jvm) 78.552±1.319 398.37±13.36 + 440.08±26.60 3295.51±37.63
Ruby/truffleruby 83.372±0.453 291.66±00.26 + 251.66±23.69 3567.05±19.14
Haskell 186.042±2.425 4.34±00.04 + 29.24±00.00 8142.94±109.17

Base64

Testing base64 encoding/decoding of the large blob into the newly allocated buffers.

Base64

Language Time, s Memory, MiB Energy, J
C/clang (aklomp) 0.100±0.001 2.04±00.04 + 0.00±00.00 4.73±00.07
C/gcc (aklomp) 0.101±0.000 2.09±00.02 + 0.00±00.00 4.80±00.05
Rust 0.969±0.000 2.58±00.03 + 0.01±00.00 39.56±00.22
V/clang 0.995±0.001 1.91±00.02 + 0.00±00.00 37.66±00.14
C/clang 0.997±0.000 2.01±00.04 + 0.00±00.00 37.14±00.36
C/gcc 1.012±0.000 2.05±00.04 + 0.00±00.00 37.95±00.09
Nim/clang 1.024±0.001 2.82±00.03 + 4.44±00.00 41.08±00.18
D/ldc2 1.074±0.003 3.58±00.03 + 3.58±00.00 45.02±00.50
Nim/gcc 1.081±0.002 2.31±00.03 + 4.44±00.06 42.48±00.56
V/gcc 1.092±0.000 1.63±00.01 + 0.44±00.04 42.03±00.17
Crystal 1.154±0.002 3.95±00.01 + 1.17±00.02 51.11±00.15
Ruby (--jit) 1.326±0.001 271.35±00.02 + 66.67±00.01 52.24±00.23
Ruby 1.336±0.002 14.83±00.01 + 58.68±00.04 52.41±00.27
Java 1.511±0.008 38.37±00.06 + 258.46±28.01 61.39±00.49
Scala 1.562±0.005 64.37±00.18 + 315.74±04.03 67.26±01.06
Vala/gcc 1.563±0.001 5.49±00.03 + 0.44±00.08 61.08±00.51
Vala/clang 1.563±0.001 5.60±00.04 + 0.30±00.03 60.93±00.56
Kotlin/JVM 1.635±0.007 40.46±00.12 + 263.10±18.51 69.46±01.12
C++/clang++ (libcrypto) 1.725±0.005 4.46±00.05 + 0.68±00.06 68.58±00.43
C++/g++ (libcrypto) 1.726±0.004 5.12±00.09 + 0.68±00.06 68.44±00.24
Perl (MIME::Base64) 1.779±0.014 14.86±00.04 + 0.07±00.00 70.08±00.39
Go 1.852±0.001 4.07±00.06 + 4.57±00.38 78.91±00.38
PHP 2.015±0.022 17.09±00.09 + 0.00±00.00 81.94±00.93
Python 2.115±0.002 10.41±00.06 + 0.18±00.00 78.44±00.28
Node.js 2.234±0.003 36.96±00.03 + 39.88±00.01 90.04±00.67
D/gdc 2.645±0.001 7.62±00.02 + 3.91±00.00 114.17±00.89
D/dmd 2.736±0.001 4.04±00.02 + 3.36±00.00 118.22±00.86
Zig 3.062±0.001 1.47±00.01 + 0.31±00.00 122.30±00.82
F#/.NET Core 3.130±0.065 38.63±00.03 + 18.63±01.03 117.75±01.18
Python/pypy 3.580±0.002 65.49±00.10 + 30.62±00.15 155.88±01.19
Go/gccgo 3.716±0.007 25.42±00.09 + 7.60±00.21 167.47±00.66
C#/.NET Core 3.748±0.081 34.49±00.06 + 20.71±03.36 137.05±01.81
Tcl 3.993±0.005 5.09±00.02 + 0.16±00.00 160.32±00.37
C#/Mono 4.698±0.019 25.75±00.07 + 18.71±00.09 194.78±00.66
Ruby/truffleruby (--jvm) 5.090±0.008 343.44±03.17 + 242.66±37.11 247.93±01.44
Julia 5.095±0.007 228.19±00.26 + 62.22±00.04 190.73±02.14
Ruby/jruby 10.597±0.088 182.58±00.94 + 77.53±03.02 422.40±02.95
Perl (MIME::Base64::Perl) 13.988±0.057 16.16±00.09 + 0.22±00.03 575.68±03.73
Ruby/truffleruby 17.438±0.022 281.42±00.63 + 235.47±06.16 701.31±05.07

Json

Testing parsing and simple calculating of values from a big JSON file.

Few notes:

Json

Language Time, s Memory, MiB Energy, J
C++/g++ (simdjson On-Demand) 0.067±0.000 113.45±00.09 + 59.81±00.00 2.78±00.03
C++/clang++ (simdjson On-Demand) 0.068±0.000 112.55±00.07 + 60.36±00.06 2.83±00.01
C++/clang++ (DAW JSON Link NoCheck) 0.077±0.000 112.44±00.03 + 0.00±00.00 3.14±00.01
C++/g++ (DAW JSON Link NoCheck) 0.079±0.000 113.19±00.03 + 0.00±00.00 3.16±00.02
C++/g++ (DAW JSON Link) 0.087±0.000 113.22±00.07 + 0.00±00.00 3.58±00.03
C++/clang++ (DAW JSON Link) 0.089±0.000 112.44±00.02 + 0.00±00.00 3.67±00.04
Rust (Serde Custom) 0.107±0.000 111.99±00.05 + 0.00±00.00 4.56±00.02
Rust (Serde Typed) 0.112±0.000 111.97±00.07 + 12.03±00.00 4.72±00.01
C++/g++ (gason) 0.133±0.000 113.14±00.05 + 96.80±00.06 5.28±00.03
C++/clang++ (simdjson DOM) 0.136±0.001 112.48±00.03 + 177.15±00.06 5.79±00.03
C++/g++ (simdjson DOM) 0.138±0.000 113.01±00.09 + 176.39±00.19 5.88±00.01
D/ldc2 (Mir Asdf DOM) 0.146±0.000 112.73±00.04 + 61.27±00.03 6.09±00.02
C++/clang++ (gason) 0.149±0.000 112.48±00.03 + 96.97±00.06 6.02±00.01
C++/g++ (RapidJSON) 0.167±0.000 113.26±00.05 + 126.54±00.42 6.93±00.04
C++/clang++ (RapidJSON) 0.221±0.000 112.45±00.00 + 129.02±00.04 8.98±00.04
D/ldc2 (Mir Amazon's Ion DOM) 0.231±0.000 112.72±00.05 + 80.80±00.00 9.50±00.02
C++/g++ (RapidJSON Precise) 0.246±0.000 113.24±00.03 + 128.82±00.06 10.41±00.10
C++/g++ (Boost.JSON) 0.394±0.000 113.22±00.03 + 435.88±00.04 16.44±00.13
Nim/clang (jsony) 0.399±0.000 112.16±00.02 + 42.34±00.03 16.95±00.22
C++/g++ (RapidJSON SAX) 0.404±0.001 112.96±00.04 + 0.00±00.00 16.68±00.07
C++/clang++ (RapidJSON Precise) 0.410±0.000 112.47±00.03 + 129.01±00.02 16.63±00.05
C++/clang++ (Boost.JSON) 0.413±0.000 112.44±00.00 + 436.25±00.06 17.23±00.20
Nim/gcc (jsony) 0.422±0.000 111.72±00.03 + 42.28±00.00 18.09±00.12
C++/g++ (RapidJSON SAX Precise) 0.450±0.001 112.96±00.01 + 0.00±00.00 19.34±00.09
C++/clang++ (RapidJSON SAX) 0.496±0.000 194.69±00.00 + 0.00±00.00 19.89±00.05
Go (jsoniter) 0.519±0.002 230.86±00.02 + 1.07±00.04 21.80±00.07
Node.js 0.548±0.003 147.87±00.07 + 188.65±00.28 25.00±00.14
Java (DSL-JSON) 0.621±0.015 258.17±00.07 + 138.46±00.77 32.18±00.99
Rust (Serde Untyped) 0.633±0.001 111.90±00.03 + 840.05±00.03 25.50±00.07
V/clang 0.635±0.001 111.23±00.05 + 496.21±00.00 26.63±00.29
V/gcc 0.635±0.001 111.25±00.02 + 496.12±00.03 26.51±00.25
C++/clang++ (RapidJSON SAX Precise) 0.658±0.001 194.71±00.03 + 0.00±00.00 26.74±00.12
Python/pypy 0.678±0.001 285.59±00.03 + 124.88±00.00 29.23±00.13
Crystal (Pull) 0.688±0.002 113.86±00.02 + 18.18±00.02 30.53±00.29
Crystal (Schema) 0.694±0.002 113.84±00.00 + 48.57±00.04 30.58±00.36
Julia (JSON3) 0.744±0.004 378.93±00.36 + 378.85±00.20 30.54±00.32
C#/.NET Core (System.Text.Json) 0.759±0.002 479.31±00.03 + 138.58±00.03 33.20±00.31
Zig 0.769±0.001 110.75±00.01 + 12.18±00.00 30.36±00.06
Perl (Cpanel::JSON::XS) 0.833±0.005 125.42±00.05 + 402.78±00.06 34.78±00.29
Go 0.855±0.001 117.10±00.06 + 79.79±00.11 35.57±00.01
Nim/clang (Packedjson) 0.929±0.000 112.50±00.02 + 294.16±00.00 38.21±00.10
Crystal 0.937±0.010 113.88±00.01 + 392.00±00.01 42.07±00.33
PHP 0.984±0.001 126.43±00.10 + 682.01±00.00 41.13±00.11
Nim/gcc (Packedjson) 1.010±0.000 112.00±00.04 + 294.16±00.00 41.38±00.17
C++/clang++ (json-c) 1.203±0.007 112.72±00.03 + 1216.17±00.04 50.96±00.27
C++/g++ (json-c) 1.208±0.009 113.28±00.10 + 1215.96±00.03 50.54±00.57
Clojure 1.217±0.014 450.12±06.93 + 535.88±48.40 64.53±00.76
C++/clang++ (Nlohmann) 1.253±0.002 112.62±00.04 + 360.14±00.02 53.05±00.28
Go/gccgo 1.263±0.003 139.06±00.07 + 83.52±00.15 52.29±00.14
C#/.NET Core 1.294±0.016 487.04±00.06 + 294.45±00.04 56.51±00.63
Nim/clang 1.315±0.002 112.34±00.12 + 925.29±00.10 54.34±00.27
Nim/gcc 1.349±0.002 111.95±00.09 + 919.68±00.06 56.49±00.36
CPython (UltraJSON) 1.440±0.003 121.97±00.01 + 543.94±02.33 53.47±00.22
Ruby (--jit) 1.465±0.002 381.01±00.02 + 262.87±00.01 61.51±00.41
Ruby 1.471±0.007 124.42±00.03 + 262.85±00.01 62.65±00.34
C++/g++ (Nlohmann) 1.476±0.005 113.25±00.04 + 447.92±00.04 60.35±00.21
Python 1.520±0.003 120.41±00.01 + 374.96±00.00 59.77±00.14
F#/.NET Core (System.Text.Json) 1.802±0.005 486.98±00.15 + 384.58±12.24 79.04±00.74
Scala (uPickle) 1.849±0.011 291.27±00.18 + 665.30±48.68 90.05±00.57
Ruby (YAJL) 1.983±0.025 124.21±00.01 + 282.40±00.00 83.42±01.05
D/ldc2 2.085±0.003 113.15±00.02 + 680.09±00.02 86.06±00.41
C#/Mono 2.293±0.021 252.49±00.14 + 31.58±00.02 96.48±01.08
Haskell 2.688±0.004 115.95±00.03 + 716.91±00.26 108.80±00.08
Rust (jq) 2.791±0.006 113.65±00.07 + 778.76±00.90 115.54±00.54
Ruby/jruby 3.113±0.027 442.08±03.76 + 933.59±104.90 157.63±03.53
C++/clang++ (Boost.PropertyTree) 3.126±0.003 194.87±00.02 + 1232.87±00.01 129.76±00.70
C++/g++ (Boost.PropertyTree) 3.338±0.006 113.14±00.03 + 1440.12±00.02 135.06±00.77
D/gdc 3.681±0.009 117.28±00.03 + 680.72±00.08 157.65±00.82
Vala/gcc 3.897±0.009 114.69±00.01 + 940.84±00.03 165.64±00.87
Vala/clang 4.031±0.009 114.69±00.03 + 876.74±00.02 170.91±00.96
D/dmd 4.524±0.007 113.67±00.02 + 680.09±00.02 182.17±00.86
Perl (JSON::Tiny) 9.756±0.035 125.79±00.04 + 528.88±00.06 422.50±03.92
Ruby/truffleruby (--jvm) 16.470±0.253 461.41±25.16 + 2492.61±30.69 1069.49±16.47
Ruby/truffleruby 22.345±0.910 417.31±01.29 + 2336.16±85.83 1080.54±36.02

Matmul

Testing allocating and multiplying matrices.

Matmul

Language Time, s Memory, MiB Energy, J
D/ldc2 (lubeck) 0.046±0.000 6.57±00.04 + 54.95±00.10 4.66±00.03
Nim/gcc (Arraymancer) 0.066±0.002 5.52±00.05 + 54.73±00.11 5.46±00.13
C++/clang++ (Eigen) 0.068±0.003 38.90±02.59 + 51.51±02.58 5.53±00.15
Python (NumPy) 0.070±0.000 28.41±00.12 + 54.96±00.04 6.49±00.04
C++/g++ (Eigen) 0.070±0.002 29.55±08.72 + 60.24±08.76 5.13±00.13
Nim/clang (Arraymancer) 0.072±0.001 6.34±00.04 + 54.76±00.08 6.07±00.07
Java (ND4J) 0.081±0.005 104.24±01.83 + 92.25±00.16 6.35±00.31
Rust (ndarray) 0.090±0.000 2.63±00.04 + 68.48±00.00 6.00±00.03
Julia (threads: 4) 0.191±0.000 235.35±00.21 + 49.23±00.90 15.54±00.02
Julia (threads: 1) 0.510±0.000 235.65±00.18 + 49.17±00.66 21.51±00.03
Julia (no BLAS) 1.046±0.010 225.72±00.30 + 51.62±00.01 46.15±00.39
D/ldc2 1.716±0.001 3.52±00.03 + 70.39±00.00 63.31±00.11
D/gdc 1.868±0.001 7.55±00.04 + 70.43±00.03 73.08±00.12
D/dmd 1.875±0.002 3.63±00.04 + 70.41±00.00 71.06±00.11
C/gcc 3.029±0.000 1.32±00.00 + 68.80±00.04 109.29±00.43
Vala/clang 3.060±0.000 4.10±00.03 + 69.79±00.08 107.55±00.90
Rust 3.065±0.000 2.45±00.05 + 68.57±00.00 105.15±00.17
C/clang 3.067±0.000 1.38±00.01 + 68.77±00.02 105.98±00.38
Zig 3.085±0.001 1.34±00.03 + 68.89±00.00 109.27±00.08
Nim/gcc 3.088±0.001 2.43±00.06 + 67.19±00.67 111.11±00.30
Swift 3.094±0.000 6.49±00.02 + 68.88±00.01 108.15±00.40
Java 3.095±0.000 38.11±00.07 + 68.41±00.13 107.24±00.38
Vala/gcc 3.125±0.001 4.13±00.02 + 69.79±00.07 114.53±00.06
Nim/clang 3.136±0.001 2.88±00.04 + 70.44±00.06 107.79±00.13
V/gcc 3.146±0.000 1.87±00.06 + 68.84±00.00 117.55±01.78
V/clang 3.147±0.000 2.22±00.07 + 68.84±00.00 115.64±00.13
Go 3.152±0.000 3.63±00.04 + 73.18±00.09 113.80±00.18
Go/gccgo 3.159±0.000 24.91±00.09 + 73.53±00.12 112.23±00.10
Crystal 3.161±0.001 4.25±00.02 + 59.67±00.03 115.88±00.10
Node.js 3.213±0.001 41.96±00.13 + 71.22±00.19 129.55±00.09
Python/pypy 3.270±0.001 65.78±00.05 + 68.63±00.01 136.14±00.12
Scala 3.322±0.007 65.01±00.11 + 133.87±00.15 118.54±00.41
C#/.NET Core 4.874±0.001 34.67±00.10 + 69.34±00.00 196.06±00.16
Kotlin/JVM 6.937±0.001 39.71±00.10 + 68.16±00.22 271.50±00.34
C#/Mono 7.413±0.001 25.56±00.12 + 69.54±00.03 299.15±00.74
Ruby/truffleruby 14.604±1.175 351.93±01.70 + 465.25±18.48 553.17±37.67
Ruby/truffleruby (--jvm) 21.393±0.051 437.31±33.21 + 961.78±75.62 804.61±05.21
Ruby 180.926±2.288 15.46±00.03 + 68.58±00.00 7927.39±113.83
Ruby (--jit) 182.285±0.371 272.05±00.05 + 68.84±00.00 7995.18±94.11
Perl 221.133±0.748 9.56±00.04 + 599.55±00.02 9001.25±37.66
Python 261.473±0.827 10.58±00.05 + 68.84±00.00 10445.40±55.25
Tcl 298.785±5.053 7.51±00.04 + 400.15±00.03 12660.24±177.21
Ruby/jruby 404.829±24.918 266.19±01.83 + 779.12±54.18 16982.36±869.82

Primes

Testing:

  • generating primes using the optimized sieve of Atkin;
  • prefix search for their decimal numbers using Trie data structure.

Primes

Language Time, s Memory, MiB Energy, J
Zig 0.059±0.000 0.75±00.00 + 49.41±00.04 2.44±00.01
Rust 0.093±0.000 2.19±00.05 + 77.27±00.00 3.78±00.01
Crystal 0.123±0.000 3.48±00.02 + 89.43±00.64 5.54±00.06
C++/g++ 0.132±0.000 3.54±00.07 + 85.27±00.26 5.36±00.04
C++/clang++ 0.143±0.000 3.08±00.08 + 75.41±00.00 5.63±00.05
V/clang 0.148±0.000 0.72±00.00 + 262.80±00.57 6.15±00.06
V/gcc 0.150±0.000 0.71±00.06 + 259.28±00.48 6.28±00.04
Java 0.155±0.002 36.96±00.11 + 144.23±05.61 8.63±00.15
Node.js 0.256±0.003 36.96±00.11 + 181.19±02.81 12.80±00.06
Lua/luajit 0.337±0.001 2.49±00.03 + 156.75±00.38 12.87±00.03
Scala 0.391±0.005 66.10±00.30 + 258.71±02.41 22.50±00.24
Julia 0.557±0.001 229.65±00.18 + 366.99±02.17 22.00±00.09
Python/pypy 0.880±0.002 64.77±00.07 + 249.36±00.15 34.55±00.17
Ruby (--jit) 1.443±0.005 270.96±00.02 + 146.80±00.04 58.79±00.46
Lua 1.480±0.005 2.32±00.01 + 283.97±00.87 57.87±00.19
Ruby/truffleruby 1.563±0.012 287.18±00.42 + 334.66±24.08 80.19±00.54
Ruby/jruby 1.883±0.026 182.58±00.29 + 370.42±55.17 101.51±01.01
Ruby/truffleruby (--jvm) 1.895±0.018 348.72±09.17 + 526.76±97.23 123.08±01.27
Ruby 2.032±0.004 14.36±00.02 + 146.75±00.00 82.09±00.60
Python 4.806±0.058 10.41±00.06 + 234.92±00.64 190.06±02.19

Tests Execution

Environment

CPU: Intel(R) Xeon(R) E-2324G

Base Docker image: Debian GNU/Linux bookworm/sid

Language Version
.NET Core 6.0.203
C#/.NET Core 4.1.0-5.22128.4 (5d10d428)
C#/Mono 6.8.0.105
Chez Scheme 9.5.4
Clojure "1.11.1"
Crystal 1.4.1
D/dmd v2.100.0
D/gdc 12.1.0
D/ldc2 1.29.0
Elixir 1.12.2
F#/.NET Core 12.0.1.0 for F# 6.0
Go go1.18.2
Go/gccgo 12.1.0
Haskell 9.2.3
Java 18.0.1.1
Julia v"1.7.3"
Kotlin 1.6.21
Lua 5.4.4
Lua/luajit 2.1.0-beta3
MLton 20210117
Nim 1.6.6
Node.js v18.2.0
OCaml 4.14.0
PHP 8.1.5
Perl v5.34.0
Python 3.10.4
Python/pypy 7.3.9-final0 for Python 3.9.12
Racket "8.5"
Ruby 3.1.2p20
Ruby/jruby 9.3.4.0
Ruby/truffleruby 22.1.0
Rust 1.61.0
Scala 3.1.2
Swift 5.6.1
Tcl 8.6
V 0.2.4 a3c0a9b
Vala 0.56.1
Zig 0.9.1
clang/clang++ 13.0.1
gcc/g++ 12.1.0

Using Docker

Build the image:

$ docker build docker/ -t benchmarks

Run the image:

$ docker run -it --rm -v $(pwd):/src benchmarks <cmd>

where <cmd> is:

  • versions (print installed language versions);
  • shell (start the shell);
  • brainfuck bench (build and run Brainfuck bench.b benchmarks);
  • brainfuck mandel (build and run Brainfuck mandel.b benchmarks);
  • base64 (build and run Base64 benchmarks);
  • json (build and run Json benchmarks);
  • matmul (build and run Matmul benchmarks);
  • primes (build and run Primes benchmarks);

Please note that the actual measurements provided in the project are taken semi-manually (via shell) as the full update takes days and could have occassional issues in Docker.

There is a ./run.sh that could be used to simplify Docker usage:

  • ./run.sh build (build the image);
  • ./run.sh make versions (run the image with the versions command);
  • sudo ./run.sh shell (run the image with the `shell' command, sudo is required to read energy levels).

Manual Execution

Makefiles contain recipes for building and executing tests with the proper dependencies. Please use make run (and make run2 where applicable). The measurements are taken using analyze.rb script:

$ cd <test suite>
$ ../analyze.rb make run
$ ../analyze.rb make run[<single test>]

Please note that the measurements could take hours. It uses 10 iterations by default, but it could be changed using ATTEMPTS environment variable:

$ ATTEMPTS=1 ../analyze.rb make run

Prerequisites

Please use Dockerfile as a reference regarding which packages and tools are required.

For all (optional):

  • Powercap for reading energy counters in Linux (Debian package powercap-utils).

For Python:

  • NumPy for matmul tests (Debian package python3-numpy).
  • UltraJSON for JSON tests (Debian package python3-ujson).

For C++:

  • Boost for JSON tests (Debian package libboost-dev).
  • JSON-C for JSON tests (Debian package libjson-c-dev).

For Rust:

  • libjq for jq test (Debian packages libjq-dev, libonig-dev and environment variable JQ_LIB_DIR=/usr/lib/x86_64-linux-gnu/).

For Java, Scala:

  • Coursier for downloading Maven artifacts.

For Haskell:

  • network for TCP connectivity between the tests and the test runner.
  • raw-strings-qq for raw string literals used in tests.

For Perl:

  • cpanminus for installing modules from CPAN (Debian package cpanminus).

For Vala:

  • JSON-GLib for JSON tests (Debian package libjson-glib-dev).

Contribution

Please follow the criteria specified in the overview. Besides that please ensure that the communication protocol between a test and the test runner is satisfied:

  • The test runner listens on localhost:9001;
  • All messages are sent using TCP sockets closed immediately after the message has been sent;
  • There are two messages sent from a test (it establishes the measurement boundary):
    1. The beginning message having the format name of the test/tprocess ID (the process ID is used to measure the memory consumption). Please note that the name of the test couldn't use Tab character as it's a delimiter;
    2. The end message with any content (mostly it's "stop" for consistency).
  • The test runner could be unavailable (if the test is launched as is) and the test should gracefully handle it.

Makefile guide

Binary executables

If the test is compiled into a single binary, then two sections of the Makefile require changes:

  • append a new target (the final binary location) into executables variable;
  • append the proper target rule.

Compiled artifacts

If the test is compiled, but can't be executed directly as a binary, then three sections of the Makefile require changes:

  • append a new target (the final artifact location) into artifacts variable;
  • append the proper target rule to compile the test;
  • append run[<target_artifact>] rule to run the test.

Scripting language

If the test doesn't require compilation, then two sections of the Makefile requires changes:

  • append run[<script_file>] into all_runners variable;
  • append run[<script_file>] rule to run the test.

README update

TOC is regenerated using git-markdown-toc:

./run.sh toc

Docker image update

Debian packages are pinned and updated with the script (first, please ensure that the image is fine with the linter):

./run.sh lint
./run.sh update_apt

About

Some benchmarks of different languages

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Makefile 9.7%
  • C++ 8.2%
  • Ruby 5.5%
  • Rust 4.8%
  • Java 4.7%
  • Zig 4.6%
  • Other 62.5%