Skip to content

awadell1/PkgJogger.jl

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

PkgJogger

Stable Dev Build Status Coverage Status

version pkgeval Downloads

PkgJogger provides a framework for running suites of BenchmarkTools.jl benchmarks without the boilerplate.

Just write benchmarks

Create a benchmark/bench_*.jl file, define a BenchmarkTools.jl suite and go!

using BenchmarkTools
using AwesomePkg
suite = BenchmarkGroup()
suite["fast"] = @benchmarkable fast_code()

PkgJogger will wrap each benchmark/bench_*.jl in a module and bundle them into JogAwesomePkg

using AwesomePkg
using PkgJogger

# Creates the JogAwesomePkg module
@jog AwesomePkg

# Warmup, tune, and run all of AwesomePkg's benchmarks
JogAwesomePkg.benchmark()

Benchmark, Revise, and Benchmark Again!

PkgJogger uses Revise.jl to track changes to your benchmark/bench_*.jl files and reload your suite as you edit. No more waiting for benchmarks to precompile!

Tracked Changes:

  • Changing your benchmarked function
  • Changing benchmarking parameters (i.e. seconds or samples)
  • Adding new benchmarks

Current Limitations:

  • New benchmark files are not tracked
  • Deleted benchmarks will stick around
  • Renamed benchmarks will create a new benchmark and retain the old name

To get around the above, run @jog PkgName to get an updated jogger.

Note: Revise must be loaded first in order to track changes to your benchmark files.

Continuous Benchmarking Baked In!

Install PkgJogger, run benchmarks, and save results to a *.bson.gz with a one-line command.

julia -e 'using Pkg; Pkg.add("PkgJogger"); using PkgJogger; PkgJogger.ci()'

What gets done:

  • Constructs a temporary benchmarking environment from Project.toml and benchmark/Project.toml.
  • Creates a jogger to run the package's benchmarks.
  • Warmup, tune and run all benchmarks.
  • Save Benchmarking results and more to a compressed *.bson.gz file.

Or for a more lightweight option, use @test_bechmarks to run each benchmark once (No Warmup, tuning, etc.), as a smoke test against benchmarking regressions.