A lightweight benchmarking framework primarily aimed at Linux kernel scheduler testing. It provides a reasonable coverage within one night time, to compare the difference between a baseline kernel and a testing kernel.
- hackbench (apt install rt-tests)
- netperf (apt install netperf)
- tbench (apt install dbench)
- schbench (https://git.kernel.org/pub/scm/linux/kernel/git/mason/schbench.git)
Note, schbench is expected to be installed at /usr/bin.
- python3 (apt install python3.x)
- numpy (pip3 install numpy)
- pandas (pip3 install pandas)
- mutt (apt install mutt)
- msmtp (apt install msmtp)
mutt and msmtp is expected to be properly configured.
There are 3 global variables to configure:
- joblist: load levels of a benchmark case
- runtime: how long to run a benchmark case before exiting
- iterations: how many times to run a benchmark case
The job list is specified by [50% 100% 200%] number of CPUs in the system by default, This list is not accurately mapped to the system utilization and can be adjusted individually for benchmarks. For example:
- hackbench_job_list="2 3 4" #number of send/recv groups
- schbench_jost_list="2 4 8" #number of message threads
Ususally, runtime >= 100 seconds and iterations >= 5 is the compliant setting for the valid result.
It is recommended to run the script as root to avoid unexpected issues.
Run all benchmarks in one time:
- #./run-schedtests.sh
Run one benchmark in one time:
- #./run-schedtests.sh hackbench
The global variable $run_name is assigned to the kernel release string by default.
To eliminate the interaction between different benchmarks, it is recommended to restart the system between two benchmarks.
Place cron.sh into "crontab -e" as follows:
@reboot /home/aubrey/work/schedtests/cron.sh
It will run at boot time and automatically run all benchmarks, system restart between the two benchmarks is done by kexec by default.
Optionally, if the variable $email_address is set, cron.sh will send out the automation progress notification.
Once the testing has completed both on the baseline kernel and the compared kernel, the report is generated by:
#./report.py --testname tbench --baseline 5.11.0-rc7-upstream
tbench
======
case load Tput/s std%
loopback thread-96 21540.86 ( 0.85)
loopback thread-192 13383.75 ( 0.57)
loopback thread-384 25403.73 ( 6.47)
#./report.py --testname tbench --baseline 5.10.13-stable --compare 5.11.0-rc7-upstream
tbench
======
case load baseline(std%) compare%( std%)
loopback thread-96 1.00 ( 0.64) -4.98 ( 0.85)
loopback thread-192 1.00 ( 0.50) -0.56 ( 0.57)
loopback thread-384 1.00 ( 7.40) -9.38 ( 6.47)
#./report.py --baseline 5.10.13-stable --compare 5.11.0-rc7-upstream
hackbench
=========
case load baseline(std%) compare%( std%)
process-pipe group-6 1.00 ( 1.00) +0.85 ( 1.50)
process-pipe group-12 1.00 ( 0.89) +0.15 ( 0.72)
process-pipe group-24 1.00 ( 12.09) -3.65 ( 9.76)
process-sockets group-6 1.00 ( 0.41) +1.06 ( 0.30)
process-sockets group-12 1.00 ( 0.56) +0.81 ( 0.34)
process-sockets group-24 1.00 ( 0.39) +2.61 ( 0.22)
threads-pipe group-6 1.00 ( 2.18) -0.24 ( 2.76)
threads-pipe group-12 1.00 ( 1.25) +0.38 ( 0.55)
threads-pipe group-24 1.00 ( 3.86) -4.64 ( 4.21)
threads-sockets group-6 1.00 ( 0.71) -0.19 ( 0.70)
threads-sockets group-12 1.00 ( 0.47) +1.13 ( 0.35)
threads-sockets group-24 1.00 ( 0.33) +2.43 ( 0.25)
netperf
=======
case load baseline(std%) compare%( std%)
TCP_RR thread-96 1.00 ( 5.14) -4.35 ( 6.15)
TCP_RR thread-192 1.00 ( 6.48) -3.51 ( 6.05)
TCP_RR thread-384 1.00 ( 15.35) -10.73 ( 11.95)
UDP_RR thread-96 1.00 ( 8.99) -4.95 ( 12.41)
UDP_RR thread-192 1.00 ( 26.66) -2.61 ( 33.05)
UDP_RR thread-384 1.00 ( 27.27) -0.89 ( 29.88)
tbench
======
case load baseline(std%) compare%( std%)
loopback thread-96 1.00 ( 0.64) -4.98 ( 0.85)
loopback thread-192 1.00 ( 0.50) -0.56 ( 0.57)
loopback thread-384 1.00 ( 7.40) -9.38 ( 6.47)
schbench
========
case load baseline(std%) compare%( std%)
normal mthread-6 1.00 ( 30.16) +52.75 ( 34.93)
normal mthread-12 1.00 ( 12.22) +3.15 ( 7.32)
normal mthread-24 1.00 ( 6.30) -2.38 ( 4.99)