compare the system llvm against "big-merge" and "pgo"

[kwk]

This container setup allows you to compare the system llvm against "big-merge" and "pgo".

How to

Just run make to build and run the container image. It takes a long time to complete.

Then you'll be promted to a terminal in the container where you'll find these files:

~/results-system-vs-pgo.txt
~/results-system-vs-big-merge.txt
~/results-big-merge-vs-pgo.txt

The names speak for themselves.

How to change to OS

If you want to change the version of the operating system, go to Containerfile and change the line that looks like this: FROM fedora:40. Change it to FROM fedora:41 or something else.

Then run make again.

How to change the date for which to compare results?

Go to entrypoint.sh and change the line that defines yyyymmdd to the year-month-date of your liking.

[kwk]

big-merge vs pgo

According to the llvm-test-suite the execution time of PGOed LLVM is 5.9% faster but 12% slower compile time and even worse 24% slower link time. This is for 20240825. I wonder how the comparison looks against the system compiler and if the results are really that bad every time.

Program                                       exec_time              compile_time              link_time             
                                              big-merge pgo    diff  big-merge    pgo    diff  big-merge pgo    diff 
SingleSource/UnitTests/testcase-Value-1                   0.00  inf%                0.02  inf%             0.02  inf%
Bitcode/Be...ral_grid/halide_bilateral_grid              15.93  inf%                0.00                   0.10  inf%
SingleSource/UnitTests/testcase-Expr-1                    0.00  inf%                0.02  inf%             0.02  inf%
SingleSour...tTests/testcase-CGExprConstant               0.00  inf%                0.02  inf%             0.02  inf%
SingleSour...nitTests/ms_struct_pack_layout               0.00  inf%                0.01  inf%             0.01  inf%
SingleSour...tTests/ms_struct_pack_layout-1               0.00  inf%                0.01  inf%             0.03  inf%
SingleSour...UnitTests/ms_struct-bitfield-1               0.00  inf%                0.01  inf%             0.02  inf%
SingleSour...Tests/member-function-pointers               0.00  inf%                0.02  inf%             0.06  inf%
SingleSour...ce/UnitTests/matrix-types-spec               0.00  inf%               13.72  inf%             0.05  inf%
SingleSource/UnitTests/initp1                             0.00  inf%                0.04  inf%             0.06  inf%
SingleSour...nitTests/2006-12-01-float_varg               0.00  inf%                0.02  inf%             0.02  inf%
SingleSour...Tests/2006-12-11-LoadConstants               0.00  inf%                0.07  inf%             0.02  inf%
SingleSour.../UnitTests/2008-04-20-LoopBug2               0.00  inf%                0.02  inf%             0.02  inf%
SingleSour...tTests/2008-07-13-InlineSetjmp               0.00  inf%                0.03  inf%             0.02  inf%
SingleSour...9-04-16-BitfieldInitialization               0.00  inf%                0.02  inf%             0.02  inf%
                           Geomean difference                  -5.9%                     12.0%                  24.3%
       exec_time                            compile_time                           link_time                          
l/r    big-merge           pgo         diff    big-merge          pgo        diff  big-merge          pgo         diff
count  40.000000  3.286000e+03  2450.000000  40.000000    2440.000000  446.000000  40.000000  2440.000000  2440.000000
mean   0.369140   1.512689e+03  inf          0.001442     0.205362     inf         0.074233   0.031629     inf        
std    1.868719   3.203588e+04 NaN           0.009123     1.697427    NaN          0.009413   0.025592    NaN         
min    0.003700   0.000000e+00 -0.577922     0.000000     0.000000     0.119584    0.059200   0.009700    -0.217391   
25%    0.006375   0.000000e+00 NaN           0.000000     0.000000    NaN          0.066775   0.016900    NaN         
50%    0.007300   8.000000e-04 NaN           0.000000     0.000000    NaN          0.073850   0.019500    NaN         
75%    0.008200   2.742774e+00 NaN           0.000000     0.000000    NaN          0.079550   0.025600    NaN         
max    11.552100  1.185607e+06  inf          0.057700     56.370900    inf         0.101200   0.174000     inf

System vs PGO

It looks like the PGOed LLVM 20 version is 3.9% slower in terms of compile time compared to the Fedora 40 system LLVM version 18.

Program                                       exec_time                                                compile_time                                               link_time                                             
                                              18.1.6    pgo-20.0.0~pre20240825.g3ef64f7ab5b865 diff    18.1.6       pgo-20.0.0~pre20240825.g3ef64f7ab5b865 diff   18.1.6    pgo-20.0.0~pre20240825.g3ef64f7ab5b865 diff 
UnitTests/initp1                                0.00      0.00                                    inf%   0.04         0.04                                   2.8%   0.07      0.06                                 -7.5%
UnitTests/testcase-CGExprConstant               0.00      0.00                                    inf%   0.02         0.02                                 -14.8%   0.02      0.02                                  3.9%
UnitTests/testcase-Value-1                      0.00      0.00                                    inf%   0.02         0.02                                 -27.0%   0.02      0.02                                 -2.6%
UnitTests/testcase-Expr-1                       0.00      0.00                                    inf%   0.01         0.02                                   6.7%   0.02      0.02                                 -9.4%
Regression...ecute/GCC-C-execute-20000511-1     0.00      0.00                                    inf%   0.00         0.00                                          0.01      0.01                                 13.0%
UnitTests/2006-01-23-UnionInit                  0.00      0.00                                    inf%   0.02         0.02                                  22.7%   0.02      0.02                                 -8.9%
UnitTests/conditional-gnu-ext                   0.00      0.00                                    inf%   0.01         0.01                                 -12.7%   0.02      0.02                                 -2.2%
Regression.../execute/GCC-C-execute-pr21173     0.00      0.00                                    inf%   0.00         0.00                                          0.02      0.02                                 14.7%
Regression...xecute/GCC-C-execute-pr21964-1     0.00      0.00                                    inf%   0.00         0.00                                          0.02      0.02                                 16.6%
Regression...xecute/GCC-C-execute-pr22098-2     0.00      0.00                                    inf%   0.00         0.00                                          0.01      0.01                                 -7.3%
Regression.../execute/GCC-C-execute-pr19606     0.00      0.00                                    inf%   0.00         0.00                                          0.01      0.02                                 31.2%
Regression...execute/GCC-C-execute-memset-3     0.00      0.00                                    inf%   0.00         0.00                                          0.02      0.02                                 30.5%
Regression...execute/GCC-C-execute-memset-4     0.00      0.00                                    inf%   0.00         0.00                                          0.02      0.02                                 16.7%
Regression...execute-mode-dependent-address     0.00      0.00                                    inf%   0.00         0.00                                          0.02      0.02                                 28.4%
Regression...ecute/GCC-C-execute-nestfunc-4     0.00      0.00                                    inf%   0.00         0.00                                          0.02      0.02                                 -3.4%
                           Geomean difference                                                  -100.0%                                                       3.9%                                                   0.6%
          exec_time                                                     compile_time                                                       link_time                                                    
l/r          18.1.6 pgo-20.0.0~pre20240825.g3ef64f7ab5b865         diff       18.1.6 pgo-20.0.0~pre20240825.g3ef64f7ab5b865        diff       18.1.6 pgo-20.0.0~pre20240825.g3ef64f7ab5b865         diff
count  3.286000e+03  3.286000e+03                           2876.000000  2440.000000  2440.000000                            446.000000  2440.000000  2440.000000                            2440.000000
mean   1.514335e+03  1.512689e+03                           inf          0.199272     0.205362                               0.058021    0.031292     0.031629                               0.030921   
std    3.216381e+04  3.203588e+04                          NaN           1.681302     1.697427                               0.201930    0.024942     0.025592                               0.235836   
min    0.000000e+00  0.000000e+00                          -1.000000     0.000000     0.000000                              -0.735994    0.010600     0.009700                              -0.513393   
25%    0.000000e+00  0.000000e+00                          -0.110011     0.000000     0.000000                              -0.044733    0.016700     0.016900                              -0.129583   
50%    8.000000e-04  8.000000e-04                           0.000000     0.000000     0.000000                               0.037552    0.019400     0.019500                               0.000000   
75%    2.726353e+00  2.742774e+00                           0.125317     0.000000     0.000000                               0.143569    0.026325     0.025600                               0.155633   
max    1.216553e+06  1.185607e+06                           inf          57.401000    56.370900                              1.108333    0.146900     0.174000                               1.288136   

System vs. big-merge

Program                                       exec_time                                                    compile_time                                                    link_time                                                    
                                              18.1.6    big-merge-20.0.0~pre20240825.g3ef64f7ab5b865 diff  18.1.6       big-merge-20.0.0~pre20240825.g3ef64f7ab5b865 diff  18.1.6    big-merge-20.0.0~pre20240825.g3ef64f7ab5b865 diff  
maxpd_210                                       0.01      0.01                                       50.9%   0.00         0.00                                               0.10      0.07                                       -35.4%
addps_69                                        0.00      0.01                                       49.0%   0.00         0.00                                               0.07      0.08                                         9.5%
maxps_123                                       0.01      0.01                                       45.0%   0.00         0.00                                               0.07      0.08                                        17.8%
addpd_220                                       0.01      0.01                                       33.9%   0.00         0.00                                               0.08      0.10                                        24.2%
cmpltps_89                                      0.01      0.01                                       33.3%   0.00         0.00                                               0.08      0.07                                       -12.5%
cmpeqps_136                                     0.01      0.01                                       32.3%   0.00         0.00                                               0.06      0.07                                        14.1%
cmpeqps_88                                      0.01      0.01                                       30.4%   0.00         0.00                                               0.09      0.06                                       -33.3%
addpd_205                                       0.01      0.01                                       27.5%   0.00         0.00                                               0.08      0.08                                         1.9%
cmpltps_185                                     0.01      0.01                                       26.2%   0.00         0.00                                               0.07      0.07                                        -5.5%
cmpeqps_184                                     0.01      0.01                                       16.2%   0.00         0.00                                               0.08      0.07                                       -21.0%
maxpd_225                                       0.01      0.01                                       15.9%   0.00         0.00                                               0.07      0.07                                         0.1%
maxpd_195                                       0.01      0.01                                       15.5%   0.00         0.00                                               0.07      0.07                                        -0.3%
cmpeqps_40                                      0.01      0.01                                       14.3%   0.00         0.00                                               0.07      0.08                                        13.8%
cmpltpd_213                                     0.01      0.01                                       13.4%   0.00         0.00                                               0.07      0.09                                        29.0%
blendvps_299                                    0.01      0.01                                       12.7%   0.00         0.00                                               0.07      0.08                                        14.2%
                           Geomean difference                                                         2.9%                                                            2.5%                                                         -3.1%
          exec_time                                                           compile_time                                                             link_time                                                          
l/r          18.1.6 big-merge-20.0.0~pre20240825.g3ef64f7ab5b865         diff       18.1.6 big-merge-20.0.0~pre20240825.g3ef64f7ab5b865        diff       18.1.6 big-merge-20.0.0~pre20240825.g3ef64f7ab5b865         diff
count  3.286000e+03  40.000000                                    2452.000000  2440.000000  40.000000                                    446.000000  2440.000000  40.000000                                    2440.000000
mean   1.514335e+03  0.369140                                    -0.982795     0.199272     0.001442                                    -0.997702    0.031292     0.074233                                    -0.983877   
std    3.216381e+04  1.868719                                     0.136631     1.681302     0.009123                                     0.048529    0.024942     0.009413                                     0.126728   
min    0.000000e+00  0.003700                                    -1.000000     0.000000     0.000000                                    -1.000000    0.010600     0.059200                                    -1.000000   
25%    0.000000e+00  0.006375                                    -1.000000     0.000000     0.000000                                    -1.000000    0.016700     0.066775                                    -1.000000   
50%    8.000000e-04  0.007300                                    -1.000000     0.000000     0.000000                                    -1.000000    0.019400     0.073850                                    -1.000000   
75%    2.726353e+00  0.008200                                    -1.000000     0.000000     0.000000                                    -1.000000    0.026325     0.079550                                    -1.000000   
max    1.216553e+06  11.552100                                    0.509091     57.401000    0.057700                                     0.024867    0.146900     0.101200                                     0.325163   

[kwk]

Experiments are done in an f41 container on x86_64 this time and the PGO profile generation was done using the llvm-test-suite instead of hello-world.

Here are the result files:

results-pgo.json
results-big-merge.json
results-system.json

big-merge vs pgo

Tests: 3307
Metric: exec_time,compile_time,link_time

Program                                       exec_time                compile_time               link_time              
                                              big-merge pgo    diff    big-merge    pgo    diff   big-merge pgo    diff  
Regression...ecute/GCC-C-execute-20060110-2     0.00      0.00    inf%   0.00         0.00          0.02      0.02  14.5%
Regression...ecute/GCC-C-execute-20070212-1     0.00      0.00    inf%   0.00         0.00          0.02      0.02  -3.0%
Regression...ecute/GCC-C-execute-20061101-2     0.00      0.00    inf%   0.00         0.00          0.02      0.02  -6.0%
Regression...ecute/GCC-C-execute-20050316-3     0.00      0.00    inf%   0.00         0.00          0.02      0.02  -4.2%
Regression...ecute/GCC-C-execute-20060110-1     0.00      0.00    inf%   0.00         0.00          0.03      0.02 -12.1%
Regression...execute/GCC-C-execute-920711-1     0.00      0.00    inf%   0.00         0.00          0.02      0.02   7.0%
Regression...execute/GCC-C-execute-920726-1     0.00      0.00    inf%   0.00         0.00          0.03      0.02 -34.3%
Regression...execute/GCC-C-execute-921029-1     0.00      0.00    inf%   0.00         0.00          0.03      0.02 -32.1%
Regression...execute/GCC-C-execute-921112-1     0.00      0.00    inf%   0.00         0.00          0.02      0.02   2.7%
Regression...execute/GCC-C-execute-921113-1     0.00      0.00    inf%   0.00         0.00          0.03      0.02 -24.1%
UnitTests/2020-01-06-coverage-007               0.00      0.00    inf%   0.02         0.02 -12.5%   0.02      0.02  -7.1%
UnitTests/2020-01-06-coverage-008               0.00      0.00    inf%   0.02         0.02 -24.3%   0.02      0.03  56.6%
Regression...execute/GCC-C-execute-921117-1     0.00      0.00    inf%   0.00         0.00          0.02      0.02 -11.7%
Regression...execute/GCC-C-execute-920829-1     0.00      0.00    inf%   0.00         0.00          0.02      0.02  14.9%
Regression...execute/GCC-C-execute-920908-1     0.00      0.00    inf%   0.00         0.00          0.02      0.02   7.8%
                           Geomean difference                  -100.0%                      -3.5%                   -8.4%
          exec_time                            compile_time                             link_time                          
l/r       big-merge           pgo         diff    big-merge          pgo        diff    big-merge          pgo         diff
count  3.286000e+03  3.286000e+03  2834.000000  2440.000000  2440.000000  446.000000  2440.000000  2440.000000  2440.000000
mean   1.511797e+03  1.512998e+03  inf          0.211009     0.206813    -0.022002    0.052811     0.049112    -0.052499   
std    3.205980e+04  3.208751e+04 NaN           1.764539     1.701700     0.158948    0.060011     0.057628     0.249021   
min    0.000000e+00  0.000000e+00 -1.000000     0.000000     0.000000    -0.504132    0.008900     0.010500    -0.704735   
25%    0.000000e+00  0.000000e+00 -0.080478     0.000000     0.000000    -0.104550    0.018900     0.017700    -0.226248   
50%    9.000000e-04  8.000000e-04  0.000000     0.000000     0.000000    -0.012053    0.022700     0.020900    -0.075171   
75%    2.726378e+00  2.701116e+00  0.142857     0.000000     0.000000     0.059316    0.041350     0.033300     0.089655   
max    1.207388e+06  1.203894e+06  inf          60.357200    56.157900    0.718750    0.265800     0.283400     1.683544

System vs PGO

Tests: 3307
Metric: exec_time,compile_time,link_time

Program                                       exec_time                compile_time              link_time              
                                              18.1.8    pgo    diff    18.1.8       pgo    diff  18.1.8    pgo    diff  
UnitTests/2007-01-04-KNR-Args                   0.00      0.00    inf%   0.01         0.02 17.9%   0.02      0.02  17.8%
UnitTests/SignlessTypes/ccc                     0.00      0.00    inf%   0.02         0.02  0.9%   0.02      0.03  61.8%
UnitTests/Vector/Vector-sumarray-dbl            0.00      0.00    inf%   0.00         0.00         0.02      0.02   1.0%
UnitTests/Vector/Vector-sumarray                0.00      0.00    inf%   0.00         0.00         0.03      0.02 -17.9%
UnitTests/conditional-gnu-ext-cxx               0.00      0.00    inf%   0.01         0.02 18.0%   0.12      0.13   7.8%
Regression.../execute/GCC-C-execute-pr43385     0.00      0.00    inf%   0.00         0.00         0.02      0.02  -0.4%
Regression.../execute/GCC-C-execute-pr43560     0.00      0.00    inf%   0.00         0.00         0.02      0.02  17.3%
Regression...xecute/GCC-C-execute-pr42269-2     0.00      0.00    inf%   0.00         0.00         0.02      0.02  -5.6%
Regression.../execute/GCC-C-execute-pr42544     0.00      0.00    inf%   0.00         0.00         0.03      0.02 -40.7%
Regression.../execute/GCC-C-execute-pr42721     0.00      0.00    inf%   0.00         0.00         0.02      0.02   5.3%
Regression...ieee/GCC-C-execute-ieee-mzero2     0.00      0.00    inf%   0.00         0.00         0.02      0.02  -9.9%
Regression...e/GCC-C-execute-ieee-fp-cmp-4e     0.00      0.00    inf%   0.00         0.00         0.02      0.02 -25.3%
Regression...ee/GCC-C-execute-ieee-fp-cmp-5     0.00      0.00    inf%   0.00         0.00         0.02      0.02  11.3%
Regression...CC-C-execute-ieee-compare-fp-2     0.00      0.00    inf%   0.00         0.00         0.02      0.02  23.2%
Regression...e/GCC-C-execute-ieee-copysign2     0.00      0.00    inf%   0.00         0.00         0.03      0.02 -15.7%
                           Geomean difference                  -100.0%                      1.9%                  -10.6%
          exec_time                            compile_time                             link_time                          
l/r          18.1.8           pgo         diff       18.1.8          pgo        diff       18.1.8          pgo         diff
count  3.286000e+03  3.286000e+03  2851.000000  2440.000000  2440.000000  446.000000  2440.000000  2440.000000  2440.000000
mean   1.470319e+03  1.512998e+03  inf          0.205977     0.206813     0.030543    0.053702     0.049112    -0.072315   
std    3.127929e+04  3.208751e+04 NaN           1.730736     1.701700     0.150352    0.060762     0.057628     0.255466   
min    0.000000e+00  0.000000e+00 -1.000000     0.000000     0.000000    -0.768002    0.008900     0.010500    -0.739558   
25%    0.000000e+00  0.000000e+00 -0.125000     0.000000     0.000000    -0.046781    0.019600     0.017700    -0.250760   
50%    8.000000e-04  8.000000e-04  0.000000     0.000000     0.000000     0.024190    0.024100     0.020900    -0.101203   
75%    2.718482e+00  2.701116e+00  0.166667     0.000000     0.000000     0.098251    0.036325     0.033300     0.063237   
max    1.210631e+06  1.203894e+06  inf          58.378900    56.157900    0.682796    0.327400     0.283400     1.786517

System vs big-merge

Tests: 3307
Metric: exec_time,compile_time,link_time

Program                                       exec_time                   compile_time                  link_time                 
                                              18.1.8    big-merge diff    18.1.8       big-merge diff   18.1.8    big-merge diff  
2007-03-02-VaCopy                               0.00      0.00       inf%   0.01         0.02     51.1%   0.02      0.02    -13.3%
2007-01-04-KNR-Args                             0.00      0.00       inf%   0.01         0.02     42.1%   0.02      0.02      6.3%
testcase-Value-1                                0.00      0.00       inf%   0.02         0.02     -3.7%   0.02      0.02     -3.4%
2006-12-01-float_varg                           0.00      0.00       inf%   0.02         0.01    -11.9%   0.02      0.02      8.5%
2006-01-23-UnionInit                            0.00      0.00       inf%   0.02         0.02     21.0%   0.02      0.02    -14.0%
stmtexpr                                        0.00      0.00       inf%   0.01         0.02      4.0%   0.11      0.13     19.8%
ms_struct-bitfield-init-1                       0.00      0.00       inf%   0.02         0.02     18.6%   0.02      0.02    -14.2%
2020-01-06-coverage-003                         0.00      0.00       inf%   0.02         0.02     11.8%   0.02      0.02     16.2%
2020-01-06-coverage-002                         0.00      0.00       inf%   0.02         0.02      9.0%   0.02      0.02      2.6%
2020-01-06-coverage-001                         0.00      0.00       inf%   0.02         0.02     10.1%   0.02      0.03     40.3%
2005-07-17-INT-To-FP                            0.00      0.00       inf%   0.02         0.02    -28.6%   0.02      0.02    -15.3%
conditional-gnu-ext                             0.00      0.00       inf%   0.01         0.02     31.4%   0.03      0.03     10.4%
SignlessTypes/rem                               0.00      0.00       inf%   0.10         0.10     -1.0%   0.03      0.02    -16.3%
2020-01-06-coverage-005                         0.00      0.00       inf%   0.02         0.02     54.8%   0.02      0.02     -6.2%
2020-01-06-coverage-006                         0.00      0.00       inf%   0.02         0.02     31.2%   0.02      0.03     79.9%
                           Geomean difference                     -100.0%                          5.6%                      -2.3%
          exec_time                            compile_time                             link_time                          
l/r          18.1.8     big-merge         diff       18.1.8    big-merge        diff       18.1.8    big-merge         diff
count  3.286000e+03  3.286000e+03  2808.000000  2440.000000  2440.000000  446.000000  2440.000000  2440.000000  2440.000000
mean   1.470319e+03  1.511797e+03  inf          0.205977     0.211009     0.073066    0.053702     0.052811     0.014390   
std    3.127929e+04  3.205980e+04 NaN           1.730736     1.764539     0.198307    0.060762     0.060011     0.288643   
min    0.000000e+00  0.000000e+00 -1.000000     0.000000     0.000000    -0.769015    0.008900     0.008900    -0.619658   
25%    0.000000e+00  0.000000e+00 -0.125000     0.000000     0.000000    -0.030729    0.019600     0.018900    -0.185247   
50%    8.000000e-04  9.000000e-04  0.000000     0.000000     0.000000     0.035385    0.024100     0.022700    -0.029251   
75%    2.718482e+00  2.726378e+00  0.143522     0.000000     0.000000     0.117019    0.036325     0.041350     0.162980   
max    1.210631e+06  1.207388e+06  inf          58.378900    60.357200    1.156566    0.327400     0.265800     1.471861