Performance / Energy regression in Alpine 3.19.0

[ArneTR]

Hey alpine linux team,

we are working in sustainable software engineering and specializing in open source energy measurement tooling.

We use the alpine docker image for a workload energy validation container we are using.

While doing measurements with it we noticed that for a reference workload the energy consumption went up and also the performance of the workload went down.

The change happened during the transition from alpine:3.18.5 to alpine:3.19.0

We have documented all the details in this repository: https://github.com/green-coding-solutions/alpine-energy-regression/blob/main/README.md

It shows the difference of -1% Performance and +5% energy increase.

We have higlighted this with charts that come from one of our open source tools and also with perf stat commands to replicate.

Super happy for feedback on this and we are also happy to further drill down if you have a component as a suspect that might be the culprit for the energy increase as the Alpine Update contains quite some libraries that we are not too familiar with

@ribalba - Looping you in

[ncopa]

Any idea what is causing this?

Exactly what components are tested in the alpine image? the C library? openssl?

[ArneTR]

The reason for the energy regression is uknown. We have so far only pinned it down to that version number of the alpine docker image.

Under https://github.com/green-coding-solutions/alpine-energy-regression/blob/main/README.md you can see what we are executing. It is this command:
sysbench cpu run --cpu-max-prime=25000 --threads=8 --time=15 --events=0 --rate=0 --debug=off

No syscalls should be done and if you strace the command it holds at a certain point and all further actions happen in userspace, which is the expected behaviour:

...
futex(0x56231280b2f0, FUTEX_WAKE_PRIVATE, 1) = 1
alarm(0)                                = 30
futex(0x762127443910, FUTEX_WAIT_BITSET|FUTEX_CLOCK_REALTIME, 5657, NULL, FUTEX_BITSET_MATCH_ANY
# [No output for around 30 seconds]
futex(0x762126b4f910, FUTEX_WAIT_BITSET|FUTEX_CLOCK_REALTIME, 5659, NULL, FUTEX_BITSET_MATCH_ANY) = 0
alarm(0)                                = 0
write(1, "CPU speed:\n", 11CPU speed:
...

Important note: the strace has been done under ubuntu, which should I hope be representative of the syscall behaviour.

To drill deeper a next step would be a flamegraph, but I thought I open the ticket first to maybe get some insights from you who know the container better and can maybe point in a possible rewarding direction :)

[ArneTR]

Also to add to this: I tried an lsof but got some unexpected results on alpine

# Alpine view
/ # ps
PID   USER     TIME  COMMAND
    1 root      0:00 /sbin/docker-init -- sysbench cpu run --cpu-max-prime=25000 --threads=8 --time=800 --events=0 --rate=0 --debug=off
    7 root      2:52 sysbench cpu run --cpu-max-prime=25000 --threads=8 --time=800 --events=0 --rate=0 --debug=off
   16 root      0:00 ash
   23 root      0:00 ps
/ # lsof -p 7
1	/sbin/docker-init	0	/dev/null
1	/sbin/docker-init	1	pipe:[65798]
1	/sbin/docker-init	2	pipe:[65799]
7	/usr/bin/sysbench	0	/dev/null
7	/usr/bin/sysbench	1	pipe:[65798]
7	/usr/bin/sysbench	2	pipe:[65799]
16	/bin/busybox	0	/dev/pts/0
16	/bin/busybox	1	/dev/pts/0
16	/bin/busybox	2	/dev/pts/0
16	/bin/busybox	10	/dev/tty

So somehow it does not show me the libraries opened.

Under Linux the tool behaves differently and I can see that ssl and libc are loaded. So I guess it will also include ssl libraries under alpine and lsof there just does not show it.

# Ubuntu view
$ sudo lsof -p 6091
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF     NODE NAME
sysbench 6091 arne  cwd    DIR  259,5     4096 20447234 /home/arne
sysbench 6091 arne  rtd    DIR  259,5     4096        2 /
sysbench 6091 arne  txt    REG  259,5   233056  3969323 /usr/bin/sysbench
sysbench 6091 arne  mem    REG  259,5   526896  3960085 /usr/lib/x86_64-linux-gnu/libgmp.so.10.4.1
sysbench 6091 arne  mem    REG  259,5   289800  3960233 /usr/lib/x86_64-linux-gnu/libhogweed.so.6.4
sysbench 6091 arne  mem    REG  259,5   281000  3960459 /usr/lib/x86_64-linux-gnu/libnettle.so.8.4
sysbench 6091 arne  mem    REG  259,5  1743016  3960876 /usr/lib/x86_64-linux-gnu/libunistring.so.2.2.0
sysbench 6091 arne  mem    REG  259,5  2000320  3932373 /usr/lib/x86_64-linux-gnu/libgnutls.so.30.31.0
sysbench 6091 arne  mem    REG  259,5  2260296  3960798 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30
sysbench 6091 arne  mem    REG  259,5    47688  3959961 /usr/lib/x86_64-linux-gnu/libffi.so.8.1.0
sysbench 6091 arne  mem    REG  259,5    92312  3960813 /usr/lib/x86_64-linux-gnu/libtasn1.so.6.6.2
sysbench 6091 arne  mem    REG  259,5   129096  3960275 /usr/lib/x86_64-linux-gnu/libidn2.so.0.3.7
sysbench 6091 arne  mem    REG  259,5  1285888  3960532 /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.3.0
sysbench 6091 arne  mem    REG  259,5   105392  3960724 /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
sysbench 6091 arne  mem    REG  259,5    63992  3936920 /usr/lib/x86_64-linux-gnu/liblber-2.5.so.0.1.12
sysbench 6091 arne  mem    REG  259,5  4455728  3934197 /usr/lib/x86_64-linux-gnu/libcrypto.so.3
sysbench 6091 arne  mem    REG  259,5    22600  3960332 /usr/lib/x86_64-linux-gnu/libkeyutils.so.1.9
sysbench 6091 arne  mem    REG  259,5   182864  3960330 /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
sysbench 6091 arne  mem    REG  259,5   827936  3960338 /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
sysbench 6091 arne  mem    REG  259,5   376512  3936921 /usr/lib/x86_64-linux-gnu/libldap-2.5.so.0.1.12
sysbench 6091 arne  mem    REG  259,5   338648  3960150 /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
sysbench 6091 arne  mem    REG  259,5  2220400  3934299 /usr/lib/x86_64-linux-gnu/libc.so.6
sysbench 6091 arne  mem    REG  259,5   125488  3960024 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
sysbench 6091 arne  mem    REG  259,5   841808  3961075 /usr/lib/x86_64-linux-gnu/libzstd.so.1.4.8
sysbench 6091 arne  mem    REG  259,5   940560  3934304 /usr/lib/x86_64-linux-gnu/libm.so.6
sysbench 6091 arne  mem    REG  259,5  6834168  3969319 /usr/lib/x86_64-linux-gnu/libmysqlclient.so.21.2.36
sysbench 6091 arne  mem    REG  259,5    52016  3960340 /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
sysbench 6091 arne  mem    REG  259,5    18504  3959801 /usr/lib/x86_64-linux-gnu/libcom_err.so.2.1
sysbench 6091 arne  mem    REG  259,5    14432  3934302 /usr/lib/x86_64-linux-gnu/libdl.so.2
sysbench 6091 arne  mem    REG  259,5    68552  3934347 /usr/lib/x86_64-linux-gnu/libresolv.so.2
sysbench 6091 arne  mem    REG  259,5   667864  3934198 /usr/lib/x86_64-linux-gnu/libssl.so.3
sysbench 6091 arne  mem    REG  259,5    14456  3969315 /usr/lib/x86_64-linux-gnu/libaio.so.1.0.1
sysbench 6091 arne  mem    REG  259,5   539424  3969317 /usr/lib/x86_64-linux-gnu/libluajit-5.1.so.2.1.0
sysbench 6091 arne  mem    REG  259,5   329912  3969321 /usr/lib/x86_64-linux-gnu/libpq.so.5.14
sysbench 6091 arne  mem    REG  259,5   240936  3934293 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
sysbench 6091 arne    0u   CHR  136,1      0t0        4 /dev/pts/1
sysbench 6091 arne    1u   CHR  136,1      0t0        4 /dev/pts/1
sysbench 6091 arne    2u   CHR  136,1      0t0        4 /dev/pts/1