This repository is the companion Web page for the ICSE 2023 submission "Twins or False Friends? A Study on Energy Consumption and Performance of Configurable Software".
The setup for the fast energy metering setup can be found in folder fast-energy-measurements.
The data to all subject systems are included in the directory data. In each of the case study directories, you will find the following files:
- FeatureModel.xml: The feature model containing the description of the configuration options that we measured.
- measurements.csv: A csv file containing all configurations and the measured performance and energy consumption values.
- measurements_ml.csv: A csv file containing all configurations and the measured performance and energy consumption values for all functions of the systems (for the systems brotli, lrzip, x264).
- [system]_prof_configurations.cfg: A text file containing all profiled configurations for the systems (brotli, lrzip, x264).
We used the given data for our empirical evaluation.
The results of the literature study data can be found in the literature directory. We also provide the table as Google Docs.
All scripts for re-running our empirical evaluation are included in the scripts directory.
- The analysis directory contains python scripts which generate, based on the measurement data in the data directory, all scatter plots and window plots for RQ1 and RQ2.
- The notebooks directory contains jupyter notebooks that generates all tables, correlation plots, and regression plots for all RQs. The requirements can be used to initialize a virtual environment for python (
python3 -m pip install -r requirements.txt) with the same software that was used in the evaluation.
For reproducing our results, we provide a docker container containing scripts for the installation, alternatively, we also provide a description for a manual setup showing the steps to perform on a Linux-based operating system to repoduce the results of this work.
To ease the installation of our tool, we provide a Dockerfile for setting up a docker container. Please note that the container will use up to 1 GB of disc storage after the setup is performed.
To apply this file, we rely on docker and refer to the documentation on how to install docker on your Linux operating system. This tutorial enables to run docker commands on Linux without the ```sudo''' prefix.
For installing docker on Windows we refer to this documentation.
After docker is installed, make sure that the docker daemon is running. On systemd, you can use systemctl status docker to check the status of the daemon and systemctl start docker to start the daemon, if necessary.
Next, download the Dockerfile.
The container is set up by invoking docker build -t twins ./ in the directory where the Dockerfile is located.
By invoking this script, all dependencies as described in Section Manual Setup are installed, which might take several minutes.
After setting up the docker container, all required ressources (i.e., packages, programs, and scripts) are installed and can now be used inside the container.
To begin an interactive session, the command docker run -i -t twins /bin/bash can be used.
After starting the interactive session, you can continue here.
Requirements:
- Operating system: Ubuntu
- git (for cloning the required repositories)
- Python3 and Pip (on Ubuntu: python3-pip)
To clone the repository containing the data (variability models, measured performance values, and predicted performance values), use the following command:
git clone https://github.com/AI-4-SE/TwinsOrFalseFriends.git
To install the python packages needed to execute the python scripts, you can install the dependencies from requirements.txt in the top folder of the repository by executing
pip3 install -r requirements.txt
Please note that the execution of the Python script can take several minutes. To follow the progress, the Python script prints out the current progress.
To execute the script, you have to run the following commands:
cd TwinsOrFalseFriends
mkdir -p output
python3 scripts/analysis/run_analysis.py ./data/ ./output/
After the execution of the script, you can find several plots and compare them with the plots in the respective RQ folders on our top level and with the plots in the output folder. The structure is as follows:
output
| examplesystems.pdf
| rq2_example.pdf
| rq3_overview_small.pdf
RQ<Number>
|
└───<CaseStudy>
|
└───<PlotType>
|
└───<Plots>
| plot1.pdf
| plot2.pdf
Since we maintain the directory structure of our evaluation, the comparison of these plots should be straightforward. You may want to copy the respective files from the Docker container to your machine by executing:
docker cp <DockerID>:/application/TwinsOrFalseFriends/output/ /tmp/output/
Assuming that you are using Linux or Mac, the command will copy the whole folder into your /tmp folder.
If you are using Windows, adjust the command accordingly.
The DockerID is the ID of the docker container that you can retrieve using docker container ls -a.
| Case Studies | correlation_window | regression_error | scatterplot |
|---|---|---|---|
| 7z | 1 | 1 | 1 |
| Apache | 1 | 1 | 1 |
| brotli | 1 | 1 | 1 |
| exastencils | 1 | 1 | 1 |
| HSQLDB | 1 | 1 | 1 |
| jump3r | 1 | 1 | 1 |
| kanzi | 1 | 1 | 1 |
| LLVM | 1 | 1 | 1 |
| lrzip | 1 | 1 | 1 |
| MongoDB | 1 | 1 | 1 |
| nginx | 1 | 1 | 1 |
| PostgreSQL | 1 | 1 | 1 |
| VP8 | 1 | 1 | 1 |
| x264 | 1 | 1 | 1 |
| Case Studies | LinRegError | OptionWiseLinearError | scatterplot |
|---|---|---|---|
| 7z | 1 | 1 2 3 4 5 6 7 8 | 1 2 3 4 5 6 7 8 |
| Apache | 1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
| brotli | 1 | 1 2 | 1 2 |
| exastencils | 1 | 1 2 3 4 5 6 7 8 9 10 11 | 1 2 3 4 5 6 7 8 9 10 11 |
| HSQLDB | 1 | 1 2 3 4 5 6 7 8 9 10 | 1 2 3 4 5 6 7 8 9 10 |
| jump3r | 1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
| kanzi | 1 | 1 2 3 4 5 | 1 2 3 4 5 |
| LLVM | 1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
| lrzip | 1 | 1 2 3 4 5 6 7 | 1 2 3 4 5 6 7 |
| MongoDB | 1 | 1 2 3 4 5 6 7 8 9 10 | 1 2 3 4 5 6 7 8 9 10 |
| nginx | 1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
| PostgreSQL | 1 | 1 2 3 4 5 6 7 8 | 1 2 3 4 5 6 7 8 |
| VP8 | 1 | 1 2 3 4 5 6 7 8 | 1 2 3 4 5 6 7 8 |
| x264 | 1 | 1 2 3 4 5 6 7 8 9 10 | 1 2 3 4 5 6 7 8 9 10 |