figures_playground.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "# Smarter Evolution: Enhancing Evolutionary Black-Box Fuzzing with Adaptive Models\n",
    "\n",
    "In our publication, we presented an approach to bridge the gap between existing gray box fuzzing strategies and the real-world black box setting of fuzzing industrial control systems.\n",
    "This Jupyter Notebook was used to analyze the data created by the various fuzzing runs and to generate the corresponding figures.\n",
    "It requires the data that we generated during our experiments which can be downloaded [here](http://dx.doi.org/10.24406/fordatis/285).\n",
    "Unzip the archive and update the `DATA_ROOT_PATH` in the second code cell to point to the unzipped folder.\n",
    "Feel free to use this Notebook and adapt it to your needs.\n",
    "\n",
    "If you use our work in a publication, we would appreciate it if you would cite our work as follows:\n",
    "\n",
    "```\n",
    "@article{borcherding2023smarter,\n",
    "  author   = {Anne Borcherding and Martin Morawetz and Steffen Pfrang},\n",
    "  title    = {Smarter Evolution: Enhancing Evolutionary Black-Box Fuzzing with Adaptive Models},\n",
    "  year     = 2023,\n",
    "  journal = {Sensors},\n",
    "  doi = {https://doi.org/10.3390/s23187864},\n",
    "  url = {https://www.mdpi.com/1424-8220/23/18/7864}\n",
    "}\n",
    "```\n",
    "\n",
    "This notebook is structured as follows. First, we load the data for the evaluation from the provided data path. Then, we define several functions which will generate the different graphs, figures and statistical output to help understanding the data. After each of the functions, we provide a small example which shows how the function can be used. After that, we use the different functions to generate the figures that we used in the paper for you to be able to reproduce them. Those figures will be saved in the `results` folder which will be created as a subdirectory of the current working directory. For details on how we conducted our evalution and what the different algorithms and interpretations mean, please have a look at our paper.\n",
    "\n",
    "In the data, we use a slightly different naming convention for the used algorithms than in the published paper. The following table shows the mapping between those two naming conventions.\n",
    "\n",
    "| Name (Paper) | Name (Data) |\n",
    "|:-------------|:------------|\n",
    "| A_DT         | MLDriven    |\n",
    "| A_NN         | NEUZZ       |\n",
    "| A_SVM        | SVM         |\n",
    "| A_BASELINE   | Baseline    |\n",
    "| A_RANDOM     | Random      |\n",
    "\n",
    "If you have any questions or remarks on this notebook or our publication itself, do not hestitate to contact us.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import datetime\n",
    "import os\n",
    "import glob\n",
    "import matplotlib.pyplot as plt\n",
    "import ntpath\n",
    "import json\n",
    "from typing import Callable\n",
    "import scipy\n",
    "import itertools"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# make sure to put the correct path here\n",
    "DATA_ROOT_PATH = \"this_is_most_likely_not_the_correct_path\"\n",
    "\n",
    "RANDOM_ROOT_PATH = os.path.join(DATA_ROOT_PATH, \"Random\")\n",
    "BASELINE_ROOT_PATH = os.path.join(DATA_ROOT_PATH, \"Baseline\")\n",
    "DT_ROOT_PATH = os.path.join(DATA_ROOT_PATH, \"MLDriven\")\n",
    "SVM_ROOT_PATH = os.path.join(DATA_ROOT_PATH, \"SVM\")\n",
    "NN_ROOT_PATH = os.path.join(DATA_ROOT_PATH, \"NEUZZ\")\n",
    "\n",
    "RESULTS_ROOT_PATH = \"results/\"\n",
    "os.makedirs(RESULTS_ROOT_PATH, exist_ok=True)\n",
    "\n",
    "algorithm_ids = ['random', 'mldriven', 'svm', 'neuzz', 'baseline']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "jp-MarkdownHeadingCollapsed": true,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "## General Data Loading\n",
    "\n",
    "We will create a dictionary with the file names as keys and pandas dataframes als items. Since loading all the data takes some time and memory, we provide several filter functions to load only parts of the data. Note that it might happen that you will not be able to run all the cells successfully if you decide to change the filter function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "\n",
    "The resulting dataframes have the following keys: `['datetime', 'test_case_hex', 'triggered_vulns', 'unique_vulns', 'new_vuln', 'string_chars_hit', 'test_case.sint', 'test_case.uint', 'test_case.str', 'services.icmp', 'services.https', 'services.http', 'services.snmp', 'services.arp', 'services.udp', 'services.tcp', 'services.ip']`. These keys have the following meaning:\n",
    "\n",
    "\n",
    "| Key              | Description                                                                                                               |\n",
    "|:-----------------|:--------------------------------------------------------------------------------------------------------------------------|\n",
    "| datetime         | The time the record was created at                                                                                        |\n",
    "| test_case_hex    | The raw hex data of the testcase which was received by the VulnDuT                                                        |\n",
    "| triggered_vulns  | Array representing the vulnerabilities that have already been triggered                                                   |\n",
    "| unique_vulns     | Number of unuqiue vulnerabilities that have already been triggered by this fuzzing run                                    |\n",
    "| new_vuln         | Boolean value to indicate whether this test case triggered a vulnerability that was not triggered by this fuzzing run yet |\n",
    "| string_chars_hit | List of chars of the String value that triggered a vulnerability                                                          |\n",
    "| test_case.sint   | The Signed Integer Value of the test case                                                                                 |\n",
    "| test_case.uint   | The Unsigned Integer Value of the test case                                                                               |\n",
    "| test_case.str    | The String Value of the test case                                                                                         |\n",
    "| services.XXX     | 1 if the service was up after this test case, 0 otherwise                                                                 |\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "def load_files(files: [str], load_condition: Callable[[str], bool]) -> pd.DataFrame:\n",
    "    \"\"\"\n",
    "    Load those of the given files that fulfil the given condition.\n",
    "    :param files: The file paths to load the data from.\n",
    "    :param load_condition: A function representing the condition the files need to fulfil to be included. Expects a file path and returns a boolean.\n",
    "    :return: A dataframe containing the data from the files that fulfil the given condition.\n",
    "    \"\"\"\n",
    "    loaded_data = {}\n",
    "    for file in files:\n",
    "        if load_condition(file):\n",
    "            with open(file) as f:\n",
    "                print(f\"Processing {file}\")\n",
    "                df = pd.json_normalize(json.load(f))\n",
    "                loaded_data[file] = df.sort_values('datetime')\n",
    "    return loaded_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "The following functions implement different conditions to restrict the time and memory the data loading needs. To ensure to be able to run all the cells based on this data, you should choose the default condition (`load_condition_default_config`) which loads the data produced by the experiments using the default values (multidimensional feedback and a feedback interval of 50)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "def load_condition_default_config(file_path: str):\n",
    "    \"\"\"\n",
    "    Select the files generated by the experiments using the default values (multidimensional feedback and a feedback interval of 50).\n",
    "    :param file_path: The path to decide on.\n",
    "    :return: True if the file was generated by an experiment using the default values, false otherwise.\n",
    "    \"\"\"\n",
    "    return \"multiple_50_1-9\" in file_path or \"binary_50_1-9\" in file_path\n",
    "\n",
    "def load_condition_unidimensional_config(file_path: str):\n",
    "    \"\"\"\n",
    "    Select the files generated by the experiments using unidimensional feedback.\n",
    "    :param file_path: The path to decide on.\n",
    "    :return: True if the file was generated by an experiment using unidimensional feedback, false otherwise.\n",
    "    \"\"\"\n",
    "    return \"binary_50_1-9\" in file_path\n",
    "\n",
    "def load_all(file_path: str):\n",
    "    \"\"\"\n",
    "    Select all files. Only use this if you have lots of time and memory.\n",
    "    :param file_path: The path to decide on.\n",
    "    :return: True.\n",
    "    \"\"\"\n",
    "    return True\n",
    "\n",
    "def load_dt_nn(file_path: str):\n",
    "    \"\"\"\n",
    "    Select the files generated by experiments with the default configurations, restricted to the two algorithms A_DT and A_NN.\n",
    "    :param file_path: The path to decide on.\n",
    "    :return: True if the file was generated by an experiment using the default values and either A_DT or A_NN.\n",
    "    \"\"\"\n",
    "    return \"multiple\" in file_path and \"50_\" in file_path and (\"neuzz\" in file_path or \"mldriven\" in file_path)\n",
    "\n",
    "def load_feedback_interval(file_path: str):\n",
    "    \"\"\"\n",
    "    Select the files generated by experiments with either 10 or 500 as feedback interval and multidimensional feedback.\n",
    "    :param file_path: The path to decide on.\n",
    "    :return: True if the file was generated by an experiment using either 10 or 500 as feedback interval and multidimensional feedback.\n",
    "    \"\"\"\n",
    "    return \"multiple\" in file_path and \"500\" in file_path or \"10_\" in file_path\n",
    "\n",
    "def load_feedback_interval_dt_nn(file_path: str):\n",
    "    \"\"\"\n",
    "    Select the files generated by experiments with either 10 or 500 as feedback interval and multidimensional feedback, restricted to the two algorithms A_DT and A_NN.\n",
    "    :param file_path:\n",
    "    :return: True if the file was generated by an experiment using either 10 or 500 as feedback interval, multidimensional feedback, and either A_DT or A_NN.\n",
    "    \"\"\"\n",
    "    return \"multiple\" in file_path and (\"500\" in file_path or \"10_\" in file_path) and (\"neuzz\" in file_path or \"mldriven\" in file_path)\n",
    "\n",
    "def load_feedback_interval_baseline(file_path: str):\n",
    "    \"\"\"\n",
    "    Select the files generated by experiments with either 10 or 500 as feedback interval and multidimensional feedback, restricted to the algorithm A_BASELINE.\n",
    "    :param file_path:\n",
    "    :return: True if the file was generated by an experiment using either 10 or 500 as feedback interval, multidimensional feedback, and A_BASELINE.\n",
    "    \"\"\"\n",
    "    return \"multiple\" in file_path and (\"500\" in file_path or \"10_\" in file_path) and (\"baseline\" in file_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "abs_path = os.path.join(DATA_ROOT_PATH)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "The following cell will actually load the data based on the given path and the chosen condition. Allow some time for it to finish."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "print('searching recursively in: ' + abs_path)\n",
    "files = glob.glob(os.path.join(abs_path, '**', '*.json'), recursive=True)\n",
    "assert len(files) > 0, \"There are no json files in the given directory, please make sure that the path is correct.\"\n",
    "data = load_files(files, load_condition_default_config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "dataset_ids = list(data.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "jp-MarkdownHeadingCollapsed": true,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "## Basic Drawing\n",
    "\n",
    "This section provides some basic functions which will be used later on to draw the figures. We included some examples for each of the drawing functions to show what kind of results they produce. The figures that have been used in the paper will be produced later on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "def draw_vulns(datasets: list[list[str]], actual_data: pd.DataFrame, title: str, ylabel: str , xaxis: str = 'testcases', num_datapoints: int = None, draw_confidence_interval:bool =True, labels: list[str] = None, label_placement: tuple[float,float] = None, save_path: str = None, big_font: bool = False, ncols: int = None, figsize: tuple[float, float] = None):\n",
    "    \"\"\"\n",
    "    Basic drawing function which draws numbers of triggered vulnerabilities either over the number of test cases or over the time. The plotting is done based on matplotlib. If you want to create your own figures, you'd probably want to use draw_vulns_over_time or draw_vulns_over_execs which build upon this function and include the necessary preprocessing.\n",
    "    :param datasets: The datasets that should be used for this figure. Only includes the path names of the experiment results, not the actual data.\n",
    "    :param actual_data: Dataframe including the actual data to be used for drawing.\n",
    "    :param title: Title to be used for the figure.\n",
    "    :param ylabel: Label for the y-axis to be used for the figure.\n",
    "    :param xaxis: Defines whether the data should be plotted over the number of test cases or over time. Should either be 'testcases' or 'time'.\n",
    "    :param num_datapoints: The number of datapoints to use per dataset. If restricted, this can help to make the calculation faster and resulting figure smaller (since less datapoints need to be represented).\n",
    "    :param draw_confidence_interval: Switch to decide whether the confidence interval should be drawn in the figure. Might provide some interesting insights but also makes the figure messy.\n",
    "    :param labels: The labels to use for the data sets. Defaults to the file names.\n",
    "    :param label_placement: The placement of the label box as a tuple of x and y position.\n",
    "    :param save_path: The path to which the resulting figure should be saved. Nothing is saved if the path is set to None.\n",
    "    :param big_font: Switch to make the font bigger (font size 16).\n",
    "    :param ncols: The number of columns the label box should have. Defaults to len(datasets).\n",
    "    :param figsize: The size of the figure. Defaults to the default figsize of matplotlib.\n",
    "    \"\"\"\n",
    "\n",
    "    if figsize:\n",
    "        fig, ax = plt.subplots(figsize=figsize)\n",
    "    else:\n",
    "        fig, ax = plt.subplots()\n",
    "    ax.set_title(title)\n",
    "    ax.set_ylabel(ylabel)\n",
    "    if xaxis == 'testcases':\n",
    "        ax.set_xlabel(\"#Test Cases\")\n",
    "    elif xaxis == 'time':\n",
    "        ax.set_xlabel(\"Time in Seconds\")\n",
    "    else:\n",
    "        print(f\"Please set either \\\"testcases\\\" or \\\"time\\\" for xaxis. You used {xaxis}.\")\n",
    "        return\n",
    "\n",
    "    for i, ds_list in enumerate(actual_data):\n",
    "\n",
    "        n = min(len(ds_list[0]['new_vuln']) if num_datapoints is None else num_datapoints, len(ds_list[0]['new_vuln']))\n",
    "        ds_concat = pd.DataFrame.from_dict(ds_list[0]['new_vuln'][:n]).rename({'new_vuln': 0}, axis=1)\n",
    "\n",
    "        for index, ds in enumerate(ds_list[1:]):\n",
    "            ds_concat.insert(loc=index+1, column=index+1, value=ds['new_vuln'][:n])\n",
    "\n",
    "        ds_concat.fillna(method='ffill', inplace=True)\n",
    "        # calculate median\n",
    "        ds_median = ds_concat.median(axis=1)\n",
    "\n",
    "        # calculate 0.95 confidence interval\n",
    "        ds_ci = ds_concat.transpose().quantile(q=[.05, .95]).transpose()\n",
    "\n",
    "        # set labels\n",
    "        label_base = labels[i] if labels else datasets[i][0].split('/')[-2]\n",
    "\n",
    "        # plot\n",
    "        if xaxis == 'testcases':\n",
    "            xs = range(len(ds_list[0]))[:n]\n",
    "        elif xaxis == 'time':\n",
    "            xs = ds_list[0]['datetime'][:n]\n",
    "\n",
    "        linestyle = \"dashed\" if label_base == \"A_RANDOM\" or label_base == \"A_BASELINE\" else \"solid\"\n",
    "        ax.plot(xs, ds_median, label=label_base, linestyle=linestyle)\n",
    "        if draw_confidence_interval:\n",
    "            ax.fill_between(xs, ds_ci[0.95], ds_ci[0.05], alpha=0.3, label=f\"95% CI\")\n",
    "\n",
    "    loc = label_placement if label_placement else (0,-0.3) if draw_confidence_interval else (0,-0.2)\n",
    "    cols = ncols if ncols else len(datasets)\n",
    "\n",
    "    if big_font:\n",
    "        ldg = ax.legend(loc=loc, ncols=cols, fontsize=16)\n",
    "        for item in ([ax.title, ax.xaxis.label, ax.yaxis.label] + ax.get_xticklabels() + ax.get_yticklabels()):\n",
    "            item.set_fontsize(16)\n",
    "    else:\n",
    "        ldg = ax.legend(loc=loc, ncols=cols)\n",
    "\n",
    "\n",
    "    if save_path:\n",
    "        fig.savefig(os.path.join(RESULTS_ROOT_PATH,save_path), format=\"svg\", bbox_extra_artists=(ldg,))\n",
    "    plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# define the datasets for the examples to follow, choosing 10 runs for each configuration\n",
    "dt_datasets = [d for d in dataset_ids if \"mldriven\" in d and \"multiple\" in d and \"50_\" in d][:10]\n",
    "nn_datasets = [d for d in dataset_ids if \"neuzz\" in d and \"multiple\" in d and \"50_\" in d][:10]\n",
    "rand_datasets = [d for d in dataset_ids if \"random\" in d and \"multiple\" in d and \"50_\" in d][:10]\n",
    "bl_datasets = [d for d in dataset_ids if \"baseline\" in d and \"multiple\" in d and \"50_\" in d][:10]\n",
    "svm_datasets = [d for d in dataset_ids if \"svm\" in d and \"multiple\" in d and \"50_\" in d][:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "jp-MarkdownHeadingCollapsed": true,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "## Vulnerabilities over Time\n",
    "\n",
    "The following function draws the number of unique triggered vulnerabilities over the time of the fuzzing runs (24h). Each figure represents the mean of the given runs, in our case that is 10 runs per configuration. With this, we can analyze how the different fuzzers perform over the time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "def format_data(ds: list[str], actual_data: pd.DataFrame) -> pd.DataFrame:\n",
    "    \"\"\"\n",
    "    Does some formatting on the time-based data to prepare it to be nicely drawn. This includes calculating relative timestamps and to interpolate the functions to account for the irregular measurements (caused by different timestamps of the test cases in different experiments).\n",
    "    :param ds: The datasets that should be used for this figure. Only includes the path names of the experiment results, not the actual data.\n",
    "    :param actual_data: Dataframe including the actual data to be used for drawing.\n",
    "    :return: A dataframe containing the data given in actual data, but with relative time stamps and interpolated data.\n",
    "    \"\"\"\n",
    "    t0 = datetime.strptime(actual_data[ds]['datetime'][0], '%Y-%m-%d %H:%M:%S,%f')\n",
    "    ds_formatted = pd.DataFrame.from_dict(actual_data[ds]['datetime'])\n",
    "    ds_formatted = ds_formatted.applymap(lambda x: (datetime.strptime(x, '%Y-%m-%d %H:%M:%S,%f') - t0).total_seconds())\n",
    "\n",
    "    interp_xs = np.linspace(0, 86400, 5000) #24h\n",
    "    interp_ys = np.interp(interp_xs, ds_formatted['datetime'], actual_data[ds]['new_vuln'])\n",
    "\n",
    "    return pd.DataFrame.from_dict({'datetime': interp_xs, 'new_vuln': interp_ys})\n",
    "\n",
    "def draw_vulns_over_time(datasets: list[list[str]], title: str, ylabel: str, num_datapoints: int = None, draw_confidence_interval: bool = True, labels: list[str] = None, label_placement: tuple[float,float] = None, save_path: str = None, big_font: bool = False, ncols: int = None, figsize: tuple[float,float] = None):\n",
    "    \"\"\"\n",
    "    Draws the vulnerabilities over time using the given datasets. See the docs of draw_vulns for details on the arguments.\n",
    "    \"\"\"\n",
    "    assert isinstance(datasets[0], list), \"Please provide a list of lists of strings, each top level list representing one configuration with several runs.\"\n",
    "    preprocessed_data = [[format_data(ds, data) for ds in ds_list] for ds_list in datasets]\n",
    "    draw_vulns(datasets=datasets, actual_data=preprocessed_data, title=title, xaxis='time', ylabel=ylabel, num_datapoints=num_datapoints, draw_confidence_interval=draw_confidence_interval, labels=labels, label_placement=label_placement, save_path=save_path, big_font=big_font, ncols=ncols, figsize=figsize)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "**Example:** The following code draws the vulnerabilities over time for A_NN and A_BASELINE for the default configuration, including the confidence intervals."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# draw vulnerabilities over time\n",
    "# change the used datasets to see other subsets of the used algorithms\n",
    "draw_vulns_over_time([nn_datasets, bl_datasets], \"Performance over Time\", ylabel=\"#Vulnerabilities Triggered\", labels=[\"A_DT\", \"A_BASELINE\"], draw_confidence_interval=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "jp-MarkdownHeadingCollapsed": true,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "## Vulnerabilities over Number of Test Cases\n",
    "\n",
    "The following code does the same as the one above, but represents the number of unique triggered vulnerabilities over the number of test cases. With this, we can analyze the behavior of the different fuzzers based on the efficiency of the generated test cases alone (setting the throughput of the fuzzers aside)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "def draw_vulns_over_testcases(datasets: list[list[str]], title: str, ylabel: str, num_datapoints: int = None, draw_confidence_interval: bool = True, labels: list[str] = None, label_placement: tuple[float,float] = None, save_path: str = None, big_font: bool = False, ncols: int = None, figsize: tuple[float,float] = None):\n",
    "    \"\"\"\n",
    "    Draws the vulnerabilities over the number of test cases using the given datasets. See the docs of draw_vulns for details on the arguments.\n",
    "    \"\"\"\n",
    "    assert isinstance(datasets[0], list), \"Please provide a list of lists of strings, each top level list representing one configuration with several runs.\"\n",
    "    actual_data = [[data[ds] for ds in ds_list] for ds_list in datasets]\n",
    "    draw_vulns(datasets, actual_data, title, ylabel, num_datapoints=num_datapoints, labels=labels, label_placement=label_placement, save_path=save_path, draw_confidence_interval=draw_confidence_interval, big_font=big_font, ncols=ncols, figsize=figsize)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "**Example:** The following code draws the vulnerabilities over time for A_NN and A_BASELINE for the default configuration, including the confidence intervals."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# draw vulnerabilities over time\n",
    "# change the used datasets to see other subsets of the used algorithms\n",
    "draw_vulns_over_testcases([nn_datasets, bl_datasets], \"Performance over Number of Test Cases\", ylabel=\"#Vulnerabilities Triggered\", labels=[\"MLDriven\", \"Baseline\"], draw_confidence_interval=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "jp-MarkdownHeadingCollapsed": true,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "## Integer Value\n",
    "\n",
    "The following function draws the integer values chosen by the fuzzers over the number of test cases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "def draw_integer_values(datasets: list[list[str]], identifier: str, title: str, ylabel: str, num_datapoints: int = None, labels: list[str] = None,  xticks: list[int] = None, label_placement: tuple[float, float] = None, save_path: str = None, draw_legend: bool = True, ncols: int = None, big_font: bool = False):\n",
    "    \"\"\"\n",
    "    Draw a plot showing the chosen integer values over time.\n",
    "    :param datasets: The datasets that should be used for this figure. Only includes the path names of the experiment results, not the actual data.\n",
    "    :param identifier: Row in the data to be used for the plot, you probably want to use either 'test_case.sint' or 'test_case.unit'\n",
    "    :param title: Title to be used for the figure.\n",
    "    :param ylabel: Label for the y-axis to be used for the figure.\n",
    "    :param num_datapoints: The number of datapoints to use per dataset. If restricted, this can help to make the calculation faster and resulting figure smaller (since less datapoints need to be represented).\n",
    "    :param labels: The labels to use for the data sets. Defaults to the file names.\n",
    "    :param xticks: The x-ticks to be used for the figure.\n",
    "    :param label_placement: The placement of the label box as a tuple of x and y position.\n",
    "    :param save_path: The path to which the resulting figure should be saved. Nothing is saved if the path is set to None.\n",
    "    :param draw_legend: Set to True if the legend should be drawn, False otherwise.\n",
    "    :param ncols: The number of columns the label box should have. Defaults to len(datasets).\n",
    "    :param big_font: Switch to make the font bigger (font size 16).\n",
    "    \"\"\"\n",
    "    fig, ax = plt.subplots()\n",
    "    ax.set_title(title)\n",
    "    ax.set_xlabel(\"#Test Cases\")\n",
    "    ax.set_ylabel(ylabel)\n",
    "    if xticks:\n",
    "        ax.set_xticks(xticks)\n",
    "\n",
    "    assert(labels is None or len(labels) == len(datasets))\n",
    "\n",
    "    for i, ds in enumerate(datasets):\n",
    "        n = min(len(data[ds][identifier]) if num_datapoints is None else num_datapoints, len(data[ds][identifier]))\n",
    "        label = labels[i] if labels else ds\n",
    "        sample_rate = int(len(data[ds][identifier]) / n)\n",
    "        sampled_data = data[ds][identifier].iloc[::sample_rate]\n",
    "        ax.plot(sampled_data, '.', label=label)\n",
    "    loc = label_placement if label_placement else (0,-0.2)\n",
    "    ncol = ncols if ncols else len(datasets)\n",
    "    if draw_legend:\n",
    "        ax.legend(loc=loc, ncol=ncol)\n",
    "\n",
    "    if big_font:\n",
    "        if draw_legend:\n",
    "            ldg = ax.legend(loc=loc, ncols=ncol, fontsize=20)\n",
    "        for item in ([ax.title, ax.xaxis.label, ax.yaxis.label] + ax.get_xticklabels() + ax.get_yticklabels()):\n",
    "            item.set_fontsize(20)\n",
    "\n",
    "    if save_path:\n",
    "        fig.savefig(os.path.join(RESULTS_ROOT_PATH,save_path))\n",
    "\n",
    "    plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "**Examples:** The following two cells show examples of a figure showing the Unsigned Integer Values and a figure showing the Signed Integer Values chosen by the fuzzers over time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "for alg in ['neuzz']:\n",
    "    ui_datasets = [id for id in dataset_ids if alg in id][:10]\n",
    "    draw_integer_values(datasets=ui_datasets, identifier='test_case.uint', title=f\"Distribution of Unsigned Integer Values Chosen by the {alg} Algorithm\", ylabel=\"Unsigned Integer Value\", labels=[f\"run {i}\" for i in range(len(ui_datasets))], label_placement=(1.1,0), ncols=1, num_datapoints=100000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "for alg in ['mldriven']:\n",
    "    ui_datasets = [id for id in dataset_ids if alg in id][:10]\n",
    "    draw_integer_values(datasets=ui_datasets, identifier='test_case.sint', title=f\"Distribution of Signed Integer Values Chosen by the {alg} Algorithm\", ylabel=\"Signed Integer Value\", labels=[f\"run {i}\" for i in range(len(ui_datasets))], num_datapoints=10000, label_placement=(1.1,0), ncols=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "jp-MarkdownHeadingCollapsed": true,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "## String Value\n",
    "\n",
    "The following function aims to visualize the String values chosen during the fuzzing runs. Note that this representation is rather hard, but we still decided to keep it in this Notebook to help with manual analysis.\n",
    "\n",
    "The String based vulnerabilities depend on the first characters of the String value of the test case and the more characters are correct, the more services crash (see the paper for more details). As a result, we want to see how the number of correct characters evolves over time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "def draw_string_triggers(datasets: list[list[str]], title: str, ylabel: str, start: int = 0, num_datapoints:int = None, save_path: str = None):\n",
    "    \"\"\"\n",
    "    Show the number of characters of the String value that lead to a triggered vulnerability.\n",
    "    :param datasets: The datasets that should be used for this figure. Only includes the path names of the experiment results, not the actual data.\n",
    "    :param title: The title for the figure.\n",
    "    :param ylabel: The label for the y-axsis of the figure.\n",
    "    :param start: The starting point for the figure (a test case number). This can be used together with `num_datapoints` to define the interval of test cases the figure will represent.\n",
    "    :param num_datapoints: The number of data points shown in the figure. This can be used together with `start` to define the interval of test cases the figure will represent.\n",
    "    :param save_path: The path to which the resulting figure should be saved. Nothing is saved if the path is set to None.\n",
    "    :return:\n",
    "    \"\"\"\n",
    "    fig, ax = plt.subplots(figsize=(8,10))\n",
    "    ax.set_title(title)\n",
    "    ax.set_xlabel(\"Number of Test Cases\")\n",
    "    ax.set_ylabel(ylabel)\n",
    "    ax.set_yticks(range(8),[1,2,3,4,5,6,7,8])\n",
    "    ax.set_height=5\n",
    "\n",
    "    for ds in datasets:\n",
    "        n = min(len(data[ds]['string_chars_hit']) if num_datapoints is None else num_datapoints, len(data[ds]['string_chars_hit']))\n",
    "        a = np.matrix(data[ds]['string_chars_hit'][start:n].to_list()).transpose()\n",
    "        ax.imshow(a, origin='lower', cmap='gray_r', aspect='auto')\n",
    "        ax.set_xticks(np.arange(n-start, step=100), np.arange(start, n, step=100))\n",
    "\n",
    "    if save_path:\n",
    "       fig.savefig(os.path.join(RESULTS_ROOT_PATH,save_path), format=\"svg\")\n",
    "\n",
    "    plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "**Example:** The following figure shows the characters of the String values that triggered a vulnerability. Note that the String has eight characters, which is represented by the interval shown on the y axis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "mydata = [os.path.join(NN_ROOT_PATH, \"neuzz_multiple_50_1-9\", \"adaptive_eval_1_dut_7.json\")]\n",
    "draw_string_triggers(mydata, \"Correctly Guessed Characters (A_NN)\", \"Characters Hit\", 8000, 8700)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "## Figures and Information for the Publication\n",
    "\n",
    "The following code generates the graphs used for our publication and provides some stats which were also used for the publication. The structure of this section is the same as the structure of the Results section in our publication. One exception from this is the presentation of results regarding the choice of feedback dimension. For these results, we load additional data which takes some time. To be able to see all the other results before investing that additional time, we moved that section to the end of this Notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "jp-MarkdownHeadingCollapsed": true,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "### General Performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define datasets and labels\n",
    "\n",
    "dt_datasets = [d for d in dataset_ids if \"mldriven\" in d and \"multiple\" in d and \"50_\" in d][:10]\n",
    "svm_datasets = [d for d in dataset_ids if \"svm\" in d and \"multiple\" in d and \"50_\" in d][:10]\n",
    "rand_datasets = [d for d in dataset_ids if \"random\" in d and \"multiple\" in d and \"50_\" in d][:10]\n",
    "nn_datasets = [d for d in dataset_ids if \"neuzz\" in d and \"multiple\" in d and \"50_\" in d][:10]\n",
    "baseline_datasets = [d for d in dataset_ids if \"baseline\" in d and \"multiple\" in d and \"50_\" in d][:10]\n",
    "\n",
    "general_performance_datasets = [dt_datasets, nn_datasets, svm_datasets, baseline_datasets, rand_datasets]\n",
    "general_performance_labels = [\"A_DT\", \"A_NN\", \"A_SVM\", \"A_BASELINE\", \"A_RANDOM\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "draw_vulns_over_testcases(datasets=general_performance_datasets, title=\"Performance of Model-based Fuzzers\", ylabel=\"#Vulnerabilities Triggered\",labels=general_performance_labels, save_path = \"performance_all_ex.svg\", num_datapoints=300000, label_placement=(0.35, 0.01),draw_confidence_interval=False, ncols=2, figsize=(5,3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "draw_vulns_over_testcases(general_performance_datasets, \"Performance of Model-based Fuzzers\", \"#Vulnerabilities Triggered\",labels=general_performance_labels, save_path = \"performance_all_ex_conv.svg\", num_datapoints=300000, draw_confidence_interval=True, ncols=3, label_placement=(0.19,0.01))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "draw_vulns_over_time(general_performance_datasets, \"Performance of Model-based Fuzzers\", \"#Vulnerabilities Triggered\",labels=general_performance_labels, save_path = \"performance_all_time.svg\", draw_confidence_interval=False, ncols=2, label_placement=(0.35, 0.01), figsize=(5,3))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "draw_vulns_over_time(general_performance_datasets, \"Performance of Model-based Algorithms\", \"#Vulnerabilities Triggered\",labels=general_performance_labels, save_path = \"performance_all_time_conf.svg\", draw_confidence_interval=True, ncols=3, label_placement=(0.2,0.01))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "#### Statistical Tests\n",
    "\n",
    "This section includes several statistical tests to analyze the statistical significance of differences in performance of the different fuzzers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "stats_datasets = [dt_datasets, nn_datasets, svm_datasets, bl_datasets, rand_datasets]\n",
    "stats_ids = [\"DT\", \"NN\", \"SVM\", \"Baseline\", \"Random\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "##### Statistical Tests for Final Results\n",
    "\n",
    "These tests use the final number of unique triggered vulnerabilities of each configuration to analyze the significance of differences in performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "#collect the data\n",
    "stats_values_final = {}\n",
    "\n",
    "for i, ds_list in enumerate(stats_datasets):\n",
    "    vals = [data[ds]['new_vuln'].iloc[-1] for ds in ds_list]\n",
    "    stats_values_final[stats_ids[i]] = vals\n",
    "res = pd.DataFrame(0, index=stats_ids, columns=stats_ids)\n",
    "for i,j in list(itertools.product(stats_ids, stats_ids)):\n",
    "    r = scipy.stats.mannwhitneyu(stats_values_final[i], stats_values_final[j])\n",
    "    res.at[i,j] = round(r.pvalue,3)\n",
    "res"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "##### Statistical Tests for Various Values\n",
    "\n",
    "These tests analyse the statistical significance of the performance of the fuzzers over time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "sample_points = range(0,5000, 100)\n",
    "stats_values = [{} for _ in range(len(sample_points))]\n",
    "\n",
    "for i, ds_list in enumerate(stats_datasets):\n",
    "    ds_list_form = [format_data(ds, data) for ds in ds_list]\n",
    "    for j, p in enumerate(sample_points):\n",
    "        vals = [ds['new_vuln'].iloc[p] for ds in ds_list_form]\n",
    "        stats_values[j][stats_ids[i]] = vals\n",
    "stats_values.append(stats_values_final)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "def calc_and_draw_pvalues(values: list[int], comparison_alg: str, save_path: str, title: str = None, labels: list[str] = None):\n",
    "    \"\"\"\n",
    "    Calculate the p-value for each algorithm at each sample point in comparison to one fixed algorithm.\n",
    "    :param values: The samples to calculate the p-values for.\n",
    "    :param comparison_alg: The algorithm to compare to, should either be \"Random\" or \"Baseline\".\n",
    "    :param save_path: The path to which the resulting figure should be saved. Nothing is saved if the path is set to None.\n",
    "    :param title: The title for the figure.\n",
    "    :param labels: The labels to be used for the figure.\n",
    "    \"\"\"\n",
    "\n",
    "    pvalues = [[scipy.stats.mannwhitneyu(sample[comparison_alg], sample[i]).pvalue for i in stats_ids] for sample in values]\n",
    "    index = pd.Series(list(sample_points) + [5000]).apply(lambda x: (x / 5000 * 86400)) # stretch it to original time frame\n",
    "    df_pvalues = pd.DataFrame(pvalues, columns=stats_ids, index=index)\n",
    "\n",
    "    fig, ax = plt.subplots(figsize=(5,3))\n",
    "\n",
    "    t = title if title else f\"p-values in Comparison to {comparison_alg}\"\n",
    "    ax.set_title(t)\n",
    "    ax.set_ylabel(\"p-value\")\n",
    "    ax.set_xlabel(\"Time in Seconds\")\n",
    "    ax.set_ylim([-0.1,1.1])\n",
    "    ax.set_prop_cycle('color', list(plt.cm.tab10([0,1,2,4]))) # change colormap to fit other graphs\n",
    "    ls = labels if labels else [i for i in stats_ids if i != \"Random\" and i != \"Baseline\"]\n",
    "    ax.plot(df_pvalues.drop([\"Random\", \"Baseline\"], axis=1), 'x', label=ls, linewidth=2)\n",
    "    ax.axhline(0.05, color='black' , linewidth=3, linestyle=\"-\") # color=plt.cm.tab10([3])\n",
    "    ax.legend(fontsize=12)#, loc=(0.61, 0.6))\n",
    "    for item in ([ax.title, ax.xaxis.label, ax.yaxis.label] + ax.get_xticklabels() + ax.get_yticklabels()):\n",
    "        item.set_fontsize(12)\n",
    "    fig.savefig(save_path, format='svg')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "calc_and_draw_pvalues(stats_values, \"Random\", \"results/pvalues_random.svg\", title=\"p-values in Comparison to A_RANDOM\", labels=[\"A_DT\", \"A_NN\", \"A_SVM\"])\n",
    "calc_and_draw_pvalues(stats_values, \"Baseline\", \"results/pvalues_baseline.svg\", title=\"p-values in Comparison to A_BASELINE\", labels=[\"A_DT\", \"A_NN\", \"A_SVM\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "jp-MarkdownHeadingCollapsed": true,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "### Choice of Feedback Dimension"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "def print_triggered_vulns(datasets: list[list[str]], actual_data: pd.DataFrame, labels: list[str]):\n",
    "    \"\"\"\n",
    "    Extracts the triggered vulnerabilities from the data and prints them in a human-readable format.\n",
    "    :param datasets: The datasets that should be used for this figure. Only includes the path names of the experiment results, not the actual data.\n",
    "    :param actual_data: Dataframe including the actual data to be used for drawing.\n",
    "    :param labels: The labels of the algorithms.\n",
    "    \"\"\"\n",
    "    for id, ds_list in enumerate(datasets):\n",
    "        vulns = [0]*25\n",
    "        for ds in ds_list:\n",
    "            vulns = [i + j for (i,j) in zip(actual_data[ds].iloc[-1]['unique_vulns'], vulns)]\n",
    "        print(f\"{labels[id]}: {vulns}\")\n",
    "        print(f\"    String1 (ABCDE): {vulns[:5]}\")\n",
    "        print(f\"    String2 (FGHIG): {vulns[5:10]}\")\n",
    "        print(f\"    String3 (KLMNOPQR): {vulns[10:18]}\")\n",
    "        print(f\"    sint max: {vulns[18]}\")\n",
    "        print(f\"    sint min: {vulns[19]}\")\n",
    "        print(f\"    sint range1: {vulns[20]}\")\n",
    "        print(f\"    sint range2: {vulns[21]}\")\n",
    "        print(f\"    uint max: {vulns[22]}\")\n",
    "        print(f\"    uint range1: {vulns[23]}\")\n",
    "        print(f\"    uint range2: {vulns[24]}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "print_triggered_vulns(general_performance_datasets, data, general_performance_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# define datasets for the configurations using unidimensional (=binary) feedback\n",
    "\n",
    "dt_datasets_bin = [d for d in dataset_ids if \"mldriven\" in d and \"binary\" in d][:10]\n",
    "svm_datasets_bin = [d for d in dataset_ids if \"svm\" in d and \"binary\" in d][:10]\n",
    "rand_datasets_bin = [d for d in dataset_ids if \"random\" in d and \"binary\" in d][:10]\n",
    "nn_datasets_bin = [d for d in dataset_ids if \"neuzz\" in d and \"binary\" in d][:10]\n",
    "baseline_datasets_bin = [d for d in dataset_ids if \"baseline\" in d and \"binary\" in d][:10]\n",
    "\n",
    "binary_datasets = [dt_datasets_bin, nn_datasets_bin, svm_datasets_bin, baseline_datasets_bin, rand_datasets_bin]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "print_triggered_vulns(binary_datasets, data, general_performance_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "draw_vulns_over_time([dt_datasets, dt_datasets_bin], \"Performance of A_DT\", \"#Vulnerabilities Triggered\", labels=[\"Multidimensional Feedback\", \"Unidimensional Feedback\"], save_path=\"feedback_dimension_mldriven.svg\", big_font=True, label_placement=(0.22,0.01), ncols=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "draw_vulns_over_time([nn_datasets, nn_datasets_bin], \"Performance of A_NN\", \"#Vulnerabilities Triggered\", labels=[\"Multidimensional Feedback\", \"Unidimensional Feedback\"], save_path=\"feedback_dimension_neuzz.svg\", big_font=True, label_placement=(0.22,0.01), ncols=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "jp-MarkdownHeadingCollapsed": true,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "### Integer Values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "draw_integer_values(datasets=[dt_datasets[i] for i in [1, 5, 8]], identifier='test_case.sint',\n",
    "                    title=f\"Distribution of Signed Integer Values\", ylabel=\"Signed Integer Value\", num_datapoints=None,\n",
    "                    save_path=\"sint_mldriven.pdf\", labels=[f\"Run {i}\" for i in range(3)], ncols=1,\n",
    "                    label_placement=(0.81, 0.01), big_font=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "draw_integer_values(datasets=[nn_datasets[i] for i in [1, 5, 8]], identifier='test_case.sint',\n",
    "                    title=f\"Distribution of Signed Integer Values\", ylabel=\"Signed Integer Value\", num_datapoints=None,\n",
    "                    save_path=\"sint_neuzz.pdf\", labels=[f\"Run {i}\" for i in range(3)], ncols=1,\n",
    "                    label_placement=(0.01, 0.01))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "draw_integer_values(datasets=[rand_datasets[i] for i in [2]], identifier='test_case.sint',\n",
    "                    title=f\"Distribution of Signed Integer Values\", ylabel=\"Signed Integer Value\", num_datapoints=None,\n",
    "                    save_path=\"sint_random.pdf\", xticks=[0, 100000, 200000, 300000],\n",
    "                    labels=[f\"Run {i}\" for i in range(1)], draw_legend=False, big_font=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "draw_integer_values(datasets=[baseline_datasets[i] for i in [9]], identifier='test_case.sint',\n",
    "                    title=f\"Distribution of Signed Integer Values\", ylabel=\"Signed Integer Value\", num_datapoints=None,\n",
    "                    save_path=\"sint_baseline.pdf\", xticks=[0, 100000, 200000, 300000],\n",
    "                    labels=[f\"Run {i}\" for i in range(1)], draw_legend=False, big_font=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "draw_integer_values(datasets=[svm_datasets[i] for i in [2]], identifier='test_case.sint',\n",
    "                    title=f\"Distribution of Signed Integer Values\", ylabel=\"Signed Integer Value\", num_datapoints=None,\n",
    "                    save_path=\"sint_svm.pdf\", labels=[f\"Run {i}\" for i in range(1)], xticks=[0, 100000, 200000],\n",
    "                    draw_legend=False, big_font=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "jp-MarkdownHeadingCollapsed": true,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "### Throughput"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "print(\"Number of test cases that have been generated by the fuzzers:\\n\")\n",
    "num_tcs = []\n",
    "mins = []\n",
    "maxs = []\n",
    "means = []\n",
    "stds = []\n",
    "overhead_stats = {}\n",
    "for i, ds in enumerate(general_performance_datasets):\n",
    "    curr = [len(data[ds[j]]['datetime']) for j,_ in enumerate(ds)]\n",
    "    num_tcs.append(curr)\n",
    "    mins.append(min(curr))\n",
    "    maxs.append(max(curr))\n",
    "    means.append(sum(curr)/len(curr))\n",
    "    stds.append(np.array(curr).std(0))\n",
    "    print(f\"{general_performance_labels[i]}     max: {max(curr)}    min: {min(curr)}    mean: {sum(curr)/len(curr)}\")\n",
    "    overhead_stats[general_performance_labels[i]] = {'max': max(curr), 'min': min(curr), 'mean': sum(curr)/len(curr)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "print(\"Performance decrease in comparison to Random:\")\n",
    "for alg in general_performance_labels:\n",
    "    print(f\"{alg}: {overhead_stats['A_RANDOM']['mean'] - overhead_stats[alg]['mean']} ({round((overhead_stats['A_RANDOM']['mean'] - overhead_stats[alg]['mean'])/overhead_stats['A_RANDOM']['mean'] * 100,2)}%)\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots()\n",
    "ax.errorbar(np.arange(5), means, stds, fmt='ok')\n",
    "#ax.errorbar(np.arange(5), means, [means - mins, maxes - means], fmt='.k', ecolor='gray', lw=1)\n",
    "ax.set_xticks([0, 1, 2, 3, 4], general_performance_labels)\n",
    "ax.set_xlabel(\"Fuzzers\")\n",
    "ax.set_ylabel(\"Number of Test Cases sent in 24h\")\n",
    "ax.set_title(\"Number of Sent Test Cases per Fuzzer\")\n",
    "for item in ([ax.title, ax.xaxis.label, ax.yaxis.label] + ax.get_xticklabels() + ax.get_yticklabels()):\n",
    "    item.set_fontsize(13)\n",
    "fig.savefig(\"results/generated_test_cases.svg\", format=\"svg\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "jp-MarkdownHeadingCollapsed": true,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "### Choice of Feedback Interval\n",
    "\n",
    "Since the additional data for these figures and evaluations are quite big, we haven't loaded them yet. If you want to see them, you can run the code in this section but be warned that this takes some time and some memory to run. This is also the reason why this section is at the end of this Notebook and not in the right place following the structure of the paper."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "load_additional_data = False\n",
    "assert load_additional_data, \"If you really want to start loading all the remaining data, please set the value above to true.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "data_baseline_interval = load_files(glob.glob(os.path.join(abs_path, '**', '*.json'), recursive=True), load_feedback_interval_baseline)\n",
    "dataset_ids_fb_baseline_interval = list(data_baseline_interval.keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "baseline_datasets_500 = [d for d in dataset_ids_fb_baseline_interval if \"baseline\" in d and \"multiple\" in d and \"500\" in d][:10]\n",
    "baseline_datasets_10 = [d for d in dataset_ids_fb_baseline_interval if \"baseline\" in d and \"multiple\" in d and \"10_\" in d][:10]\n",
    "preprocessed_data = [[format_data(ds, data_baseline_interval) for ds in ds_list] for ds_list in [baseline_datasets_10, baseline_datasets_500]]\n",
    "preprocessed_data += [[format_data(ds, data) for ds in ds_list] for ds_list in [baseline_datasets]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "#### Draw the Vulnerabilities over Time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "draw_vulns(datasets=[baseline_datasets, baseline_datasets_500, baseline_datasets_10], actual_data=preprocessed_data, title=\"Performance of A_BASELINE Intervals\", xaxis='time', ylabel=\"#Vulnerabilities Triggered\", num_datapoints=None, draw_confidence_interval=True, labels=[\"10\", \"500\", \"50\"], label_placement=(0.67,0.01), save_path=\"feedback_interval_baseline.svg\", big_font=True, ncols=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "#### Show the Triggered Vulnerabilities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "print_triggered_vulns([baseline_datasets_10, baseline_datasets_500], data_baseline_interval, [\"Baseline 10\", \"Baseline 500\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "#### Calculate some Stats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "stats_values_last_baseline = {}\n",
    "\n",
    "ids = [\"10\", \"500\", \"50\"]\n",
    "for i, ds_list in enumerate(preprocessed_data):\n",
    "    vals = [ds['new_vuln'].iloc[-1] for ds in ds_list]\n",
    "    stats_values_last_baseline[ids[i]] = vals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "res_baseline = pd.DataFrame(0, index=ids, columns=ids)\n",
    "for i, j in list(itertools.product(ids, ids)):\n",
    "    r = scipy.stats.mannwhitneyu(stats_values_last_baseline[i], stats_values_last_baseline[j])\n",
    "    res_baseline.at[i, j] = round(r.pvalue, 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "res_baseline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "data_fb_interval = load_files(glob.glob(os.path.join(abs_path, '**', '*.json'), recursive=True), load_feedback_interval_dt_nn)\n",
    "dataset_ids_fb_interval = list(data_fb_interval.keys())\n",
    "dt_datasets_500 = [d for d in dataset_ids_fb_interval if \"mldriven\" in d and \"multiple\" in d and \"500\" in d][:10]\n",
    "dt_datasets_10 = [d for d in dataset_ids_fb_interval if \"mldriven\" in d and \"multiple\" in d and \"10_\" in d][:10]\n",
    "nn_datasets_500 = [d for d in dataset_ids_fb_interval if \"neuzz\" in d and \"multiple\" in d and \"500\" in d][:10]\n",
    "nn_datasets_10 = [d for d in dataset_ids_fb_interval if \"neuzz\" in d and \"multiple\" in d and \"10_\" in d][:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "dataset_ids_fb_interval"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "preprocessed_data = [[format_data(ds, data_fb_interval) for ds in ds_list] for ds_list in [dt_datasets_10, dt_datasets_500]]\n",
    "preprocessed_data += [[format_data(ds, data) for ds in ds_list] for ds_list in [dt_datasets]]\n",
    "draw_vulns(datasets=[dt_datasets, dt_datasets_500, dt_datasets_10], actual_data=preprocessed_data, title=\"Performance of A_DT\", xaxis='time', ylabel=\"#Vulnerabilities Triggered\", num_datapoints=None, draw_confidence_interval=True, labels=[\"10\", \"500\", \"50\"], label_placement=(0.67,0.01), save_path=\"feedback_interval_mldriven.svg\", big_font=True, ncols=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "preprocessed_data = [[format_data(ds, data_fb_interval) for ds in ds_list] for ds_list in [nn_datasets_10, nn_datasets_500]]\n",
    "preprocessed_data += [[format_data(ds, data) for ds in ds_list] for ds_list in [nn_datasets]]\n",
    "draw_vulns(datasets=[dt_datasets, nn_datasets_500, nn_datasets_10], actual_data=preprocessed_data, title=\"Performance of A_NN\", xaxis='time', ylabel=\"#Vulnerabilities Triggered\", num_datapoints=None, draw_confidence_interval=True, labels=[\"10\", \"500\", \"50\"], label_placement=(0.29,0.01), save_path=\"feedback_interval_neuzz.svg\", big_font=True, ncols=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}