BugClassificationAnalysis.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "dfa69d30-7ffc-4d7f-bd69-935334b96745",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as npa\n",
    "import math\n",
    "import matplotlib.pyplot as plt\n",
    "import warnings\n",
    "warnings.simplefilter(action='ignore', category=UserWarning)\n",
    "from IPython.display import display, HTML\n",
    "pd.set_option('display.max_colwidth', None)\n",
    "pd.set_option('display.max_rows',None)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "baae85ca-de55-4463-896b-9daca5e7f30f",
   "metadata": {},
   "source": [
    "# Bug Classification Analysis\n",
    "\n",
    "In this notebook we will show the degree of agreement reached between 4 different classifications.\n",
    "\n",
    "The classifications considered are the following:\n",
    "\n",
    "- **Codethink - Round 1:** 47 bugs classified by developers (several authors)\n",
    "- **Codethink - Round 2:** 10 bugs classified by developers (several authors)\n",
    "- **Ben Dooks:** 47 bugs classified by a Linux Kernel Developer\n",
    "- **Michel Maes:** 50 bugs classified by a researcher (only if its a bug-fixing commit or not)\n",
    "\n",
    "**Preliminary conclusions**:\n",
    "\n",
    "- In the \"*Did this fix a bug?*\" classification, we found similar results with slight differences in complex cases.\n",
    "- In the classification \"*Safety relevant bug?*\" we found a great disparity of results between Codethink and Ben Dooks (16 agree and 16 differ).\n",
    "- In the specific classification of safety relevant bug, we also found notable differences\n",
    "\n",
    "**Recommendations**:\n",
    "- It is necessary to clearly define what we understand as a \"safety related bug\" and that all evaluators have the same notion of the concept."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54655ba1-4149-4334-a80f-bc2f47d423f5",
   "metadata": {},
   "source": [
    "#### Data clean\n",
    "\n",
    "In order to carry out the analysis, we have made the following data processing decisions:\n",
    "\n",
    "- Yes/no answers have been changed to lower case.\n",
    "- If the answer contained a \"?\" (e.g. \"yes?\") the value without \"?\" was taken.\n",
    "- Considered \"?\", \"maybe\", \"possible\" as a third response **\"undetermined\"**=>**?**.\n",
    "- For the categories 'Timing + Execution', 'Memory' and 'Exchange of Info', if there was no answer, it was considered a \"no\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "0caf6df1-bba9-4865-bba7-89d00421877b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def showResults(results_df):\n",
    "    if len(results_df.index) > 0:\n",
    "        display(HTML( results_df.to_html().replace(\"\\\\n\",\"<br>\") ))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f894b50-cbef-4959-a225-e316d48373a7",
   "metadata": {},
   "source": [
    "## 1. Merging Codethink round's results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "8c0f12b9-a8b9-4cca-a9ed-f5eec8d596d8",
   "metadata": {},
   "outputs": [],
   "source": [
    "codethink_results_round_1 = pd.read_csv('Codethink Hivemind Linux Bug Classification - Round 1.csv') \n",
    "codethink_results_round_2 = pd.read_csv('Codethink Hivemind Linux Bug Classification - Round 2.csv') "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "116695a6-6440-4f8c-af91-86949280da2a",
   "metadata": {},
   "outputs": [],
   "source": [
    "codethink_results_round_1[['Name']] = codethink_results_round_1[['Name']].fillna(\"None\")\n",
    "codethink_results_round_2[['Name']] = codethink_results_round_2[['Name']].fillna(\"None\")\n",
    "\n",
    "for column in ['Timing + Execution', 'Memory', 'Exchange of Info']:\n",
    "    codethink_results_round_1[[column]] = codethink_results_round_1[[column]].fillna(\"no\")\n",
    "    codethink_results_round_2[[column]] = codethink_results_round_2[[column]].fillna(\"no\")\n",
    "\n",
    "for column in ['Did this fix a bug?','Safety relevant bug?','Timing + Execution', 'Memory', 'Exchange of Info']:\n",
    "    codethink_results_round_1[[column]] = codethink_results_round_1[[column]].replace(\"No\", \"no\")\n",
    "    codethink_results_round_1[[column]] = codethink_results_round_1[[column]].replace(\"Yes\", \"yes\")\n",
    "    codethink_results_round_1[[column]] = codethink_results_round_1[[column]].replace(\"maybe\", \"?\")\n",
    "    codethink_results_round_1[[column]] = codethink_results_round_1[[column]].replace(\"possible\", \"?\")\n",
    "    codethink_results_round_1[[column]] = codethink_results_round_1[[column]].replace(\"??\", \"?\")\n",
    "    \n",
    "    codethink_results_round_2[[column]] = codethink_results_round_2[[column]].replace(\"No\", \"no\")\n",
    "    codethink_results_round_2[[column]] = codethink_results_round_2[[column]].replace(\"Yes\", \"yes\")\n",
    "    codethink_results_round_2[[column]] = codethink_results_round_2[[column]].replace(\"maybe\", \"?\")\n",
    "    codethink_results_round_2[[column]] = codethink_results_round_2[[column]].replace(\"possible\", \"?\")\n",
    "    codethink_results_round_2[[column]] = codethink_results_round_2[[column]].replace(\"?\", \"?\")\n",
    "    codethink_results_round_2[[column]] = codethink_results_round_2[[column]].replace(\"??\", \"?\")\n",
    "\n",
    "codethink_results_round_1[[\"Reasoning\"]] = codethink_results_round_1[[\"Reasoning\"]].fillna(\"No reasoning\")\n",
    "codethink_results_round_2[[\"Reasoning\"]] = codethink_results_round_2[[\"Reasoning\"]].fillna(\"No reasoning\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0278ccd4-63aa-4d42-8852-2308ad8032a8",
   "metadata": {},
   "source": [
    "#### Classification - Codethink Round 1 (only commmits reviewed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "c4b0c628-7db1-45e7-9f00-0fdcaf2da457",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Round 1 - results count:  47\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>Did this fix a bug?</th>\n",
       "      <th>Safety relevant bug?</th>\n",
       "      <th>Timing + Execution</th>\n",
       "      <th>Memory</th>\n",
       "      <th>Exchange of Info</th>\n",
       "      <th>Reasoning</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Adnan Chowdhury</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>Fix of a race condition which could possibly corrupt memory</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>John G</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>\"could [stuff...]... causing a use-after-free memory access\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Hao Hu</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>Change SAS interface pointer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Hao Hu</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>Remove extra dirty node flag</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Basit A.</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>commit + superfluos function section data generation</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Aidan MacDonald</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>?</td>\n",
       "      <td>cluster filesystem locking error causing hang - according to commit message, difficult to verify otherwise</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Chris Phang</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>kbuild dependency fix</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Tom Bloor</td>\n",
       "      <td>yes</td>\n",
       "      <td>?</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>if youre rebooting during normal operation then... good luck?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Tom Bloor</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>wouldnt compile without this fix</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Aidan MacDonald</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>Kbuild fix</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>William Salmon</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>Incorect handling of memory results in` NULL pointer dereference` this fix avoids this case by checking before asigning pointer, this causes a kernel crash which would cascade to a system failure.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>William Salmon</td>\n",
       "      <td>yes</td>\n",
       "      <td>?</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>Fixes bug improving filesystem performace, could effect timing but would not be safety critical in a first order sence</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Weyman</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>Involves an event, a lock, and access of released resource (a crash)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>John G</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>Makes different things (\"DPIA\"?) preferred, sounds like enhancement rather than fx</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Paul Sherwood</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>bug related to timing in io function, so scheduling + info exchange?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>John G</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>Commit message said so, diff includes a free*()ing function</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Paul A</td>\n",
       "      <td>yes</td>\n",
       "      <td>?</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>[Why] section in commit message identifiies incorrect outcome \"userspace will react to the hotplug event based on a wrong state\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Ivan Orlov</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>Commit message + code which stops FS trimming when suspend</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Basit A.</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>commit message + interrupt service routines using mutexes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Sam L</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>Previous code doesn't reschedule packet, packets are now rescheduled properly I think??</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>Anton Pyntikov</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>Fix memory coherency</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>Sam L</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>Previous code locked the wrong lock...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>Basit A.</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>commit message + use after free</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Poppy</td>\n",
       "      <td>yes</td>\n",
       "      <td>?</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>Metrics table filler fails correctly.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>Sam L</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>I think HDP registers are only related to caching so not safety related</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>Paul Sherwood</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>No reasoning</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>Basit Ayantunde</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>commit message + underreserved memory</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>Paul Sherwood</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>i think this is just a slight performance improvement</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>Gavin McCall</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>\"not valid\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>Tom Bloor</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>Race condition leading to incorrect reference counting and use after free. also strawberries</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>Chris Phang</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>Fixing accuracy of error reporting</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>Dickon Hood</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>Socket handling error under error conditions</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>Nathan</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>Reverts a commit that had a false assumption that did an error out instead of a garbage out. This doesn't effect the outcome both still count as a fail.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>John G</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>Fixes issue in network address handling</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>Ivan Orlov</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>one instruction compiles as another</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>Frank M</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>Yes</td>\n",
       "      <td>no</td>\n",
       "      <td>Removes dereferencing of  null pointer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>John G</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>\"it makes page table base points to wrong address if...\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>Kas</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>Multiple folio fscache start calls can cause page dumps - simple check if the start has been called already fixes the bug.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>Ivan Orlov</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>fwnode is released somewhere else, so possible double free</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>JZ</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>race condition between adding and reading data, resulting in trying to add two entries with the same index</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>Aidan MacDonald</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>another kbuild fix!!</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>Tomas Veiga</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>No reasoning</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>Bob</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>yes?</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>i think this fixes a race condition when two processes are trying to write at once?  Though if someone is using FUSE or btrfs in a safety critical system we have other issues :D</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>Jeeeun</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>fixes the issue that igb module might hang or clash by clearing the variable</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>Tomas Veiga</td>\n",
       "      <td>yes</td>\n",
       "      <td>?</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>No reasoning</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>Gavin McCall</td>\n",
       "      <td>yes</td>\n",
       "      <td>?</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>Null data returned</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>Richard Dale</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>Fixes RISC-V memory alignment issue at startup</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               Name Did this fix a bug? Safety relevant bug?  \\\n",
       "0   Adnan Chowdhury                 yes                  yes   \n",
       "1            John G                 yes                  yes   \n",
       "2            Hao Hu                 yes                  yes   \n",
       "3            Hao Hu                  no                   no   \n",
       "4          Basit A.                 yes                   no   \n",
       "5   Aidan MacDonald                 yes                  yes   \n",
       "6       Chris Phang                 yes                   no   \n",
       "7         Tom Bloor                 yes                    ?   \n",
       "8         Tom Bloor                 yes                   no   \n",
       "9   Aidan MacDonald                 yes                   no   \n",
       "10   William Salmon                 yes                  yes   \n",
       "11   William Salmon                 yes                    ?   \n",
       "12           Weyman                 yes                  yes   \n",
       "13           John G                  no                   no   \n",
       "14    Paul Sherwood                 yes                  yes   \n",
       "15           John G                 yes                  yes   \n",
       "16           Paul A                 yes                    ?   \n",
       "17       Ivan Orlov                 yes                   no   \n",
       "18         Basit A.                 yes                   no   \n",
       "19            Sam L                 yes                   no   \n",
       "20   Anton Pyntikov                 yes                  yes   \n",
       "21            Sam L                 yes                  yes   \n",
       "22         Basit A.                 yes                  yes   \n",
       "23            Poppy                 yes                    ?   \n",
       "24            Sam L                 yes                   no   \n",
       "25    Paul Sherwood                 yes                   no   \n",
       "26  Basit Ayantunde                 yes                  yes   \n",
       "27    Paul Sherwood                  no                   no   \n",
       "28     Gavin McCall                 yes                   no   \n",
       "29        Tom Bloor                 yes                  yes   \n",
       "30      Chris Phang                 yes                   no   \n",
       "31      Dickon Hood                 yes                  yes   \n",
       "33           Nathan                 yes                   no   \n",
       "34           John G                 yes                  yes   \n",
       "36       Ivan Orlov                 yes                   no   \n",
       "37          Frank M                 yes                  yes   \n",
       "38           John G                 yes                  yes   \n",
       "39              Kas                 yes                  yes   \n",
       "40       Ivan Orlov                 yes                   no   \n",
       "41               JZ                 yes                  yes   \n",
       "42  Aidan MacDonald                 yes                   no   \n",
       "43      Tomas Veiga                 yes                   no   \n",
       "44              Bob                 yes                   no   \n",
       "45           Jeeeun                 yes                  yes   \n",
       "46      Tomas Veiga                 yes                    ?   \n",
       "48     Gavin McCall                 yes                    ?   \n",
       "49     Richard Dale                 yes                   no   \n",
       "\n",
       "   Timing + Execution Memory Exchange of Info  \\\n",
       "0                  no    yes               no   \n",
       "1                  no    yes               no   \n",
       "2                  no     no               no   \n",
       "3                 yes     no               no   \n",
       "4                  no     no               no   \n",
       "5                 yes     no                ?   \n",
       "6                  no     no               no   \n",
       "7                  no    yes               no   \n",
       "8                  no     no               no   \n",
       "9                  no     no               no   \n",
       "10                 no    yes               no   \n",
       "11                yes     no               no   \n",
       "12                yes     no               no   \n",
       "13                 no     no               no   \n",
       "14                yes     no              yes   \n",
       "15                 no    yes               no   \n",
       "16                yes     no              yes   \n",
       "17                yes     no              yes   \n",
       "18                yes     no               no   \n",
       "19                 no     no              yes   \n",
       "20                yes    yes               no   \n",
       "21                yes    yes               no   \n",
       "22                 no    yes              yes   \n",
       "23                 no     no              yes   \n",
       "24                yes     no               no   \n",
       "25                 no     no              yes   \n",
       "26                yes    yes               no   \n",
       "27                 no     no               no   \n",
       "28                yes     no               no   \n",
       "29                yes    yes               no   \n",
       "30                 no     no               no   \n",
       "31                 no    yes              yes   \n",
       "33                 no     no              yes   \n",
       "34                 no     no              yes   \n",
       "36                yes     no               no   \n",
       "37                 no   Yes                no   \n",
       "38                 no    yes               no   \n",
       "39                 no    yes               no   \n",
       "40                 no    yes               no   \n",
       "41                yes    yes              yes   \n",
       "42                 no     no               no   \n",
       "43                yes     no               no   \n",
       "44               yes?     no               no   \n",
       "45                 no    yes               no   \n",
       "46                yes     no               no   \n",
       "48                 no     no              yes   \n",
       "49                yes    yes               no   \n",
       "\n",
       "                                                                                                                                                                                               Reasoning  \n",
       "0                                                                                                                                            Fix of a race condition which could possibly corrupt memory  \n",
       "1                                                                                                                                           \"could [stuff...]... causing a use-after-free memory access\"  \n",
       "2                                                                                                                                                                           Change SAS interface pointer  \n",
       "3                                                                                                                                                                           Remove extra dirty node flag  \n",
       "4                                                                                                                                                   commit + superfluos function section data generation  \n",
       "5                                                                                             cluster filesystem locking error causing hang - according to commit message, difficult to verify otherwise  \n",
       "6                                                                                                                                                                                  kbuild dependency fix  \n",
       "7                                                                                                                                          if youre rebooting during normal operation then... good luck?  \n",
       "8                                                                                                                                                                       wouldnt compile without this fix  \n",
       "9                                                                                                                                                                                             Kbuild fix  \n",
       "10  Incorect handling of memory results in` NULL pointer dereference` this fix avoids this case by checking before asigning pointer, this causes a kernel crash which would cascade to a system failure.  \n",
       "11                                                                                Fixes bug improving filesystem performace, could effect timing but would not be safety critical in a first order sence  \n",
       "12                                                                                                                                  Involves an event, a lock, and access of released resource (a crash)  \n",
       "13                                                                                                                    Makes different things (\"DPIA\"?) preferred, sounds like enhancement rather than fx  \n",
       "14                                                                                                                                  bug related to timing in io function, so scheduling + info exchange?  \n",
       "15                                                                                                                                           Commit message said so, diff includes a free*()ing function  \n",
       "16                                                                      [Why] section in commit message identifiies incorrect outcome \"userspace will react to the hotplug event based on a wrong state\"  \n",
       "17                                                                                                                                            Commit message + code which stops FS trimming when suspend  \n",
       "18                                                                                                                                            commit message + interrupt service routines using mutexes   \n",
       "19                                                                                                               Previous code doesn't reschedule packet, packets are now rescheduled properly I think??  \n",
       "20                                                                                                                                                                                  Fix memory coherency  \n",
       "21                                                                                                                                                                Previous code locked the wrong lock...  \n",
       "22                                                                                                                                                                       commit message + use after free  \n",
       "23                                                                                                                                                                 Metrics table filler fails correctly.  \n",
       "24                                                                                                                               I think HDP registers are only related to caching so not safety related  \n",
       "25                                                                                                                                                                                          No reasoning  \n",
       "26                                                                                                                                                                 commit message + underreserved memory  \n",
       "27                                                                                                                                                 i think this is just a slight performance improvement  \n",
       "28                                                                                                                                                                                           \"not valid\"  \n",
       "29                                                                                                          Race condition leading to incorrect reference counting and use after free. also strawberries  \n",
       "30                                                                                                                                                                    Fixing accuracy of error reporting  \n",
       "31                                                                                                                                                          Socket handling error under error conditions  \n",
       "33                                              Reverts a commit that had a false assumption that did an error out instead of a garbage out. This doesn't effect the outcome both still count as a fail.  \n",
       "34                                                                                                                                                               Fixes issue in network address handling  \n",
       "36                                                                                                                                                                   one instruction compiles as another  \n",
       "37                                                                                                                                                                Removes dereferencing of  null pointer  \n",
       "38                                                                                                                                              \"it makes page table base points to wrong address if...\"  \n",
       "39                                                                           Multiple folio fscache start calls can cause page dumps - simple check if the start has been called already fixes the bug.   \n",
       "40                                                                                                                                            fwnode is released somewhere else, so possible double free  \n",
       "41                                                                                            race condition between adding and reading data, resulting in trying to add two entries with the same index  \n",
       "42                                                                                                                                                                                  another kbuild fix!!  \n",
       "43                                                                                                                                                                                          No reasoning  \n",
       "44                     i think this fixes a race condition when two processes are trying to write at once?  Though if someone is using FUSE or btrfs in a safety critical system we have other issues :D  \n",
       "45                                                                                                                          fixes the issue that igb module might hang or clash by clearing the variable  \n",
       "46                                                                                                                                                                                          No reasoning  \n",
       "48                                                                                                                                                                                    Null data returned  \n",
       "49                                                                                                                                                        Fixes RISC-V memory alignment issue at startup  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "codethink_results_round_1_filled = codethink_results_round_1[codethink_results_round_1['Name'] != \"None\"]\n",
    "print(\"Round 1 - results count: \",len(codethink_results_round_1_filled))\n",
    "(\n",
    "    codethink_results_round_1_filled\n",
    "    [[\"Name\",\"Did this fix a bug?\",\"Safety relevant bug?\",\"Timing + Execution\",\"Memory\",\"Exchange of Info\",\"Reasoning\"]]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a7947f0-14ac-492d-92bd-88c4e53f9379",
   "metadata": {},
   "source": [
    "#### Classification - Codethink Round 2 (only commmits reviewed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "5e7bfded-6b54-4612-b43b-e3d784356527",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Round 2 - results count:  10\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>Did this fix a bug?</th>\n",
       "      <th>Safety relevant bug?</th>\n",
       "      <th>Timing + Execution</th>\n",
       "      <th>Memory</th>\n",
       "      <th>Exchange of Info</th>\n",
       "      <th>Reasoning</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Chris Phang</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>NULL pointer dereference bug in eventfs, which is used for tracing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Weyman</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>About adjusting flow of something, an optimization</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>Paul Sherwood</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>the commit message doesn't give enough info to understand this</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>Gavin McCall</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>use after free</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>Aidan MacDonald</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>looks like possible data corruption in networking related code</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>Gavin McCall</td>\n",
       "      <td>yes</td>\n",
       "      <td>?</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>NULL pointer is dereferenced</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>Anton Pyntikov</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>Memory corruption if phys addr beyond 64Gb</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>Aidan MacDonald</td>\n",
       "      <td>yes</td>\n",
       "      <td>?</td>\n",
       "      <td>?</td>\n",
       "      <td>no</td>\n",
       "      <td>?</td>\n",
       "      <td>SCSI adapter drops all drives after suspend/resume cycle</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>Aidan MacDonald</td>\n",
       "      <td>yes</td>\n",
       "      <td>?</td>\n",
       "      <td>?</td>\n",
       "      <td>?</td>\n",
       "      <td>?</td>\n",
       "      <td>rmmod of module causes hang or crash</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>Adnan Chowdhury</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>This fixes a null pointer reference error on some systems with APU</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               Name Did this fix a bug? Safety relevant bug?  \\\n",
       "10      Chris Phang                 yes                  yes   \n",
       "16           Weyman                  no                   no   \n",
       "20    Paul Sherwood                 yes                   no   \n",
       "29     Gavin McCall                 yes                  yes   \n",
       "31  Aidan MacDonald                 yes                  yes   \n",
       "37     Gavin McCall                 yes                    ?   \n",
       "38   Anton Pyntikov                 yes                  yes   \n",
       "43  Aidan MacDonald                 yes                    ?   \n",
       "45  Aidan MacDonald                 yes                    ?   \n",
       "48  Adnan Chowdhury                 yes                  yes   \n",
       "\n",
       "   Timing + Execution Memory Exchange of Info  \\\n",
       "10                 no    yes               no   \n",
       "16                yes     no               no   \n",
       "20                 no     no               no   \n",
       "29                yes     no               no   \n",
       "31                 no    yes              yes   \n",
       "37                yes     no               no   \n",
       "38                yes    yes               no   \n",
       "43                  ?     no                ?   \n",
       "45                  ?      ?                ?   \n",
       "48                 no     no               no   \n",
       "\n",
       "                                                             Reasoning  \n",
       "10  NULL pointer dereference bug in eventfs, which is used for tracing  \n",
       "16                  About adjusting flow of something, an optimization  \n",
       "20      the commit message doesn't give enough info to understand this  \n",
       "29                                                      use after free  \n",
       "31      looks like possible data corruption in networking related code  \n",
       "37                                        NULL pointer is dereferenced  \n",
       "38                          Memory corruption if phys addr beyond 64Gb  \n",
       "43            SCSI adapter drops all drives after suspend/resume cycle  \n",
       "45                                rmmod of module causes hang or crash  \n",
       "48  This fixes a null pointer reference error on some systems with APU  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "codethink_results_round_2_filled = codethink_results_round_2[codethink_results_round_2['Name'] != \"None\"]\n",
    "print(\"Round 2 - results count: \",len(codethink_results_round_2_filled))\n",
    "(\n",
    "    codethink_results_round_2_filled\n",
    "    [[\"Name\",\"Did this fix a bug?\",\"Safety relevant bug?\",\"Timing + Execution\",\"Memory\",\"Exchange of Info\",\"Reasoning\"]]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "2fab6e1d-7153-4b86-b901-0a1d53cea27b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def getComparativeMatrix(column, set1, set2, name1, name2):\n",
    "\n",
    "    results_1 = set1.to_dict('records')\n",
    "    results_2 = set2.to_dict('records')\n",
    "    count_result = {\n",
    "        'YES': { 'YES': 0, 'NO':0, '?': 0 },\n",
    "        'NO':  { 'YES': 0, 'NO':0, '?': 0 },\n",
    "        '?':   { 'YES': 0, 'NO':0, '?': 0 },\n",
    "    }\n",
    "    \n",
    "    for idx, result_1 in enumerate(results_1):\n",
    "        result_2 = results_2[idx]\n",
    "\n",
    "        if not isinstance(result_2['Did this fix a bug?'], str): continue # Only values in Results 1\n",
    "    \n",
    "        if not isinstance(result_1['Did this fix a bug?'], str): continue # Only values in Results 2\n",
    "\n",
    "        if result_1[column] not in ['yes', 'no', '?']: continue # Discard conflict results\n",
    "        if result_2[column] not in ['yes', 'no', '?']: continue # Discard conflict results\n",
    "\n",
    "        r1_key = result_1[column].upper().strip()\n",
    "        r2_key = result_2[column].upper().strip()\n",
    "        \n",
    "        # Fill Matrix\n",
    "        if column == \"Did this fix a bug?\":\n",
    "            count_result[r1_key][r2_key] += 1\n",
    "        elif column == \"Safety relevant bug?\":\n",
    "            if result_1[\"Did this fix a bug?\"] == result_2[\"Did this fix a bug?\"]:\n",
    "                count_result[r1_key][r2_key] += 1\n",
    "        elif column in ['Timing + Execution', 'Memory', 'Exchange of Info']:\n",
    "            if result_1[\"Safety relevant bug?\"] == result_2[\"Safety relevant bug?\"]:\n",
    "                count_result[r1_key][r2_key] += 1\n",
    "    display(pd.DataFrame(count_result, \n",
    "        index=pd.Index(['YES', 'NO', '?'], name=name2+':'),\n",
    "    ).style.set_caption(\"<b style='float: right;'>\"+name1+\":</b>\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b74a7fcf-8068-41a4-bbb8-b7af03b6382e",
   "metadata": {},
   "source": [
    "#### Comparative results (Codethink - Round 1 vs Codethink - Round 2)\n",
    "\n",
    "The following results show only the agreements and discrepancies between the evaluated commits in both sets of results (10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "5e42f131-6019-4b1d-b12c-51624b753d4a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "</style>\n",
       "<table id=\"T_fa03c\">\n",
       "  <caption><b style='float: right;'>Codethink R1:</b></caption>\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_fa03c_level0_col0\" class=\"col_heading level0 col0\" >YES</th>\n",
       "      <th id=\"T_fa03c_level0_col1\" class=\"col_heading level0 col1\" >NO</th>\n",
       "      <th id=\"T_fa03c_level0_col2\" class=\"col_heading level0 col2\" >?</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th class=\"index_name level0\" >Codethink R2:</th>\n",
       "      <th class=\"blank col0\" >&nbsp;</th>\n",
       "      <th class=\"blank col1\" >&nbsp;</th>\n",
       "      <th class=\"blank col2\" >&nbsp;</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_fa03c_level0_row0\" class=\"row_heading level0 row0\" >YES</th>\n",
       "      <td id=\"T_fa03c_row0_col0\" class=\"data row0 col0\" >9</td>\n",
       "      <td id=\"T_fa03c_row0_col1\" class=\"data row0 col1\" >0</td>\n",
       "      <td id=\"T_fa03c_row0_col2\" class=\"data row0 col2\" >0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_fa03c_level0_row1\" class=\"row_heading level0 row1\" >NO</th>\n",
       "      <td id=\"T_fa03c_row1_col0\" class=\"data row1 col0\" >1</td>\n",
       "      <td id=\"T_fa03c_row1_col1\" class=\"data row1 col1\" >0</td>\n",
       "      <td id=\"T_fa03c_row1_col2\" class=\"data row1 col2\" >0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_fa03c_level0_row2\" class=\"row_heading level0 row2\" >?</th>\n",
       "      <td id=\"T_fa03c_row2_col0\" class=\"data row2 col0\" >0</td>\n",
       "      <td id=\"T_fa03c_row2_col1\" class=\"data row2 col1\" >0</td>\n",
       "      <td id=\"T_fa03c_row2_col2\" class=\"data row2 col2\" >0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7f3345c77190>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "getComparativeMatrix(\"Did this fix a bug?\",codethink_results_round_1, codethink_results_round_2, \"Codethink R1\", \"Codethink R2\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "82a9496c-3bda-467f-b554-1014813ef4bd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "</style>\n",
       "<table id=\"T_fa04b\">\n",
       "  <caption><b style='float: right;'>Codethink R1:</b></caption>\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_fa04b_level0_col0\" class=\"col_heading level0 col0\" >YES</th>\n",
       "      <th id=\"T_fa04b_level0_col1\" class=\"col_heading level0 col1\" >NO</th>\n",
       "      <th id=\"T_fa04b_level0_col2\" class=\"col_heading level0 col2\" >?</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th class=\"index_name level0\" >Codethink R2:</th>\n",
       "      <th class=\"blank col0\" >&nbsp;</th>\n",
       "      <th class=\"blank col1\" >&nbsp;</th>\n",
       "      <th class=\"blank col2\" >&nbsp;</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_fa04b_level0_row0\" class=\"row_heading level0 row0\" >YES</th>\n",
       "      <td id=\"T_fa04b_row0_col0\" class=\"data row0 col0\" >4</td>\n",
       "      <td id=\"T_fa04b_row0_col1\" class=\"data row0 col1\" >0</td>\n",
       "      <td id=\"T_fa04b_row0_col2\" class=\"data row0 col2\" >1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_fa04b_level0_row1\" class=\"row_heading level0 row1\" >NO</th>\n",
       "      <td id=\"T_fa04b_row1_col0\" class=\"data row1 col0\" >1</td>\n",
       "      <td id=\"T_fa04b_row1_col1\" class=\"data row1 col1\" >0</td>\n",
       "      <td id=\"T_fa04b_row1_col2\" class=\"data row1 col2\" >0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_fa04b_level0_row2\" class=\"row_heading level0 row2\" >?</th>\n",
       "      <td id=\"T_fa04b_row2_col0\" class=\"data row2 col0\" >2</td>\n",
       "      <td id=\"T_fa04b_row2_col1\" class=\"data row2 col1\" >1</td>\n",
       "      <td id=\"T_fa04b_row2_col2\" class=\"data row2 col2\" >0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7f3345861590>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "getComparativeMatrix(\"Safety relevant bug?\",codethink_results_round_1, codethink_results_round_2, \"Codethink R1\", \"Codethink R2\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "f2909bc9-56d3-4695-a3dd-1d09d459ac53",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Timing + Execution'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "</style>\n",
       "<table id=\"T_ed247\">\n",
       "  <caption><b style='float: right;'>Codethink R1:</b></caption>\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_ed247_level0_col0\" class=\"col_heading level0 col0\" >YES</th>\n",
       "      <th id=\"T_ed247_level0_col1\" class=\"col_heading level0 col1\" >NO</th>\n",
       "      <th id=\"T_ed247_level0_col2\" class=\"col_heading level0 col2\" >?</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th class=\"index_name level0\" >Codethink R2:</th>\n",
       "      <th class=\"blank col0\" >&nbsp;</th>\n",
       "      <th class=\"blank col1\" >&nbsp;</th>\n",
       "      <th class=\"blank col2\" >&nbsp;</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_ed247_level0_row0\" class=\"row_heading level0 row0\" >YES</th>\n",
       "      <td id=\"T_ed247_row0_col0\" class=\"data row0 col0\" >1</td>\n",
       "      <td id=\"T_ed247_row0_col1\" class=\"data row0 col1\" >1</td>\n",
       "      <td id=\"T_ed247_row0_col2\" class=\"data row0 col2\" >0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_ed247_level0_row1\" class=\"row_heading level0 row1\" >NO</th>\n",
       "      <td id=\"T_ed247_row1_col0\" class=\"data row1 col0\" >0</td>\n",
       "      <td id=\"T_ed247_row1_col1\" class=\"data row1 col1\" >2</td>\n",
       "      <td id=\"T_ed247_row1_col2\" class=\"data row1 col2\" >0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_ed247_level0_row2\" class=\"row_heading level0 row2\" >?</th>\n",
       "      <td id=\"T_ed247_row2_col0\" class=\"data row2 col0\" >0</td>\n",
       "      <td id=\"T_ed247_row2_col1\" class=\"data row2 col1\" >0</td>\n",
       "      <td id=\"T_ed247_row2_col2\" class=\"data row2 col2\" >0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7f3345855590>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'Memory'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "</style>\n",
       "<table id=\"T_2bfb3\">\n",
       "  <caption><b style='float: right;'>Codethink R1:</b></caption>\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_2bfb3_level0_col0\" class=\"col_heading level0 col0\" >YES</th>\n",
       "      <th id=\"T_2bfb3_level0_col1\" class=\"col_heading level0 col1\" >NO</th>\n",
       "      <th id=\"T_2bfb3_level0_col2\" class=\"col_heading level0 col2\" >?</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th class=\"index_name level0\" >Codethink R2:</th>\n",
       "      <th class=\"blank col0\" >&nbsp;</th>\n",
       "      <th class=\"blank col1\" >&nbsp;</th>\n",
       "      <th class=\"blank col2\" >&nbsp;</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_2bfb3_level0_row0\" class=\"row_heading level0 row0\" >YES</th>\n",
       "      <td id=\"T_2bfb3_row0_col0\" class=\"data row0 col0\" >3</td>\n",
       "      <td id=\"T_2bfb3_row0_col1\" class=\"data row0 col1\" >0</td>\n",
       "      <td id=\"T_2bfb3_row0_col2\" class=\"data row0 col2\" >0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_2bfb3_level0_row1\" class=\"row_heading level0 row1\" >NO</th>\n",
       "      <td id=\"T_2bfb3_row1_col0\" class=\"data row1 col0\" >1</td>\n",
       "      <td id=\"T_2bfb3_row1_col1\" class=\"data row1 col1\" >0</td>\n",
       "      <td id=\"T_2bfb3_row1_col2\" class=\"data row1 col2\" >0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_2bfb3_level0_row2\" class=\"row_heading level0 row2\" >?</th>\n",
       "      <td id=\"T_2bfb3_row2_col0\" class=\"data row2 col0\" >0</td>\n",
       "      <td id=\"T_2bfb3_row2_col1\" class=\"data row2 col1\" >0</td>\n",
       "      <td id=\"T_2bfb3_row2_col2\" class=\"data row2 col2\" >0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7f334584ee90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'Exchange of Info'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "</style>\n",
       "<table id=\"T_038d6\">\n",
       "  <caption><b style='float: right;'>Codethink R1:</b></caption>\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_038d6_level0_col0\" class=\"col_heading level0 col0\" >YES</th>\n",
       "      <th id=\"T_038d6_level0_col1\" class=\"col_heading level0 col1\" >NO</th>\n",
       "      <th id=\"T_038d6_level0_col2\" class=\"col_heading level0 col2\" >?</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th class=\"index_name level0\" >Codethink R2:</th>\n",
       "      <th class=\"blank col0\" >&nbsp;</th>\n",
       "      <th class=\"blank col1\" >&nbsp;</th>\n",
       "      <th class=\"blank col2\" >&nbsp;</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_038d6_level0_row0\" class=\"row_heading level0 row0\" >YES</th>\n",
       "      <td id=\"T_038d6_row0_col0\" class=\"data row0 col0\" >1</td>\n",
       "      <td id=\"T_038d6_row0_col1\" class=\"data row0 col1\" >0</td>\n",
       "      <td id=\"T_038d6_row0_col2\" class=\"data row0 col2\" >0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_038d6_level0_row1\" class=\"row_heading level0 row1\" >NO</th>\n",
       "      <td id=\"T_038d6_row1_col0\" class=\"data row1 col0\" >0</td>\n",
       "      <td id=\"T_038d6_row1_col1\" class=\"data row1 col1\" >3</td>\n",
       "      <td id=\"T_038d6_row1_col2\" class=\"data row1 col2\" >0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_038d6_level0_row2\" class=\"row_heading level0 row2\" >?</th>\n",
       "      <td id=\"T_038d6_row2_col0\" class=\"data row2 col0\" >0</td>\n",
       "      <td id=\"T_038d6_row2_col1\" class=\"data row2 col1\" >0</td>\n",
       "      <td id=\"T_038d6_row2_col2\" class=\"data row2 col2\" >0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7f334584ec10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "for column in ['Timing + Execution', 'Memory', 'Exchange of Info']:\n",
    "    display(column)\n",
    "    getComparativeMatrix(column,codethink_results_round_1, codethink_results_round_2, \"Codethink R1\", \"Codethink R2\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "a06847b3-a2d8-4b57-b341-0328449916ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "def setConflict(result_1, result_2, column, changeOtherColumns=True):\n",
    "    result_1[column] = 'Conflict: '+str(result_1[column])+'/'+str(result_2[column])\n",
    "    if changeOtherColumns:\n",
    "        result_1['Reasoning'] = '#' + str(result_1['Name']) + '# ' + str(result_1['Reasoning']) + '\\\\n' + '#' + str(result_2['Name'])+ '# ' + str(result_2['Reasoning'])\n",
    "        result_1['Name'] = str(result_1['Name']) + \"/\" + str(result_2['Name'])\n",
    "\n",
    "def solveConflict(result_1, result_2):\n",
    "    # Same result on BFC identification\n",
    "    if result_2['Did this fix a bug?'] == result_1['Did this fix a bug?']:\n",
    "\n",
    "        # Same result on safety/non-safety relevant bug\n",
    "        if result_2['Safety relevant bug?'] == result_1['Safety relevant bug?']:\n",
    "            new_result = result_1.copy()\n",
    "            any_conflict = False\n",
    "            \n",
    "            for column in ['Timing + Execution', 'Memory', 'Exchange of Info']:\n",
    "                if result_2[column] != result_1[column]:\n",
    "                    setConflict(new_result,result_2,column, changeOtherColumns= not any_conflict)\n",
    "                    any_conflict = True\n",
    "\n",
    "            if any_conflict:\n",
    "                # Conflicts on classification\n",
    "                return new_result\n",
    "            else:\n",
    "                # No conflicts in classification\n",
    "                return result_1\n",
    "        \n",
    "        # Different result on safety/non-safety relevant bug\n",
    "        else: \n",
    "            new_result = result_1.copy()\n",
    "            setConflict(new_result,result_2,'Safety relevant bug?')\n",
    "            return new_result\n",
    "    # Different result on BFC identification\n",
    "    else: \n",
    "        new_result = result_1.copy()\n",
    "        setConflict(new_result,result_2,'Did this fix a bug?')\n",
    "        return new_result\n",
    "        \n",
    "def combineResults(set1, set2):\n",
    "\n",
    "    results_1 = set1.to_dict('records')\n",
    "    results_2 = set2.to_dict('records')\n",
    "\n",
    "    combined_results = []\n",
    "\n",
    "    for idx, result_1 in enumerate(results_1):\n",
    "        result_2 = results_2[idx]\n",
    "\n",
    "        if not isinstance(result_2['Did this fix a bug?'], str): # Only values in Results 1\n",
    "            combined_results.append(result_1)\n",
    "            continue # Only values in Results 1\n",
    "    \n",
    "        if not isinstance(result_1['Did this fix a bug?'], str): # Only values in Results 2\n",
    "            combined_results.append(result_2)\n",
    "            continue # Only values in Results 2\n",
    "\n",
    "        # Values in Resuls 1 and results 2\n",
    "        new_result = solveConflict(result_1, result_2)\n",
    "        combined_results.append(new_result)\n",
    "        \n",
    "    return pd.DataFrame.from_dict(combined_results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "304f5f55-3d7b-475c-a627-7fe5016983cd",
   "metadata": {},
   "outputs": [],
   "source": [
    "combined_codethink_results_df = combineResults(codethink_results_round_1, codethink_results_round_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2567e97f-8c42-4b34-9d3c-c81444ff1765",
   "metadata": {},
   "source": [
    "### 1.1 Conflicts in bug-fixing commit identification (Codethink Round 1 vs Round 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "d0ac464c-68f4-4edd-9585-5f83aa6f1190",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>Did this fix a bug?</th>\n",
       "      <th>Reasoning</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Paul A/Weyman</td>\n",
       "      <td>Conflict: yes/no</td>\n",
       "      <td>#Paul A# [Why] section in commit message identifiies incorrect outcome \"userspace will react to the hotplug event based on a wrong state\"<br>#Weyman# About adjusting flow of something, an optimization</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "combined_codethink_results_df_filtered = combined_codethink_results_df[combined_codethink_results_df['Did this fix a bug?'].notnull()]\n",
    "showResults(\n",
    "    combined_codethink_results_df_filtered[combined_codethink_results_df_filtered['Did this fix a bug?'].str.contains('Conflict')]\n",
    "    [['Name','Did this fix a bug?','Reasoning']]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4603478d-01be-45c3-aa4c-862610aa38b4",
   "metadata": {},
   "source": [
    "### 1.2 Conflicts in safety bugs identification (Codethink Round 1 vs Round 2)\n",
    "\n",
    "**Note**: At this point, we only consider those commits for which the same result has been obtained by classifying \"Did this fix a bug?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "40655682-336c-4428-8871-12831f1d8aef",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>Did this fix a bug?</th>\n",
       "      <th>Safety relevant bug?</th>\n",
       "      <th>Reasoning</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>Anton Pyntikov/Paul Sherwood</td>\n",
       "      <td>yes</td>\n",
       "      <td>Conflict: yes/no</td>\n",
       "      <td>#Anton Pyntikov# Fix memory coherency<br>#Paul Sherwood# the commit message doesn't give enough info to understand this</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>Frank M/Gavin McCall</td>\n",
       "      <td>yes</td>\n",
       "      <td>Conflict: yes/?</td>\n",
       "      <td>#Frank M# Removes dereferencing of  null pointer<br>#Gavin McCall# NULL pointer is dereferenced</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>Tomas Veiga/Aidan MacDonald</td>\n",
       "      <td>yes</td>\n",
       "      <td>Conflict: no/?</td>\n",
       "      <td>#Tomas Veiga# No reasoning<br>#Aidan MacDonald# SCSI adapter drops all drives after suspend/resume cycle</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>Jeeeun/Aidan MacDonald</td>\n",
       "      <td>yes</td>\n",
       "      <td>Conflict: yes/?</td>\n",
       "      <td>#Jeeeun# fixes the issue that igb module might hang or clash by clearing the variable<br>#Aidan MacDonald# rmmod of module causes hang or crash</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>Gavin McCall/Adnan Chowdhury</td>\n",
       "      <td>yes</td>\n",
       "      <td>Conflict: ?/yes</td>\n",
       "      <td>#Gavin McCall# Null data returned<br>#Adnan Chowdhury# This fixes a null pointer reference error on some systems with APU</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "combined_codethink_results_df_filtered = combined_codethink_results_df[combined_codethink_results_df['Safety relevant bug?'].notnull()]\n",
    "showResults(\n",
    "    combined_codethink_results_df_filtered[combined_codethink_results_df_filtered['Safety relevant bug?'].str.contains('Conflict')]\n",
    "    [['Name','Did this fix a bug?','Safety relevant bug?','Reasoning']]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6844d2ca-99f4-47b2-aeb5-127a427ce36d",
   "metadata": {},
   "source": [
    "### 1.3 Conflicts in safety classification (Codethink Round 1 vs Round 2)\n",
    "\n",
    "**Note**: At this point, we only consider those commits for which the same result has been obtained by classifying \"Safety relevant bug?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "645f3cdf-b32b-4a59-b488-92287a504d12",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Timing + Execution'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>Did this fix a bug?</th>\n",
       "      <th>Safety relevant bug?</th>\n",
       "      <th>Timing + Execution</th>\n",
       "      <th>Reasoning</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>John G/Anton Pyntikov</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>Conflict: no/yes</td>\n",
       "      <td>#John G# \"it makes page table base points to wrong address if...\"<br>#Anton Pyntikov# Memory corruption if phys addr beyond 64Gb</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'Memory'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>Did this fix a bug?</th>\n",
       "      <th>Safety relevant bug?</th>\n",
       "      <th>Memory</th>\n",
       "      <th>Reasoning</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>Tom Bloor/Gavin McCall</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>Conflict: yes/no</td>\n",
       "      <td>#Tom Bloor# Race condition leading to incorrect reference counting and use after free. also strawberries<br>#Gavin McCall# use after free</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'Exchange of Info'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "for column in ['Timing + Execution', 'Memory', 'Exchange of Info']:\n",
    "    display(column)\n",
    "    combined_codethink_results_df_filtered = combined_codethink_results_df[combined_codethink_results_df[column].notnull()]\n",
    "    showResults(\n",
    "        combined_codethink_results_df_filtered[combined_codethink_results_df_filtered[column].str.contains('Conflict')]\n",
    "        [['Name','Did this fix a bug?','Safety relevant bug?',column,'Reasoning']]\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f44a67d1-c39c-4a49-a336-1e6def37f864",
   "metadata": {},
   "source": [
    "### 1.4 Results of CodeThink (Round 1 + Round 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "bfea55e2-d597-4b84-97a0-b0a44a117fa4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BFC identified:  43\n",
      "Safety relevant bugs identified:  17\n",
      "   > Timing + Execution:  7\n",
      "   > Memory:  11\n",
      "   > Exchange of Info:  5\n",
      "Conflicts: \n",
      "    > Did this fix a bug?:  1\n",
      "    > Safety relevant bug?:  5\n",
      "    > Timing + Execution:  1\n",
      "    > Memory:  1\n",
      "    > Exchange of Info:  0\n"
     ]
    }
   ],
   "source": [
    "def reportResuts(results, show_conflicts=True):\n",
    "    results_filtered = results[results['Name'].notnull()]\n",
    "    print(\"BFC identified: \",len(results_filtered[results_filtered['Did this fix a bug?']==\"yes\"]))\n",
    "    print(\"Safety relevant bugs identified: \",len(results_filtered[results_filtered['Safety relevant bug?']==\"yes\"]))\n",
    "    for column in ['Timing + Execution', 'Memory', 'Exchange of Info']:\n",
    "        print(\"   > \"+column+\": \",\n",
    "            len(results_filtered.loc[\n",
    "                (results_filtered['Safety relevant bug?']==\"yes\") & (results[column]==\"yes\")]     \n",
    "            )\n",
    "        )\n",
    "    if show_conflicts:\n",
    "        print(\"Conflicts: \")\n",
    "        for column in ['Did this fix a bug?', 'Safety relevant bug?', 'Timing + Execution', 'Memory', 'Exchange of Info']:\n",
    "            results_filtered_aux = results_filtered[results_filtered[column].notnull()]\n",
    "            print(\"    > \"+column+\": \", len(results_filtered_aux[results_filtered_aux[column].str.contains('Conflict')]))\n",
    "reportResuts(combined_codethink_results_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f25ad05-ae3b-4edd-a210-6f0a96c95643",
   "metadata": {},
   "source": [
    "## 2. Ben Dooks results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "9596c43f-d943-4fb3-b1a2-0838e7d91d0e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BFC identified:  40\n",
      "Safety relevant bugs identified:  14\n",
      "   > Timing + Execution:  1\n",
      "   > Memory:  8\n",
      "   > Exchange of Info:  0\n"
     ]
    }
   ],
   "source": [
    "codethink_results_ben_dooks = pd.read_csv('Codethink Hivemind Linux Bug Classification - Ben Dooks.csv')\n",
    "codethink_results_ben_dooks['Name'] = [\"Ben D.\"] * 50\n",
    "for column in ['Timing + Execution', 'Memory', 'Exchange of Info']:\n",
    "    codethink_results_ben_dooks[[column]] = codethink_results_ben_dooks[[column]].fillna(\"no\")\n",
    "    codethink_results_ben_dooks[[column]] = codethink_results_ben_dooks[[column]].fillna(\"no\")\n",
    "\n",
    "for column in ['Did this fix a bug?','Safety relevant bug?','Timing + Execution', 'Memory', 'Exchange of Info']:\n",
    "    codethink_results_ben_dooks[[column]] = codethink_results_ben_dooks[[column]].replace(\"maybe?\", \"?\")\n",
    "    codethink_results_ben_dooks[[column]] = codethink_results_ben_dooks[[column]].replace(\"no?\", \"no\")\n",
    "    codethink_results_ben_dooks[[column]] = codethink_results_ben_dooks[[column]].replace(\"yes?\", \"yes\")\n",
    "codethink_results_ben_dooks[[\"Reasoning\"]] = codethink_results_ben_dooks[[\"Reasoning\"]].fillna(\"No reasoning\")\n",
    "reportResuts(codethink_results_ben_dooks, show_conflicts=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "939f8155-aeaf-4b77-be89-5103819a7d66",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ben Dooks- results count:  47\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>Did this fix a bug?</th>\n",
       "      <th>Safety relevant bug?</th>\n",
       "      <th>Timing + Execution</th>\n",
       "      <th>Memory</th>\n",
       "      <th>Exchange of Info</th>\n",
       "      <th>Reasoning</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>locking and reference fix in dm layer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>?</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>scsi driver fix for lock after free</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>?</td>\n",
       "      <td>fixes specific scsi scanning bug</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>removed unnecessary work already done elsewhere</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>removes warning due to linker output</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>fixes gfs2 locking bug over suspend/resume</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>build configuration dependency issue</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>reliable boot failure if pagetable size is set to 5 levels</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>kernel page allocation api change</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>build configuration dependency issue</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>bad memory de-reference may destabilise system</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>kernel page allocation api change</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>possible lock usage on freed memory</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>bug is hardware assignment at init time</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>possible sleep in invalid place may cause system hang</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>jbd2 may fail to release memory in error paths</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>resume may send invalid hotplug events</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>?</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>ext4 fstrim may block suspend</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>fixes deadlock in drm/vkms driver</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>?</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>nfs requests may be lost - maybe get retried?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>?</td>\n",
       "      <td>?</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>not enough info in commit</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>incorrect locking in nfs</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>possible read of invalid memory / BUG trigger</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>caller should detect invalid response and retry metrics</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>?</td>\n",
       "      <td>?</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>No reasoning</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>?</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>nfs bugfix</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>nvme init timeout fix - will always timeout with bad device</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>kunit test bug</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>backlight bug on certain amd hw during shutdown</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>?</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>use after free during driver removal - probably not fatal to the runtime of the system</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>accounting bug on error path</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>could cause errors in the network data quees</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>?</td>\n",
       "      <td>no</td>\n",
       "      <td>nfs issue where data will be retransmitted - todo check if memory leak?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>?</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>network socket lookup with ipv4/ipv6 interaction -</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>removes extra unused branch protection instructions</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>nno</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>fixes OOPS if procfs file gets accessed</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>?</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>fix for amd gpu page-table code for large memory systems</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>nfs oops can be generated at run time</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>?</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>extra fwnode put -&gt; possibly safety issue on driver remove due to bad resource account</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>possible runtime failiure in btrfs under user conditions</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>packaging bug so doesn't effect runtime</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>specific scsi driver will fail to resume</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>reloading intel network driver may fail due to previous incorrect error free</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>? issue with superblock handling on selinux -&gt; is it an real issue?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>call fails if device does not support info request, should be caught by testing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>?</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>riscv kexec issue -&gt; not sure if safety issue? could be transient failure in restart?</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Name Did this fix a bug? Safety relevant bug? Timing + Execution Memory  \\\n",
       "0   Ben D.                 yes                   no                 no     no   \n",
       "1   Ben D.                 yes                    ?                 no    yes   \n",
       "2   Ben D.                 yes                   no                 no     no   \n",
       "3   Ben D.                  no                   no                 no     no   \n",
       "4   Ben D.                 yes                   no                 no     no   \n",
       "5   Ben D.                 yes                  yes                 no     no   \n",
       "6   Ben D.                 yes                   no                 no     no   \n",
       "7   Ben D.                 yes                   no                 no     no   \n",
       "8   Ben D.                  no                   no                 no     no   \n",
       "9   Ben D.                 yes                   no                 no     no   \n",
       "10  Ben D.                 yes                  yes                 no    yes   \n",
       "11  Ben D.                  no                   no                 no    yes   \n",
       "12  Ben D.                 yes                  yes                 no    yes   \n",
       "13  Ben D.                 yes                   no                 no     no   \n",
       "14  Ben D.                 yes                  yes                 no    yes   \n",
       "15  Ben D.                 yes                   no                 no    yes   \n",
       "16  Ben D.                 yes                  yes                 no     no   \n",
       "17  Ben D.                 yes                    ?                 no     no   \n",
       "18  Ben D.                 yes                  yes                 no     no   \n",
       "19  Ben D.                 yes                    ?                 no     no   \n",
       "20  Ben D.                   ?                    ?                 no     no   \n",
       "21  Ben D.                 yes                  yes                 no     no   \n",
       "22  Ben D.                 yes                  yes                 no     no   \n",
       "23  Ben D.                 yes                   no                 no     no   \n",
       "24  Ben D.                   ?                    ?                 no     no   \n",
       "25  Ben D.                 yes                    ?                 no     no   \n",
       "26  Ben D.                 yes                   no                 no     no   \n",
       "27  Ben D.                 yes                   no                 no     no   \n",
       "28  Ben D.                 yes                   no                 no     no   \n",
       "29  Ben D.                 yes                    ?                 no     no   \n",
       "30  Ben D.                 yes                   no                 no     no   \n",
       "31  Ben D.                 yes                  yes                 no    yes   \n",
       "33  Ben D.                  no                   no                 no      ?   \n",
       "34  Ben D.                 yes                    ?                 no     no   \n",
       "36  Ben D.                  no                   no                yes     no   \n",
       "37  Ben D.                 yes                  yes                nno    yes   \n",
       "38  Ben D.                 yes                    ?                 no    yes   \n",
       "39  Ben D.                 yes                  yes                 no    yes   \n",
       "40  Ben D.                 yes                    ?                 no    yes   \n",
       "41  Ben D.                 yes                  yes                 no    yes   \n",
       "42  Ben D.                 yes                   no                 no     no   \n",
       "43  Ben D.                 yes                  yes                yes     no   \n",
       "44  Ben D.                 NaN                  NaN                 no     no   \n",
       "45  Ben D.                 yes                  yes                 no    yes   \n",
       "46  Ben D.                 NaN                  NaN                 no     no   \n",
       "48  Ben D.                 yes                   no                 no    yes   \n",
       "49  Ben D.                 yes                    ?                 no    yes   \n",
       "\n",
       "   Exchange of Info  \\\n",
       "0                no   \n",
       "1                no   \n",
       "2                 ?   \n",
       "3                no   \n",
       "4                no   \n",
       "5                no   \n",
       "6                no   \n",
       "7                no   \n",
       "8                no   \n",
       "9                no   \n",
       "10               no   \n",
       "11               no   \n",
       "12               no   \n",
       "13               no   \n",
       "14               no   \n",
       "15               no   \n",
       "16               no   \n",
       "17               no   \n",
       "18               no   \n",
       "19               no   \n",
       "20               no   \n",
       "21               no   \n",
       "22               no   \n",
       "23               no   \n",
       "24               no   \n",
       "25               no   \n",
       "26               no   \n",
       "27               no   \n",
       "28               no   \n",
       "29               no   \n",
       "30               no   \n",
       "31               no   \n",
       "33               no   \n",
       "34              yes   \n",
       "36               no   \n",
       "37               no   \n",
       "38               no   \n",
       "39               no   \n",
       "40               no   \n",
       "41               no   \n",
       "42               no   \n",
       "43               no   \n",
       "44               no   \n",
       "45               no   \n",
       "46               no   \n",
       "48               no   \n",
       "49               no   \n",
       "\n",
       "                                                                                 Reasoning  \n",
       "0                                                    locking and reference fix in dm layer  \n",
       "1                                                      scsi driver fix for lock after free  \n",
       "2                                                         fixes specific scsi scanning bug  \n",
       "3                                          removed unnecessary work already done elsewhere  \n",
       "4                                                     removes warning due to linker output  \n",
       "5                                               fixes gfs2 locking bug over suspend/resume  \n",
       "6                                                     build configuration dependency issue  \n",
       "7                               reliable boot failure if pagetable size is set to 5 levels  \n",
       "8                                                        kernel page allocation api change  \n",
       "9                                                     build configuration dependency issue  \n",
       "10                                          bad memory de-reference may destabilise system  \n",
       "11                                                       kernel page allocation api change  \n",
       "12                                                     possible lock usage on freed memory  \n",
       "13                                                 bug is hardware assignment at init time  \n",
       "14                                   possible sleep in invalid place may cause system hang  \n",
       "15                                          jbd2 may fail to release memory in error paths  \n",
       "16                                                  resume may send invalid hotplug events  \n",
       "17                                                           ext4 fstrim may block suspend  \n",
       "18                                                       fixes deadlock in drm/vkms driver  \n",
       "19                                           nfs requests may be lost - maybe get retried?  \n",
       "20                                                               not enough info in commit  \n",
       "21                                                                incorrect locking in nfs  \n",
       "22                                           possible read of invalid memory / BUG trigger  \n",
       "23                                 caller should detect invalid response and retry metrics  \n",
       "24                                                                            No reasoning  \n",
       "25                                                                              nfs bugfix  \n",
       "26                             nvme init timeout fix - will always timeout with bad device  \n",
       "27                                                                          kunit test bug  \n",
       "28                                         backlight bug on certain amd hw during shutdown  \n",
       "29  use after free during driver removal - probably not fatal to the runtime of the system  \n",
       "30                                                            accounting bug on error path  \n",
       "31                                            could cause errors in the network data quees  \n",
       "33                 nfs issue where data will be retransmitted - todo check if memory leak?  \n",
       "34                                     network socket lookup with ipv4/ipv6 interaction -   \n",
       "36                                     removes extra unused branch protection instructions  \n",
       "37                                                 fixes OOPS if procfs file gets accessed  \n",
       "38                                fix for amd gpu page-table code for large memory systems  \n",
       "39                                                   nfs oops can be generated at run time  \n",
       "40  extra fwnode put -> possibly safety issue on driver remove due to bad resource account  \n",
       "41                                possible runtime failiure in btrfs under user conditions  \n",
       "42                                                 packaging bug so doesn't effect runtime  \n",
       "43                                                specific scsi driver will fail to resume  \n",
       "44                                                                                       ?  \n",
       "45            reloading intel network driver may fail due to previous incorrect error free  \n",
       "46                     ? issue with superblock handling on selinux -> is it an real issue?  \n",
       "48         call fails if device does not support info request, should be caught by testing  \n",
       "49   riscv kexec issue -> not sure if safety issue? could be transient failure in restart?  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "codethink_results_ben_dooks_filled = codethink_results_ben_dooks[codethink_results_round_1['Name'] != \"None\"]\n",
    "print(\"Ben Dooks- results count: \",len(codethink_results_ben_dooks_filled))\n",
    "(\n",
    "    codethink_results_ben_dooks_filled\n",
    "    [[\"Name\",\"Did this fix a bug?\",\"Safety relevant bug?\",\"Timing + Execution\",\"Memory\",\"Exchange of Info\",\"Reasoning\"]]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41dbd8dd-b6f2-4fc1-8643-ff24be996f68",
   "metadata": {},
   "source": [
    "## 3. Comparing Codethink round's results with Ben Dooks results\n",
    "\n",
    "**Note**: When comparing Codethink and Ben's results, we have left out those commits for which there has been conflict between Codethink developers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "2edf7e6e-d448-43e4-8019-ab04748c756c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "</style>\n",
       "<table id=\"T_f5593\">\n",
       "  <caption><b style='float: right;'>Codethink R1+R2:</b></caption>\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_f5593_level0_col0\" class=\"col_heading level0 col0\" >YES</th>\n",
       "      <th id=\"T_f5593_level0_col1\" class=\"col_heading level0 col1\" >NO</th>\n",
       "      <th id=\"T_f5593_level0_col2\" class=\"col_heading level0 col2\" >?</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th class=\"index_name level0\" >Ben Dooks:</th>\n",
       "      <th class=\"blank col0\" >&nbsp;</th>\n",
       "      <th class=\"blank col1\" >&nbsp;</th>\n",
       "      <th class=\"blank col2\" >&nbsp;</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_f5593_level0_row0\" class=\"row_heading level0 row0\" >YES</th>\n",
       "      <td id=\"T_f5593_row0_col0\" class=\"data row0 col0\" >35</td>\n",
       "      <td id=\"T_f5593_row0_col1\" class=\"data row0 col1\" >2</td>\n",
       "      <td id=\"T_f5593_row0_col2\" class=\"data row0 col2\" >0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_f5593_level0_row1\" class=\"row_heading level0 row1\" >NO</th>\n",
       "      <td id=\"T_f5593_row1_col0\" class=\"data row1 col0\" >4</td>\n",
       "      <td id=\"T_f5593_row1_col1\" class=\"data row1 col1\" >1</td>\n",
       "      <td id=\"T_f5593_row1_col2\" class=\"data row1 col2\" >0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_f5593_level0_row2\" class=\"row_heading level0 row2\" >?</th>\n",
       "      <td id=\"T_f5593_row2_col0\" class=\"data row2 col0\" >2</td>\n",
       "      <td id=\"T_f5593_row2_col1\" class=\"data row2 col1\" >0</td>\n",
       "      <td id=\"T_f5593_row2_col2\" class=\"data row2 col2\" >0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7f3345692190>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "getComparativeMatrix(\"Did this fix a bug?\",combined_codethink_results_df, codethink_results_ben_dooks, \"Codethink R1+R2\", \"Ben Dooks\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "f5d7ab80-cbfe-4669-930a-ff210f9ae7a7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "</style>\n",
       "<table id=\"T_67249\">\n",
       "  <caption><b style='float: right;'>Codethink R1+R2:</b></caption>\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_67249_level0_col0\" class=\"col_heading level0 col0\" >YES</th>\n",
       "      <th id=\"T_67249_level0_col1\" class=\"col_heading level0 col1\" >NO</th>\n",
       "      <th id=\"T_67249_level0_col2\" class=\"col_heading level0 col2\" >?</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th class=\"index_name level0\" >Ben Dooks:</th>\n",
       "      <th class=\"blank col0\" >&nbsp;</th>\n",
       "      <th class=\"blank col1\" >&nbsp;</th>\n",
       "      <th class=\"blank col2\" >&nbsp;</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_67249_level0_row0\" class=\"row_heading level0 row0\" >YES</th>\n",
       "      <td id=\"T_67249_row0_col0\" class=\"data row0 col0\" >9</td>\n",
       "      <td id=\"T_67249_row0_col1\" class=\"data row0 col1\" >1</td>\n",
       "      <td id=\"T_67249_row0_col2\" class=\"data row0 col2\" >0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_67249_level0_row1\" class=\"row_heading level0 row1\" >NO</th>\n",
       "      <td id=\"T_67249_row1_col0\" class=\"data row1 col0\" >4</td>\n",
       "      <td id=\"T_67249_row1_col1\" class=\"data row1 col1\" >7</td>\n",
       "      <td id=\"T_67249_row1_col2\" class=\"data row1 col2\" >2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_67249_level0_row2\" class=\"row_heading level0 row2\" >?</th>\n",
       "      <td id=\"T_67249_row2_col0\" class=\"data row2 col0\" >4</td>\n",
       "      <td id=\"T_67249_row2_col1\" class=\"data row2 col1\" >5</td>\n",
       "      <td id=\"T_67249_row2_col2\" class=\"data row2 col2\" >0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7f3345693010>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "getComparativeMatrix(\"Safety relevant bug?\",combined_codethink_results_df, codethink_results_ben_dooks, \"Codethink R1+R2\", \"Ben Dooks\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "7237fe72-522a-457b-945f-1c087364317b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Timing + Execution'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "</style>\n",
       "<table id=\"T_c1221\">\n",
       "  <caption><b style='float: right;'>Codethink R1+R2:</b></caption>\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_c1221_level0_col0\" class=\"col_heading level0 col0\" >YES</th>\n",
       "      <th id=\"T_c1221_level0_col1\" class=\"col_heading level0 col1\" >NO</th>\n",
       "      <th id=\"T_c1221_level0_col2\" class=\"col_heading level0 col2\" >?</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th class=\"index_name level0\" >Ben Dooks:</th>\n",
       "      <th class=\"blank col0\" >&nbsp;</th>\n",
       "      <th class=\"blank col1\" >&nbsp;</th>\n",
       "      <th class=\"blank col2\" >&nbsp;</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_c1221_level0_row0\" class=\"row_heading level0 row0\" >YES</th>\n",
       "      <td id=\"T_c1221_row0_col0\" class=\"data row0 col0\" >1</td>\n",
       "      <td id=\"T_c1221_row0_col1\" class=\"data row0 col1\" >0</td>\n",
       "      <td id=\"T_c1221_row0_col2\" class=\"data row0 col2\" >0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_c1221_level0_row1\" class=\"row_heading level0 row1\" >NO</th>\n",
       "      <td id=\"T_c1221_row1_col0\" class=\"data row1 col0\" >7</td>\n",
       "      <td id=\"T_c1221_row1_col1\" class=\"data row1 col1\" >13</td>\n",
       "      <td id=\"T_c1221_row1_col2\" class=\"data row1 col2\" >0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_c1221_level0_row2\" class=\"row_heading level0 row2\" >?</th>\n",
       "      <td id=\"T_c1221_row2_col0\" class=\"data row2 col0\" >0</td>\n",
       "      <td id=\"T_c1221_row2_col1\" class=\"data row2 col1\" >0</td>\n",
       "      <td id=\"T_c1221_row2_col2\" class=\"data row2 col2\" >0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7f33456918d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'Memory'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "</style>\n",
       "<table id=\"T_a5491\">\n",
       "  <caption><b style='float: right;'>Codethink R1+R2:</b></caption>\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_a5491_level0_col0\" class=\"col_heading level0 col0\" >YES</th>\n",
       "      <th id=\"T_a5491_level0_col1\" class=\"col_heading level0 col1\" >NO</th>\n",
       "      <th id=\"T_a5491_level0_col2\" class=\"col_heading level0 col2\" >?</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th class=\"index_name level0\" >Ben Dooks:</th>\n",
       "      <th class=\"blank col0\" >&nbsp;</th>\n",
       "      <th class=\"blank col1\" >&nbsp;</th>\n",
       "      <th class=\"blank col2\" >&nbsp;</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_a5491_level0_row0\" class=\"row_heading level0 row0\" >YES</th>\n",
       "      <td id=\"T_a5491_row0_col0\" class=\"data row0 col0\" >4</td>\n",
       "      <td id=\"T_a5491_row0_col1\" class=\"data row0 col1\" >2</td>\n",
       "      <td id=\"T_a5491_row0_col2\" class=\"data row0 col2\" >0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_a5491_level0_row1\" class=\"row_heading level0 row1\" >NO</th>\n",
       "      <td id=\"T_a5491_row1_col0\" class=\"data row1 col0\" >2</td>\n",
       "      <td id=\"T_a5491_row1_col1\" class=\"data row1 col1\" >12</td>\n",
       "      <td id=\"T_a5491_row1_col2\" class=\"data row1 col2\" >0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_a5491_level0_row2\" class=\"row_heading level0 row2\" >?</th>\n",
       "      <td id=\"T_a5491_row2_col0\" class=\"data row2 col0\" >0</td>\n",
       "      <td id=\"T_a5491_row2_col1\" class=\"data row2 col1\" >1</td>\n",
       "      <td id=\"T_a5491_row2_col2\" class=\"data row2 col2\" >0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7f3345691010>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'Exchange of Info'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "</style>\n",
       "<table id=\"T_9e54c\">\n",
       "  <caption><b style='float: right;'>Codethink R1+R2:</b></caption>\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_9e54c_level0_col0\" class=\"col_heading level0 col0\" >YES</th>\n",
       "      <th id=\"T_9e54c_level0_col1\" class=\"col_heading level0 col1\" >NO</th>\n",
       "      <th id=\"T_9e54c_level0_col2\" class=\"col_heading level0 col2\" >?</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th class=\"index_name level0\" >Ben Dooks:</th>\n",
       "      <th class=\"blank col0\" >&nbsp;</th>\n",
       "      <th class=\"blank col1\" >&nbsp;</th>\n",
       "      <th class=\"blank col2\" >&nbsp;</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_9e54c_level0_row0\" class=\"row_heading level0 row0\" >YES</th>\n",
       "      <td id=\"T_9e54c_row0_col0\" class=\"data row0 col0\" >0</td>\n",
       "      <td id=\"T_9e54c_row0_col1\" class=\"data row0 col1\" >0</td>\n",
       "      <td id=\"T_9e54c_row0_col2\" class=\"data row0 col2\" >0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_9e54c_level0_row1\" class=\"row_heading level0 row1\" >NO</th>\n",
       "      <td id=\"T_9e54c_row1_col0\" class=\"data row1 col0\" >5</td>\n",
       "      <td id=\"T_9e54c_row1_col1\" class=\"data row1 col1\" >15</td>\n",
       "      <td id=\"T_9e54c_row1_col2\" class=\"data row1 col2\" >1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_9e54c_level0_row2\" class=\"row_heading level0 row2\" >?</th>\n",
       "      <td id=\"T_9e54c_row2_col0\" class=\"data row2 col0\" >0</td>\n",
       "      <td id=\"T_9e54c_row2_col1\" class=\"data row2 col1\" >0</td>\n",
       "      <td id=\"T_9e54c_row2_col2\" class=\"data row2 col2\" >0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7f334587bb50>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "for column in ['Timing + Execution', 'Memory', 'Exchange of Info']:\n",
    "    display(column)\n",
    "    getComparativeMatrix(column,combined_codethink_results_df, codethink_results_ben_dooks, \"Codethink R1+R2\", \"Ben Dooks\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "de52f832-9645-446b-9d85-fe9b6e5281e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "combined_codethink_and_ben_results_df = combineResults(combined_codethink_results_df, codethink_results_ben_dooks)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8354cd99-e9c8-41a2-bff8-3d697cf41679",
   "metadata": {},
   "source": [
    "### 3.1 Conflicts in bug-fixing commit identification (Codethink R1+R2 vs Ben Dooks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "961e5d1f-9210-48ee-9acf-20ceff2d1949",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>Did this fix a bug?</th>\n",
       "      <th>Reasoning</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Tom Bloor/Ben D.</td>\n",
       "      <td>Conflict: yes/no</td>\n",
       "      <td>#Tom Bloor# wouldnt compile without this fix<br>#Ben D.# kernel page allocation api change</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>William Salmon/Ben D.</td>\n",
       "      <td>Conflict: yes/no</td>\n",
       "      <td>#William Salmon# Fixes bug improving filesystem performace, could effect timing but would not be safety critical in a first order sence<br>#Ben D.# kernel page allocation api change</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>John G/Ben D.</td>\n",
       "      <td>Conflict: no/yes</td>\n",
       "      <td>#John G# Makes different things (\"DPIA\"?) preferred, sounds like enhancement rather than fx<br>#Ben D.# bug is hardware assignment at init time</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>Anton Pyntikov/Paul Sherwood/Ben D.</td>\n",
       "      <td>Conflict: yes/?</td>\n",
       "      <td>#Anton Pyntikov/Paul Sherwood# #Anton Pyntikov# Fix memory coherency<br>#Paul Sherwood# the commit message doesn't give enough info to understand this<br>#Ben D.# not enough info in commit</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>Sam L/Ben D.</td>\n",
       "      <td>Conflict: yes/?</td>\n",
       "      <td>#Sam L# I think HDP registers are only related to caching so not safety related<br>#Ben D.# No reasoning</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>Paul Sherwood/Ben D.</td>\n",
       "      <td>Conflict: no/yes</td>\n",
       "      <td>#Paul Sherwood# i think this is just a slight performance improvement<br>#Ben D.# kunit test bug</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>Nathan/Ben D.</td>\n",
       "      <td>Conflict: yes/no</td>\n",
       "      <td>#Nathan# Reverts a commit that had a false assumption that did an error out instead of a garbage out. This doesn't effect the outcome both still count as a fail.<br>#Ben D.# nfs issue where data will be retransmitted - todo check if memory leak?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>Ivan Orlov/Ben D.</td>\n",
       "      <td>Conflict: yes/no</td>\n",
       "      <td>#Ivan Orlov# one instruction compiles as another<br>#Ben D.# removes extra unused branch protection instructions</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "combined_codethink_and_ben_results_df_filtered = combined_codethink_and_ben_results_df[combined_codethink_and_ben_results_df['Did this fix a bug?'].notnull()]\n",
    "showResults(\n",
    "    combined_codethink_and_ben_results_df_filtered\n",
    "    [combined_codethink_and_ben_results_df_filtered['Did this fix a bug?'].str.contains('Conflict: Conflict:')==False]\n",
    "    [combined_codethink_and_ben_results_df_filtered['Did this fix a bug?'].str.contains('Conflict')]\n",
    "    [['Name','Did this fix a bug?','Reasoning']]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30da85c6-fac8-4067-b046-cb603a1a5831",
   "metadata": {},
   "source": [
    "### 3.2 Conflicts in safety bugs identification (Codethink R1+R2 vs Ben Dooks)\n",
    "\n",
    "**Note**: At this point, we only consider those commits for which the same result has been obtained by classifying \"Did this fix a bug?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "fdcbb59e-3e33-4451-9d37-8292a5d607d0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>Did this fix a bug?</th>\n",
       "      <th>Safety relevant bug?</th>\n",
       "      <th>Reasoning</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Adnan Chowdhury/Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>Conflict: yes/no</td>\n",
       "      <td>#Adnan Chowdhury# Fix of a race condition which could possibly corrupt memory<br>#Ben D.# locking and reference fix in dm layer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>John G/Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>Conflict: yes/?</td>\n",
       "      <td>#John G# \"could [stuff...]... causing a use-after-free memory access\"<br>#Ben D.# scsi driver fix for lock after free</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Hao Hu/Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>Conflict: yes/no</td>\n",
       "      <td>#Hao Hu# Change SAS interface pointer<br>#Ben D.# fixes specific scsi scanning bug</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Tom Bloor/Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>Conflict: ?/no</td>\n",
       "      <td>#Tom Bloor# if youre rebooting during normal operation then... good luck?<br>#Ben D.# reliable boot failure if pagetable size is set to 5 levels</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>John G/Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>Conflict: yes/no</td>\n",
       "      <td>#John G# Commit message said so, diff includes a free*()ing function<br>#Ben D.# jbd2 may fail to release memory in error paths</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Ivan Orlov/Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>Conflict: no/?</td>\n",
       "      <td>#Ivan Orlov# Commit message + code which stops FS trimming when suspend<br>#Ben D.# ext4 fstrim may block suspend</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Basit A./Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>Conflict: no/yes</td>\n",
       "      <td>#Basit A.# commit message + interrupt service routines using mutexes <br>#Ben D.# fixes deadlock in drm/vkms driver</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Sam L/Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>Conflict: no/?</td>\n",
       "      <td>#Sam L# Previous code doesn't reschedule packet, packets are now rescheduled properly I think??<br>#Ben D.# nfs requests may be lost - maybe get retried?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Poppy/Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>Conflict: ?/no</td>\n",
       "      <td>#Poppy# Metrics table filler fails correctly.<br>#Ben D.# caller should detect invalid response and retry metrics</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>Paul Sherwood/Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>Conflict: no/?</td>\n",
       "      <td>#Paul Sherwood# No reasoning<br>#Ben D.# nfs bugfix</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>Basit Ayantunde/Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>Conflict: yes/no</td>\n",
       "      <td>#Basit Ayantunde# commit message + underreserved memory<br>#Ben D.# nvme init timeout fix - will always timeout with bad device</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>Tom Bloor/Gavin McCall/Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>Conflict: yes/?</td>\n",
       "      <td>#Tom Bloor/Gavin McCall# #Tom Bloor# Race condition leading to incorrect reference counting and use after free. also strawberries<br>#Gavin McCall# use after free<br>#Ben D.# use after free during driver removal - probably not fatal to the runtime of the system</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>John G/Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>Conflict: yes/?</td>\n",
       "      <td>#John G# Fixes issue in network address handling<br>#Ben D.# network socket lookup with ipv4/ipv6 interaction -</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>John G/Anton Pyntikov/Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>Conflict: yes/?</td>\n",
       "      <td>#John G/Anton Pyntikov# #John G# \"it makes page table base points to wrong address if...\"<br>#Anton Pyntikov# Memory corruption if phys addr beyond 64Gb<br>#Ben D.# fix for amd gpu page-table code for large memory systems</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>Ivan Orlov/Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>Conflict: no/?</td>\n",
       "      <td>#Ivan Orlov# fwnode is released somewhere else, so possible double free<br>#Ben D.# extra fwnode put -&gt; possibly safety issue on driver remove due to bad resource account</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>Richard Dale/Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>Conflict: no/?</td>\n",
       "      <td>#Richard Dale# Fixes RISC-V memory alignment issue at startup<br>#Ben D.# riscv kexec issue -&gt; not sure if safety issue? could be transient failure in restart?</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "combined_codethink_and_ben_results_df_filtered = combined_codethink_and_ben_results_df[combined_codethink_and_ben_results_df['Safety relevant bug?'].notnull()]\n",
    "showResults(\n",
    "    combined_codethink_and_ben_results_df_filtered\n",
    "    [combined_codethink_and_ben_results_df_filtered['Did this fix a bug?'].str.contains('Conflict:')==False]\n",
    "    [combined_codethink_and_ben_results_df_filtered['Safety relevant bug?'].str.contains('Conflict: Conflict:')==False]\n",
    "    [combined_codethink_and_ben_results_df_filtered['Safety relevant bug?'].str.contains('Conflict')]\n",
    "    [['Name','Did this fix a bug?','Safety relevant bug?','Reasoning']]\n",
    ")\n",
    "# Ben Dooks Commit 40"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "825c7647-5046-491d-bd22-c6c73a8a7c7c",
   "metadata": {},
   "source": [
    "### 3.3 Conflicts in safety classification (Codethink R1+R2 vs Ben Dooks)\n",
    "\n",
    "**Note**: At this point, we only consider those commits for which the same result has been obtained by classifying \"Safety relevant bug?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "04ec4237-cf79-4745-bbb7-092b751878c0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Timing + Execution'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>Did this fix a bug?</th>\n",
       "      <th>Safety relevant bug?</th>\n",
       "      <th>Timing + Execution</th>\n",
       "      <th>Reasoning</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Hao Hu/Ben D.</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>Conflict: yes/no</td>\n",
       "      <td>#Hao Hu# Remove extra dirty node flag<br>#Ben D.# removed unnecessary work already done elsewhere</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Aidan MacDonald/Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>Conflict: yes/no</td>\n",
       "      <td>#Aidan MacDonald# cluster filesystem locking error causing hang - according to commit message, difficult to verify otherwise<br>#Ben D.# fixes gfs2 locking bug over suspend/resume</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Weyman/Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>Conflict: yes/no</td>\n",
       "      <td>#Weyman# Involves an event, a lock, and access of released resource (a crash)<br>#Ben D.# possible lock usage on freed memory</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Paul Sherwood/Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>Conflict: yes/no</td>\n",
       "      <td>#Paul Sherwood# bug related to timing in io function, so scheduling + info exchange?<br>#Ben D.# possible sleep in invalid place may cause system hang</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>Sam L/Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>Conflict: yes/no</td>\n",
       "      <td>#Sam L# Previous code locked the wrong lock...<br>#Ben D.# incorrect locking in nfs</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>Gavin McCall/Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>Conflict: yes/no</td>\n",
       "      <td>#Gavin McCall# \"not valid\"<br>#Ben D.# backlight bug on certain amd hw during shutdown</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>JZ/Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>Conflict: yes/no</td>\n",
       "      <td>#JZ# race condition between adding and reading data, resulting in trying to add two entries with the same index<br>#Ben D.# possible runtime failiure in btrfs under user conditions</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'Memory'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>Did this fix a bug?</th>\n",
       "      <th>Safety relevant bug?</th>\n",
       "      <th>Memory</th>\n",
       "      <th>Reasoning</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Weyman/Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>Conflict: no/yes</td>\n",
       "      <td>#Weyman# Involves an event, a lock, and access of released resource (a crash)<br>#Ben D.# possible lock usage on freed memory</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Paul Sherwood/Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>Conflict: no/yes</td>\n",
       "      <td>#Paul Sherwood# bug related to timing in io function, so scheduling + info exchange?<br>#Ben D.# possible sleep in invalid place may cause system hang</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>Sam L/Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>Conflict: yes/no</td>\n",
       "      <td>#Sam L# Previous code locked the wrong lock...<br>#Ben D.# incorrect locking in nfs</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>Basit A./Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>Conflict: yes/no</td>\n",
       "      <td>#Basit A.# commit message + use after free<br>#Ben D.# possible read of invalid memory / BUG trigger</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'Exchange of Info'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>Did this fix a bug?</th>\n",
       "      <th>Safety relevant bug?</th>\n",
       "      <th>Exchange of Info</th>\n",
       "      <th>Reasoning</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Aidan MacDonald/Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>Conflict: ?/no</td>\n",
       "      <td>#Aidan MacDonald# cluster filesystem locking error causing hang - according to commit message, difficult to verify otherwise<br>#Ben D.# fixes gfs2 locking bug over suspend/resume</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Paul Sherwood/Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>Conflict: yes/no</td>\n",
       "      <td>#Paul Sherwood# bug related to timing in io function, so scheduling + info exchange?<br>#Ben D.# possible sleep in invalid place may cause system hang</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>Basit A./Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>Conflict: yes/no</td>\n",
       "      <td>#Basit A.# commit message + use after free<br>#Ben D.# possible read of invalid memory / BUG trigger</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>Dickon Hood/Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>Conflict: yes/no</td>\n",
       "      <td>#Dickon Hood# Socket handling error under error conditions<br>#Ben D.# could cause errors in the network data quees</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>JZ/Ben D.</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>Conflict: yes/no</td>\n",
       "      <td>#JZ# race condition between adding and reading data, resulting in trying to add two entries with the same index<br>#Ben D.# possible runtime failiure in btrfs under user conditions</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "for column in ['Timing + Execution', 'Memory', 'Exchange of Info']:\n",
    "    display(column)\n",
    "    combined_codethink_and_ben_results_df_filtered = combined_codethink_and_ben_results_df[combined_codethink_and_ben_results_df[column].notnull()]\n",
    "    showResults(\n",
    "        combined_codethink_and_ben_results_df_filtered\n",
    "        [combined_codethink_and_ben_results_df_filtered['Did this fix a bug?'].str.contains('Conflict:')==False]\n",
    "        [combined_codethink_and_ben_results_df_filtered['Safety relevant bug?'].str.contains('Conflict:')==False]\n",
    "        [combined_codethink_and_ben_results_df_filtered[column].str.contains('Conflict')]\n",
    "        [['Name','Did this fix a bug?','Safety relevant bug?',column,'Reasoning']]\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7bb15c0f-ad0e-41c0-b184-c153ed7efec5",
   "metadata": {},
   "source": [
    "## 4. Comparing with our results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "4256da6d-b8e5-4702-aee7-d8493496fded",
   "metadata": {},
   "outputs": [],
   "source": [
    "michel_results = pd.read_csv('Linux Bug Classification - Michel Maes.csv')  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6fee2b63-8db0-413e-a586-b989f2febedd",
   "metadata": {},
   "source": [
    "### 4.1 Comparing Michel Maes results with Codethink R1+R2 results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "1abc0a29-0a07-476c-a4b2-d8640af59863",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "</style>\n",
       "<table id=\"T_86193\">\n",
       "  <caption><b style='float: right;'>Michel Maes:</b></caption>\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_86193_level0_col0\" class=\"col_heading level0 col0\" >YES</th>\n",
       "      <th id=\"T_86193_level0_col1\" class=\"col_heading level0 col1\" >NO</th>\n",
       "      <th id=\"T_86193_level0_col2\" class=\"col_heading level0 col2\" >?</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th class=\"index_name level0\" >Codethink R1+R2:</th>\n",
       "      <th class=\"blank col0\" >&nbsp;</th>\n",
       "      <th class=\"blank col1\" >&nbsp;</th>\n",
       "      <th class=\"blank col2\" >&nbsp;</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_86193_level0_row0\" class=\"row_heading level0 row0\" >YES</th>\n",
       "      <td id=\"T_86193_row0_col0\" class=\"data row0 col0\" >37</td>\n",
       "      <td id=\"T_86193_row0_col1\" class=\"data row0 col1\" >3</td>\n",
       "      <td id=\"T_86193_row0_col2\" class=\"data row0 col2\" >3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_86193_level0_row1\" class=\"row_heading level0 row1\" >NO</th>\n",
       "      <td id=\"T_86193_row1_col0\" class=\"data row1 col0\" >2</td>\n",
       "      <td id=\"T_86193_row1_col1\" class=\"data row1 col1\" >0</td>\n",
       "      <td id=\"T_86193_row1_col2\" class=\"data row1 col2\" >1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_86193_level0_row2\" class=\"row_heading level0 row2\" >?</th>\n",
       "      <td id=\"T_86193_row2_col0\" class=\"data row2 col0\" >0</td>\n",
       "      <td id=\"T_86193_row2_col1\" class=\"data row2 col1\" >0</td>\n",
       "      <td id=\"T_86193_row2_col2\" class=\"data row2 col2\" >0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7f33458e6390>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "getComparativeMatrix(\"Did this fix a bug?\",michel_results, combined_codethink_results_df, \"Michel Maes\", \"Codethink R1+R2\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "3baee1bc-ae4f-4bde-bb24-a9f82241eaca",
   "metadata": {},
   "outputs": [],
   "source": [
    "combined_codethink_and_michel_results_df = combineResults(combined_codethink_results_df, michel_results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "c4cff7c8-c5f2-43c0-b66f-648d532518db",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>Did this fix a bug?</th>\n",
       "      <th>Reasoning</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Hao Hu/Michel M.</td>\n",
       "      <td>Conflict: no/?</td>\n",
       "      <td>#Hao Hu# Remove extra dirty node flag<br>#Michel M.# not sure</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Basit A./Michel M.</td>\n",
       "      <td>Conflict: yes/no</td>\n",
       "      <td>#Basit A.# commit + superfluos function section data generation<br>#Michel M.# change on Makefile - no code change</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Chris Phang/Michel M.</td>\n",
       "      <td>Conflict: yes/no</td>\n",
       "      <td>#Chris Phang# kbuild dependency fix<br>#Michel M.# change on Kconfig - no code change</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Aidan MacDonald/Michel M.</td>\n",
       "      <td>Conflict: yes/no</td>\n",
       "      <td>#Aidan MacDonald# Kbuild fix<br>#Michel M.# change on Kconfig - no code change</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>William Salmon/Michel M.</td>\n",
       "      <td>Conflict: yes/?</td>\n",
       "      <td>#William Salmon# Fixes bug improving filesystem performace, could effect timing but would not be safety critical in a first order sence<br>#Michel M.# looks like a refactor - rename variable page</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>John G/Michel M.</td>\n",
       "      <td>Conflict: no/yes</td>\n",
       "      <td>#John G# Makes different things (\"DPIA\"?) preferred, sounds like enhancement rather than fx<br>#Michel M.# commit message indcates that is a fix</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>Sam L/Michel M.</td>\n",
       "      <td>Conflict: yes/?</td>\n",
       "      <td>#Sam L# I think HDP registers are only related to caching so not safety related<br>#Michel M.# not sure</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>Paul Sherwood/Michel M.</td>\n",
       "      <td>Conflict: no/yes</td>\n",
       "      <td>#Paul Sherwood# i think this is just a slight performance improvement<br>#Michel M.# commit msg indicates that this change fixes a bug introduced in another commit</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>Gavin McCall/Michel M.</td>\n",
       "      <td>Conflict: yes/?</td>\n",
       "      <td>#Gavin McCall# \"not valid\"<br>#Michel M.# not sure</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "combined_codethink_and_michel_results_df_filtered = combined_codethink_and_michel_results_df[combined_codethink_and_michel_results_df['Did this fix a bug?'].notnull()]\n",
    "showResults(\n",
    "    combined_codethink_and_michel_results_df_filtered\n",
    "    [combined_codethink_and_michel_results_df_filtered['Did this fix a bug?'].str.contains('Conflict: Conflict:')==False]\n",
    "    [combined_codethink_and_michel_results_df_filtered['Did this fix a bug?'].str.contains('Conflict')]\n",
    "    [['Name','Did this fix a bug?','Reasoning']]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a4883a4-f612-45e5-8359-4dae09c99c24",
   "metadata": {},
   "source": [
    "### 4.2 Comparing Michel Maes results with Ben Dooks results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "fc20b989-aa2e-4759-9206-b2330eecd7c4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "</style>\n",
       "<table id=\"T_45aa5\">\n",
       "  <caption><b style='float: right;'>Michel Maes:</b></caption>\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_45aa5_level0_col0\" class=\"col_heading level0 col0\" >YES</th>\n",
       "      <th id=\"T_45aa5_level0_col1\" class=\"col_heading level0 col1\" >NO</th>\n",
       "      <th id=\"T_45aa5_level0_col2\" class=\"col_heading level0 col2\" >?</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th class=\"index_name level0\" >Ben Dooks:</th>\n",
       "      <th class=\"blank col0\" >&nbsp;</th>\n",
       "      <th class=\"blank col1\" >&nbsp;</th>\n",
       "      <th class=\"blank col2\" >&nbsp;</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_45aa5_level0_row0\" class=\"row_heading level0 row0\" >YES</th>\n",
       "      <td id=\"T_45aa5_row0_col0\" class=\"data row0 col0\" >36</td>\n",
       "      <td id=\"T_45aa5_row0_col1\" class=\"data row0 col1\" >3</td>\n",
       "      <td id=\"T_45aa5_row0_col2\" class=\"data row0 col2\" >1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_45aa5_level0_row1\" class=\"row_heading level0 row1\" >NO</th>\n",
       "      <td id=\"T_45aa5_row1_col0\" class=\"data row1 col0\" >3</td>\n",
       "      <td id=\"T_45aa5_row1_col1\" class=\"data row1 col1\" >0</td>\n",
       "      <td id=\"T_45aa5_row1_col2\" class=\"data row1 col2\" >2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_45aa5_level0_row2\" class=\"row_heading level0 row2\" >?</th>\n",
       "      <td id=\"T_45aa5_row2_col0\" class=\"data row2 col0\" >1</td>\n",
       "      <td id=\"T_45aa5_row2_col1\" class=\"data row2 col1\" >0</td>\n",
       "      <td id=\"T_45aa5_row2_col2\" class=\"data row2 col2\" >1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7f33456bfe10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "getComparativeMatrix(\"Did this fix a bug?\",michel_results, codethink_results_ben_dooks, \"Michel Maes\", \"Ben Dooks\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "49a545a9-f18b-4f75-8221-5d209a60e127",
   "metadata": {},
   "outputs": [],
   "source": [
    "combined_ben_and_michel_results_df = combineResults(codethink_results_ben_dooks, michel_results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "d99e089d-4f38-45f4-afd4-35ad8567c44a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>Did this fix a bug?</th>\n",
       "      <th>Reasoning</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Ben D./Michel M.</td>\n",
       "      <td>Conflict: no/?</td>\n",
       "      <td>#Ben D.# removed unnecessary work already done elsewhere<br>#Michel M.# not sure</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Ben D./Michel M.</td>\n",
       "      <td>Conflict: yes/no</td>\n",
       "      <td>#Ben D.# removes warning due to linker output<br>#Michel M.# change on Makefile - no code change</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Ben D./Michel M.</td>\n",
       "      <td>Conflict: yes/no</td>\n",
       "      <td>#Ben D.# build configuration dependency issue<br>#Michel M.# change on Kconfig - no code change</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Ben D./Michel M.</td>\n",
       "      <td>Conflict: no/yes</td>\n",
       "      <td>#Ben D.# kernel page allocation api change<br>#Michel M.# commit msg indicates that this change fixes a bug introduced in another commit</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Ben D./Michel M.</td>\n",
       "      <td>Conflict: yes/no</td>\n",
       "      <td>#Ben D.# build configuration dependency issue<br>#Michel M.# change on Kconfig - no code change</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Ben D./Michel M.</td>\n",
       "      <td>Conflict: no/?</td>\n",
       "      <td>#Ben D.# kernel page allocation api change<br>#Michel M.# looks like a refactor - rename variable page</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>Ben D./Michel M.</td>\n",
       "      <td>Conflict: ?/yes</td>\n",
       "      <td>#Ben D.# not enough info in commit<br>#Michel M.# commit msg indicates that the change is needed for HDP flush to work correctly</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>Ben D./Michel M.</td>\n",
       "      <td>Conflict: yes/?</td>\n",
       "      <td>#Ben D.# backlight bug on certain amd hw during shutdown<br>#Michel M.# not sure</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>Ben D./Michel M.</td>\n",
       "      <td>Conflict: no/yes</td>\n",
       "      <td>#Ben D.# nfs issue where data will be retransmitted - todo check if memory leak?<br>#Michel M.# revert a problematic change</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>Ben D./Michel M.</td>\n",
       "      <td>Conflict: no/yes</td>\n",
       "      <td>#Ben D.# removes extra unused branch protection instructions<br>#Michel M.# commit msg indicates that this change fixes a bug introduced in another commit</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "combined_ben_and_michel_results_df_filtered = combined_ben_and_michel_results_df[combined_ben_and_michel_results_df['Did this fix a bug?'].notnull()]\n",
    "showResults(\n",
    "    combined_ben_and_michel_results_df_filtered\n",
    "    [combined_ben_and_michel_results_df_filtered['Did this fix a bug?'].str.contains('Conflict: Conflict:')==False]\n",
    "    [combined_ben_and_michel_results_df_filtered['Did this fix a bug?'].str.contains('Conflict')]\n",
    "    [['Name','Did this fix a bug?','Reasoning']]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a71d25e-8038-4ef2-8155-360b200c555b",
   "metadata": {},
   "source": [
    "#### Michel Comments:\n",
    "\n",
    "- I have not considered as corrected errors those that did not touch source code lines, as in the case of the Makefile and Kconfig configuration files.\n",
    "- I have relied on the format of the commit message, which when it is a bug fix, in most cases, contains the commit which fixes: \"Fixes: \\<commit-hash\\>\"\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}