04_evaluating.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment 4: Evaluating Search Engines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this assignment, we leave aside the code we developed so far, and look into the more general issue of how to evaluate and compare different search engines. The ultimate test for any Information Retrieval system is how well it is able to satisfy the information needs of users."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cohen's Kappa"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our evaluation will involve the calculation of [Cohen's Kappa](https://en.wikipedia.org/wiki/Cohen's_kappa) to quantify the degree to which two human assessors agree or disagree on whether results are considered relevant or not. To calculate Cohen's Kappa, we are going to use the [scikit-learn library](http://scikit-learn.org/stable/):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: scikit-learn in /Users/andrewphilipharrison/anaconda3/lib/python3.7/site-packages (0.21.3)\r\n",
      "Requirement already satisfied: scipy>=0.17.0 in /Users/andrewphilipharrison/anaconda3/lib/python3.7/site-packages (from scikit-learn) (1.3.1)\r\n",
      "Requirement already satisfied: numpy>=1.11.0 in /Users/andrewphilipharrison/anaconda3/lib/python3.7/site-packages (from scikit-learn) (1.17.2)\r\n",
      "Requirement already satisfied: joblib>=0.11 in /Users/andrewphilipharrison/anaconda3/lib/python3.7/site-packages (from scikit-learn) (0.13.2)\r\n"
     ]
    }
   ],
   "source": [
    "! pip install --user scikit-learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import cohen_kappa_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This library expects relevance assessments as lists of elements where `1` stands for _relevant_ and `0` stands for _not relevant_, for example like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "a1=[1,0,1,0,1,0,1,0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This list means that the first document was assessed to be relevant, the second to be not relevant, the third to be relevant etc.\n",
    "\n",
    "We need two assessments in order to calculate Cohen's Kappa, so let's make another exemplary list that only differs on the last element:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "a2=[1,0,1,0,1,0,1,1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now invoke the library as follows to calculate the agreement between the two:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.75"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cohen_kappa_score(a1, a2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This value represents high agreement. We can reach maximal agreement if the two assessments are identical:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cohen_kappa_score(a1, a1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's see what happens for a third assessment that differs on three positions with the first one (the three last positions):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.25"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a3=[1,0,1,0,1,1,0,1]\n",
    "\n",
    "cohen_kappa_score(a1, a3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We get a smaller but still positive value, because these two assessments still mostly agree. If we make a further example that differs on 6 of the 8 positions, we get the following result:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.5"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a4=[1,0,0,1,0,1,0,1]\n",
    "\n",
    "cohen_kappa_score(a1, a4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The score is now negative, because the two differ on more positions than they agree. The agreement is in fact less than what you would expect to occur just by chance. We get the maximal disagreement if we define a fifth example that disagrees on all positions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-1.0"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a5=[0,1,0,1,0,1,0,1]\n",
    "\n",
    "cohen_kappa_score(a1, a5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we understand how this function works, we will apply it below for our specific evaluation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Results and Assessments\n",
    "\n",
    "Next, we will define some auxilary code to deal with lists of URLs from search engines and associated relevance assessments. We will encode result lists like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "urls = [\n",
    "    'https://en.wikipedia.org/wiki/Information_retrieval/',  # 1st result\n",
    "    'http://www.dictionary.com/browse/information',          # 2nd result\n",
    "    'https://nlp.stanford.edu/IR-book/'                      # ...\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we represent corresponding assessments, as above, as lists of the same size containing relevance values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_assessment = [1, 0, 1]\n",
    "another_assessment = [0, 0, 1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to nicely display URL lists, with or without related assessments, we define a function called `display_results`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import display, HTML\n",
    "\n",
    "def display_results(urls, assessment1=None, assessment2=None):\n",
    "    lines = []\n",
    "    lines.append('<table>')\n",
    "    header = '<tr><th>#</th><th>Result URL</th>'\n",
    "    if (assessment1):\n",
    "        header += '<th>Assessment 1</th>'\n",
    "    if (assessment2):\n",
    "        header += '<th>Assessment 2</th>'\n",
    "    header += '</tr>'\n",
    "    lines.append(header)\n",
    "    i = 0\n",
    "    for url in urls:\n",
    "        show_url = url\n",
    "        if (len(url) > 80):\n",
    "            show_url = url[:75] + '...'\n",
    "        line = '<tr><td>{}</td><td><a href=\"{:s}\">{:s}</a></td>'.format(i+1, url, show_url)\n",
    "        if (assessment1):\n",
    "            if (assessment1[i] == 0):\n",
    "                line += '<td><em>Not relevant</em></td>'\n",
    "            else:\n",
    "                line += '<td><strong>Relevant</strong></td>'\n",
    "        if (assessment2):\n",
    "            if (assessment2[i] == 0):\n",
    "                line += '<td><em>Not relevant</em></td>'\n",
    "            else:\n",
    "                line += '<td><strong>Relevant</strong></td>'\n",
    "        line += '</tr>'\n",
    "        lines.append(line)\n",
    "        i = i+1\n",
    "    lines.append('</table>')\n",
    "    display( HTML(''.join(lines)) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use this function to display a list of URLs, optionally together with one or two assessment lists:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Just a list of URLs:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table><tr><th>#</th><th>Result URL</th></tr><tr><td>1</td><td><a href=\"https://en.wikipedia.org/wiki/Information_retrieval/\">https://en.wikipedia.org/wiki/Information_retrieval/</a></td></tr><tr><td>2</td><td><a href=\"http://www.dictionary.com/browse/information\">http://www.dictionary.com/browse/information</a></td></tr><tr><td>3</td><td><a href=\"https://nlp.stanford.edu/IR-book/\">https://nlp.stanford.edu/IR-book/</a></td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "With one assessment:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table><tr><th>#</th><th>Result URL</th><th>Assessment 1</th></tr><tr><td>1</td><td><a href=\"https://en.wikipedia.org/wiki/Information_retrieval/\">https://en.wikipedia.org/wiki/Information_retrieval/</a></td><td><strong>Relevant</strong></td></tr><tr><td>2</td><td><a href=\"http://www.dictionary.com/browse/information\">http://www.dictionary.com/browse/information</a></td><td><em>Not relevant</em></td></tr><tr><td>3</td><td><a href=\"https://nlp.stanford.edu/IR-book/\">https://nlp.stanford.edu/IR-book/</a></td><td><strong>Relevant</strong></td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "With two assessments:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table><tr><th>#</th><th>Result URL</th><th>Assessment 1</th><th>Assessment 2</th></tr><tr><td>1</td><td><a href=\"https://en.wikipedia.org/wiki/Information_retrieval/\">https://en.wikipedia.org/wiki/Information_retrieval/</a></td><td><strong>Relevant</strong></td><td><em>Not relevant</em></td></tr><tr><td>2</td><td><a href=\"http://www.dictionary.com/browse/information\">http://www.dictionary.com/browse/information</a></td><td><em>Not relevant</em></td><td><em>Not relevant</em></td></tr><tr><td>3</td><td><a href=\"https://nlp.stanford.edu/IR-book/\">https://nlp.stanford.edu/IR-book/</a></td><td><strong>Relevant</strong></td><td><strong>Relevant</strong></td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(\"Just a list of URLs:\")\n",
    "display_results(urls)\n",
    "\n",
    "print(\"With one assessment:\")\n",
    "display_results(urls, my_assessment)\n",
    "\n",
    "print(\"With two assessments:\")\n",
    "display_results(urls, my_assessment, another_assessment)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are ready to perform an actual evaluation, which will involve a substantial amount of manual work."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tasks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Your name:** Andrew Harrison"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 1\n",
    "\n",
    "Think up and formulate an information need in the areas of Computer Science or the Life Sciences (medicine, biology, etc.) for which you think the answer can be found in scientific publications. On page 152 in the book an example of such an information need is shown: \"Information on whether drinking red wine is more effective at reducing your risk of heart attacks than white wine.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Answer:** _Information on whether Artificial Intelligence and Machine Learning academic communities are conducting research on privacy and data protection issues_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, write down specifically what documents have to look like to satisfy your information need. For example if your information need is about finding an overview of different cancer types, you could state that a document would need to list at least ten types of cancer to satisfy your information need (among other criteria). Write this down as a protocol with rules and examples. For example, such a protocol could state that at least three out of five given criteria have to be fulfilled for a document to be considered relevant for the information need, and then specify the criteria. Or your protocol could have the form of a sequence of rules, where each rule lets you either label the document as relevant or not relevant, or proceed with the next rule. Such rules and criteria can, for example, be about the general topic of the paper, the concepts mentioned in it, the covered relations between concepts, the type of publication (research paper, overview paper, etc.), the number of references, the types of contained diagrams, and so on, depending on your specified information need."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Answer:** _The protocol contains a sequence of two rules, the first of which is that the document needs to contain either Artificial Intelligence or Machine Learning. The second rule, is that the document also needs to contain privacy or data protection related words (further details below). There are many documents that either of these rules would catch, but the intersection of both should hopefully limit the scope of returned documents those serving the information need.\n",
    "Artificial Intelligence and Machine Learning research uses a multitude of specific and specalised terms, focusing for example on multi-agent learning, or robotic navigation, or NLP, or machine vision. However, as Artificial Intelligence is so in vogue, and the subfield of Machine Learning is currently dominant in AI, either of these terms are likely to be mentioned in relevant papers. AI is also the higher level grouping term.\n",
    "For privacy and data, these are very exacting terms. Privacy is a human right, and very specifically used. Data protection is more multifarious, hence data privacy, data processing, and GDPR (newer EU legislation) terms are also used. Thus the second rule should also catch most papers that are researching these areas._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 2\n",
    "\n",
    "Formulate a keyword query that represents the information need. For the example on page 152 in the book (see above), the example query \"wine AND red AND white AND heart AND attack AND effective\" is given. (You don't need to use connectors like \"AND\", but if you do, make first sure your chosen search engines below actually support them.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Answer:** _(\"artificial intelligence\" OR \"AI\" OR \"machine learning\" OR \"ML\") AND (privacy OR \"data privacy\" OR \"data protection\" OR \"data processing\" OR GDPR)_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then submit your query to **two** of the following academic search engines:\n",
    "\n",
    "- [Google Scholar](https://scholar.google.com) (all science disciplines)\n",
    "- [Semantic Scholar](https://www.semanticscholar.org) (all science disciplines)\n",
    "- [PubMed Search](https://www.ncbi.nlm.nih.gov/pubmed) (Life Sciences / biomedicine)\n",
    "\n",
    "The right choice of two from the three search engine depends on the topic of your information need. If your information need is in the Life Sciences and biomedicine, it's probably best to include PubMed Search, but otherwise you should pick Google Scholar and Semantic Scholar.\n",
    "\n",
    "Extract a list of the top 10 URLs of the lists of each of the search engines\n",
    "given the query. Try to access the resulting publications. For the publications\n",
    "where that is not possible (because of dead links or because the publication is\n",
    "pay-walled even within the VU network), exclude them from the list and add more publications to the end of\n",
    "your list (that is, append results number 11, then 12, etc. to ensure you have\n",
    "two lists of 10 publications each). In order to deal with paywalls, you should try accessing the articles from the VU network, use\n",
    "[UBVU Off-Campus\n",
    "Access](http://www.ub.vu.nl.vu-nl.idm.oclc.org/nl/faciliteiten/toegang-buiten-de-campus/index.aspx), or try to find the respective documents from alternative sources (Google Scholar, for example, is very good at finding free PDFs of articles). If you get fewer than 10 results for one of the search engines, modify the keyword query above to make it more inclusive, and then redo the steps of this task.\n",
    "\n",
    "Store your two lists of URLs in the form of Python lists as introduced above. Then, use the `display_results` function to nicely display them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th>#</th><th>Result URL</th></tr><tr><td>1</td><td><a href=\"https://ergodicity.net/2013/07/\">https://ergodicity.net/2013/07/</a></td></tr><tr><td>2</td><td><a href=\"https://link-springer-com.vu-nl.idm.oclc.org/article/10.1186/s13634-016-0355-x\">https://link-springer-com.vu-nl.idm.oclc.org/article/10.1186/s13634-016-0355-x</a></td></tr><tr><td>3</td><td><a href=\"https://ieeexplore-ieee-org.vu-nl.idm.oclc.org/abstract/document/7958569\">https://ieeexplore-ieee-org.vu-nl.idm.oclc.org/abstract/document/7958569</a></td></tr><tr><td>4</td><td><a href=\"https://arxiv.org/abs/1611.03814\">https://arxiv.org/abs/1611.03814</a></td></tr><tr><td>5</td><td><a href=\"https://ieeexplore-ieee-org.vu-nl.idm.oclc.org/abstract/document/6582713\">https://ieeexplore-ieee-org.vu-nl.idm.oclc.org/abstract/document/6582713</a></td></tr><tr><td>6</td><td><a href=\"https://books.google.com/books?hl=en&lr=&id=azyjBQAAQBAJ&oi=fnd&pg=PP1&dq=(%22artificial+intelligence%22+OR+%22AI%22+OR+%22machine+learning%22+OR+%22ML%22)+AND+(privacy+OR+%22data+privacy%22+OR+%22data+protection%22+OR+%22data+processing%22+OR+GDPR)&ots=nikG9cP8Hc&sig=rXKSapUp9psWT5Nj-HDHPeqfl0o\">https://books.google.com/books?hl=en&lr=&id=azyjBQAAQBAJ&oi=fnd&pg=PP1&dq=(...</a></td></tr><tr><td>7</td><td><a href=\"http://iot.stanford.edu/pubs/bost-learning-ndss15.pdf\">http://iot.stanford.edu/pubs/bost-learning-ndss15.pdf</a></td></tr><tr><td>8</td><td><a href=\"https://dl.acm.org/citation.cfm?id=573193\">https://dl.acm.org/citation.cfm?id=573193</a></td></tr><tr><td>9</td><td><a href=\"https://www.sciencedirect.com/science/article/pii/0022519367900975\">https://www.sciencedirect.com/science/article/pii/0022519367900975</a></td></tr><tr><td>10</td><td><a href=\"https://ieeexplore.ieee.org/abstract/document/5403147/\">https://ieeexplore.ieee.org/abstract/document/5403147/</a></td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table><tr><th>#</th><th>Result URL</th></tr><tr><td>1</td><td><a href=\"https://www.semanticscholar.org/paper/Privacy-preserving-distributed-mining-of-rules-on-Kantarcioglu-Clifton/7435b796a7b3136b772e0d8d826fa927c04c5e2e\">https://www.semanticscholar.org/paper/Privacy-preserving-distributed-mining...</a></td></tr><tr><td>2</td><td><a href=\"https://www.semanticscholar.org/paper/Transforming-data-to-satisfy-privacy-constraints-Iyengar/2676d77b4e4cc58250ed20b4f85576a9fb33ae5a\">https://www.semanticscholar.org/paper/Transforming-data-to-satisfy-privacy-...</a></td></tr><tr><td>3</td><td><a href=\"https://www.semanticscholar.org/paper/Data-privacy-through-optimal-k-anonymization-Bayardo-Agrawal/d429d387f94e16fe2fcf63224758df9b0ff81430\">https://www.semanticscholar.org/paper/Data-privacy-through-optimal-k-anonym...</a></td></tr><tr><td>4</td><td><a href=\"https://www.semanticscholar.org/paper/Atomic-Data-and-Nuclear-Data-Tables-Sawey-Berrington/dcee8aeff596fb54fb649ed6633c9316f56db9b3\">https://www.semanticscholar.org/paper/Atomic-Data-and-Nuclear-Data-Tables-S...</a></td></tr><tr><td>5</td><td><a href=\"https://www.semanticscholar.org/paper/Our-Data%2C-Ourselves%3A-Privacy-Via-Distributed-Noise-Dwork-Kenthapadi/1808b64aec21863489f0fe66f250890a3ac2b843\">https://www.semanticscholar.org/paper/Our-Data%2C-Ourselves%3A-Privacy-Via-...</a></td></tr><tr><td>6</td><td><a href=\"https://www.semanticscholar.org/paper/Privacy-Preserving-Data-Mining-Lindell-Pinkas/9a1d75a5c79e603ab7ceecc4bb07ee736e402c18\">https://www.semanticscholar.org/paper/Privacy-Preserving-Data-Mining-Lindel...</a></td></tr><tr><td>7</td><td><a href=\"https://www.semanticscholar.org/paper/On-the-Design-and-Quantification-of-Privacy-Data-Agrawal-Aggarwal/20780ad665e496aa128f82713bb78d13fd87cd0a\">https://www.semanticscholar.org/paper/On-the-Design-and-Quantification-of-P...</a></td></tr><tr><td>8</td><td><a href=\"https://www.semanticscholar.org/paper/ON-DATA-BANKS-AND-PRIVACY-HOMOMORPHISMS-Rivest-Dertouzos/c365f01d330b2211e74069120e88cff37eacbcf5\">https://www.semanticscholar.org/paper/ON-DATA-BANKS-AND-PRIVACY-HOMOMORPHIS...</a></td></tr><tr><td>9</td><td><a href=\"https://www.semanticscholar.org/paper/Limiting-privacy-breaches-in-privacy-preserving-Evfimievski-Gehrke/03b01daccd140e7d65358f31f8c4472d18573a5a\">https://www.semanticscholar.org/paper/Limiting-privacy-breaches-in-privacy-...</a></td></tr><tr><td>10</td><td><a href=\"https://www.semanticscholar.org/paper/Privacy-Preserving-Multi-keyword-Ranked-Search-over-Vishvapathi-Reddy/fe274555cc1dd0609b72f666d0aa063ab68106e6\">https://www.semanticscholar.org/paper/Privacy-Preserving-Multi-keyword-Rank...</a></td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Create two of the lists below, depending on your chosen engines:\n",
    "\n",
    "urls_google = [\n",
    "    'https://ergodicity.net/2013/07/',  # 1st result\n",
    "    'https://link-springer-com.vu-nl.idm.oclc.org/article/10.1186/s13634-016-0355-x',          # 2nd result\n",
    "    'https://ieeexplore-ieee-org.vu-nl.idm.oclc.org/abstract/document/7958569',                     # 3rd result\n",
    "    'https://arxiv.org/abs/1611.03814', # 4th result\n",
    "    'https://ieeexplore-ieee-org.vu-nl.idm.oclc.org/abstract/document/6582713', # 5th result\n",
    "    'https://books.google.com/books?hl=en&lr=&id=azyjBQAAQBAJ&oi=fnd&pg=PP1&dq=(%22artificial+intelligence%22+OR+%22AI%22+OR+%22machine+learning%22+OR+%22ML%22)+AND+(privacy+OR+%22data+privacy%22+OR+%22data+protection%22+OR+%22data+processing%22+OR+GDPR)&ots=nikG9cP8Hc&sig=rXKSapUp9psWT5Nj-HDHPeqfl0o', # 6th result\n",
    "    'http://iot.stanford.edu/pubs/bost-learning-ndss15.pdf', # 7th result\n",
    "    'https://dl.acm.org/citation.cfm?id=573193', # 8th result\n",
    "    'https://www.sciencedirect.com/science/article/pii/0022519367900975', # 9th result\n",
    "    'https://ieeexplore.ieee.org/abstract/document/5403147/', #10th result\n",
    "]\n",
    "\n",
    "urls_semantic = [\n",
    "    'https://www.semanticscholar.org/paper/Privacy-preserving-distributed-mining-of-rules-on-Kantarcioglu-Clifton/7435b796a7b3136b772e0d8d826fa927c04c5e2e',  # 1st result\n",
    "    'https://www.semanticscholar.org/paper/Transforming-data-to-satisfy-privacy-constraints-Iyengar/2676d77b4e4cc58250ed20b4f85576a9fb33ae5a',          # 2nd result\n",
    "    'https://www.semanticscholar.org/paper/Data-privacy-through-optimal-k-anonymization-Bayardo-Agrawal/d429d387f94e16fe2fcf63224758df9b0ff81430',                      # 3rd result\n",
    "    'https://www.semanticscholar.org/paper/Atomic-Data-and-Nuclear-Data-Tables-Sawey-Berrington/dcee8aeff596fb54fb649ed6633c9316f56db9b3', # 4th result\n",
    "    'https://www.semanticscholar.org/paper/Our-Data%2C-Ourselves%3A-Privacy-Via-Distributed-Noise-Dwork-Kenthapadi/1808b64aec21863489f0fe66f250890a3ac2b843', # 5th result\n",
    "    'https://www.semanticscholar.org/paper/Privacy-Preserving-Data-Mining-Lindell-Pinkas/9a1d75a5c79e603ab7ceecc4bb07ee736e402c18', # 6th result\n",
    "    'https://www.semanticscholar.org/paper/On-the-Design-and-Quantification-of-Privacy-Data-Agrawal-Aggarwal/20780ad665e496aa128f82713bb78d13fd87cd0a', # 7th result\n",
    "    'https://www.semanticscholar.org/paper/ON-DATA-BANKS-AND-PRIVACY-HOMOMORPHISMS-Rivest-Dertouzos/c365f01d330b2211e74069120e88cff37eacbcf5', # 8th result\n",
    "    'https://www.semanticscholar.org/paper/Limiting-privacy-breaches-in-privacy-preserving-Evfimievski-Gehrke/03b01daccd140e7d65358f31f8c4472d18573a5a', # 9th result\n",
    "    'https://www.semanticscholar.org/paper/Privacy-Preserving-Multi-keyword-Ranked-Search-over-Vishvapathi-Reddy/fe274555cc1dd0609b72f666d0aa063ab68106e6', #10th result\n",
    "]\n",
    "#urls_pubmed = ...\n",
    "\n",
    "# Call display_results here\n",
    "display_results(urls_google)\n",
    "display_results(urls_semantic)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 3\n",
    "\n",
    "Then, find a fellow student who will **independently**\n",
    "assess the results as \"relevant\" or \"not relevant\" using the protocol that you\n",
    "have defined above, and also help (at least) one other student for his/her\n",
    "assessment. Write down their names here:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Name of the student who assesses my results:** _Mathias Parisot_\n",
    "\n",
    "**Name of the student who I help to assess his/her results:** _Mathias Parisot_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show to the other assessor everything you have written down above for Tasks 1 and 2 (and you might also want to give him/her the PDFs you got for these papers to simplify the process).\n",
    "\n",
    "You as assessors need to stick to the protocol you made in Task 1 and should not discuss with each other, especially when you doubt whether a result is relevant or not. Write down your assessments as lists of relevance values, as introduced above, and make sure they correctly map to the URLs by displaying them together with the `display_results` function.\n",
    "\n",
    "To avoid problems with extreme results, mark in each list at least one paper as 'relevant' and at least one paper as 'not relevant'. That is, if all papers seem relevant, mark the one that seems least relevant 'not relevant', and conversely, if none of the papers seem relevant, mark the one that seems a bit more relevant than the others as 'relevant'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th>#</th><th>Result URL</th><th>Assessment 1</th><th>Assessment 2</th></tr><tr><td>1</td><td><a href=\"https://ergodicity.net/2013/07/\">https://ergodicity.net/2013/07/</a></td><td><em>Not relevant</em></td><td><em>Not relevant</em></td></tr><tr><td>2</td><td><a href=\"https://link-springer-com.vu-nl.idm.oclc.org/article/10.1186/s13634-016-0355-x\">https://link-springer-com.vu-nl.idm.oclc.org/article/10.1186/s13634-016-0355-x</a></td><td><strong>Relevant</strong></td><td><em>Not relevant</em></td></tr><tr><td>3</td><td><a href=\"https://ieeexplore-ieee-org.vu-nl.idm.oclc.org/abstract/document/7958569\">https://ieeexplore-ieee-org.vu-nl.idm.oclc.org/abstract/document/7958569</a></td><td><strong>Relevant</strong></td><td><strong>Relevant</strong></td></tr><tr><td>4</td><td><a href=\"https://arxiv.org/abs/1611.03814\">https://arxiv.org/abs/1611.03814</a></td><td><strong>Relevant</strong></td><td><strong>Relevant</strong></td></tr><tr><td>5</td><td><a href=\"https://ieeexplore-ieee-org.vu-nl.idm.oclc.org/abstract/document/6582713\">https://ieeexplore-ieee-org.vu-nl.idm.oclc.org/abstract/document/6582713</a></td><td><strong>Relevant</strong></td><td><strong>Relevant</strong></td></tr><tr><td>6</td><td><a href=\"https://books.google.com/books?hl=en&lr=&id=azyjBQAAQBAJ&oi=fnd&pg=PP1&dq=(%22artificial+intelligence%22+OR+%22AI%22+OR+%22machine+learning%22+OR+%22ML%22)+AND+(privacy+OR+%22data+privacy%22+OR+%22data+protection%22+OR+%22data+processing%22+OR+GDPR)&ots=nikG9cP8Hc&sig=rXKSapUp9psWT5Nj-HDHPeqfl0o\">https://books.google.com/books?hl=en&lr=&id=azyjBQAAQBAJ&oi=fnd&pg=PP1&dq=(...</a></td><td><em>Not relevant</em></td><td><em>Not relevant</em></td></tr><tr><td>7</td><td><a href=\"http://iot.stanford.edu/pubs/bost-learning-ndss15.pdf\">http://iot.stanford.edu/pubs/bost-learning-ndss15.pdf</a></td><td><strong>Relevant</strong></td><td><strong>Relevant</strong></td></tr><tr><td>8</td><td><a href=\"https://dl.acm.org/citation.cfm?id=573193\">https://dl.acm.org/citation.cfm?id=573193</a></td><td><em>Not relevant</em></td><td><em>Not relevant</em></td></tr><tr><td>9</td><td><a href=\"https://www.sciencedirect.com/science/article/pii/0022519367900975\">https://www.sciencedirect.com/science/article/pii/0022519367900975</a></td><td><em>Not relevant</em></td><td><em>Not relevant</em></td></tr><tr><td>10</td><td><a href=\"https://ieeexplore.ieee.org/abstract/document/5403147/\">https://ieeexplore.ieee.org/abstract/document/5403147/</a></td><td><em>Not relevant</em></td><td><em>Not relevant</em></td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table><tr><th>#</th><th>Result URL</th><th>Assessment 1</th><th>Assessment 2</th></tr><tr><td>1</td><td><a href=\"https://www.semanticscholar.org/paper/Privacy-preserving-distributed-mining-of-rules-on-Kantarcioglu-Clifton/7435b796a7b3136b772e0d8d826fa927c04c5e2e\">https://www.semanticscholar.org/paper/Privacy-preserving-distributed-mining...</a></td><td><em>Not relevant</em></td><td><em>Not relevant</em></td></tr><tr><td>2</td><td><a href=\"https://www.semanticscholar.org/paper/Transforming-data-to-satisfy-privacy-constraints-Iyengar/2676d77b4e4cc58250ed20b4f85576a9fb33ae5a\">https://www.semanticscholar.org/paper/Transforming-data-to-satisfy-privacy-...</a></td><td><strong>Relevant</strong></td><td><strong>Relevant</strong></td></tr><tr><td>3</td><td><a href=\"https://www.semanticscholar.org/paper/Data-privacy-through-optimal-k-anonymization-Bayardo-Agrawal/d429d387f94e16fe2fcf63224758df9b0ff81430\">https://www.semanticscholar.org/paper/Data-privacy-through-optimal-k-anonym...</a></td><td><em>Not relevant</em></td><td><strong>Relevant</strong></td></tr><tr><td>4</td><td><a href=\"https://www.semanticscholar.org/paper/Atomic-Data-and-Nuclear-Data-Tables-Sawey-Berrington/dcee8aeff596fb54fb649ed6633c9316f56db9b3\">https://www.semanticscholar.org/paper/Atomic-Data-and-Nuclear-Data-Tables-S...</a></td><td><em>Not relevant</em></td><td><em>Not relevant</em></td></tr><tr><td>5</td><td><a href=\"https://www.semanticscholar.org/paper/Our-Data%2C-Ourselves%3A-Privacy-Via-Distributed-Noise-Dwork-Kenthapadi/1808b64aec21863489f0fe66f250890a3ac2b843\">https://www.semanticscholar.org/paper/Our-Data%2C-Ourselves%3A-Privacy-Via-...</a></td><td><strong>Relevant</strong></td><td><strong>Relevant</strong></td></tr><tr><td>6</td><td><a href=\"https://www.semanticscholar.org/paper/Privacy-Preserving-Data-Mining-Lindell-Pinkas/9a1d75a5c79e603ab7ceecc4bb07ee736e402c18\">https://www.semanticscholar.org/paper/Privacy-Preserving-Data-Mining-Lindel...</a></td><td><strong>Relevant</strong></td><td><strong>Relevant</strong></td></tr><tr><td>7</td><td><a href=\"https://www.semanticscholar.org/paper/On-the-Design-and-Quantification-of-Privacy-Data-Agrawal-Aggarwal/20780ad665e496aa128f82713bb78d13fd87cd0a\">https://www.semanticscholar.org/paper/On-the-Design-and-Quantification-of-P...</a></td><td><em>Not relevant</em></td><td><strong>Relevant</strong></td></tr><tr><td>8</td><td><a href=\"https://www.semanticscholar.org/paper/ON-DATA-BANKS-AND-PRIVACY-HOMOMORPHISMS-Rivest-Dertouzos/c365f01d330b2211e74069120e88cff37eacbcf5\">https://www.semanticscholar.org/paper/ON-DATA-BANKS-AND-PRIVACY-HOMOMORPHIS...</a></td><td><em>Not relevant</em></td><td><em>Not relevant</em></td></tr><tr><td>9</td><td><a href=\"https://www.semanticscholar.org/paper/Limiting-privacy-breaches-in-privacy-preserving-Evfimievski-Gehrke/03b01daccd140e7d65358f31f8c4472d18573a5a\">https://www.semanticscholar.org/paper/Limiting-privacy-breaches-in-privacy-...</a></td><td><strong>Relevant</strong></td><td><strong>Relevant</strong></td></tr><tr><td>10</td><td><a href=\"https://www.semanticscholar.org/paper/Privacy-Preserving-Multi-keyword-Ranked-Search-over-Vishvapathi-Reddy/fe274555cc1dd0609b72f666d0aa063ab68106e6\">https://www.semanticscholar.org/paper/Privacy-Preserving-Multi-keyword-Rank...</a></td><td><em>Not relevant</em></td><td><strong>Relevant</strong></td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 0 = not relevant; 1 = relevant\n",
    "\n",
    "# You only need to create 4 of the following 6 lists, again depending on which search engines you chose.\n",
    "\n",
    "# Assessment 1 is from you:\n",
    "\n",
    "assessment1_google = [0,1,1,1,1,0,1,0,0,0]\n",
    "assessment1_semantic = [0,1,0,0,1,1,0,0,1,0]\n",
    "#assessment1_pubmed = ...\n",
    "\n",
    "# Assessment 2 is from your fellow student (don't show him/her your own assessment!):\n",
    "\n",
    "assessment2_google = [0,0,1,1,1,0,1,0,0,0]\n",
    "assessment2_semantic = [0,1,1,0,1,1,1,0,1,1]\n",
    "#assessment2_pubmed = ...\n",
    "\n",
    "# Call display_results here\n",
    "display_results(urls_google,assessment1_google, assessment2_google)\n",
    "display_results(urls_semantic, assessment1_semantic, assessment2_semantic)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 4\n",
    "\n",
    "Compute Cohen's kappa to quantify how much the two assessors agreed. Use the function `cohen_kappa_score` demonstrated above to calculate two times the inter-annotator agreement (once for each of the two search engines), and print out the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Kappa for Google Scholar: 0.8\n",
      "Kappa for Semantic Scholar: 0.44444444444444453\n"
     ]
    }
   ],
   "source": [
    "# Add your code here:\n",
    "\n",
    "kappa_google = cohen_kappa_score(assessment1_google, assessment2_google)\n",
    "kappa_semantic = cohen_kappa_score(assessment1_semantic, assessment2_semantic)\n",
    "#kappa_pubmed = ...\n",
    "\n",
    "print(\"Kappa for Google Scholar:\", kappa_google)\n",
    "print(\"Kappa for Semantic Scholar:\", kappa_semantic)\n",
    "#print(\"Kappa for PubMed:\", kappa_pubmed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Explain whether the agreement can be considered high or not, based on the interpretation table on [this Wikipedia page](https://en.wikipedia.org/wiki/Fleiss'_kappa#Interpretation) (this Wikipedia page is about a different type of kappa but the interpretation table can also be used for Cohen's kappa)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Answer:** _Agreement for Google Scholar is considered \"Substantial agreement\" according to the interpretation table. Whereas agreement on Semantic Scholar is considered \"Moderate agreement\". Note the kappa will also be higher when the number of catergories are low (and here there are only 2), and that these intervals are just personal opinion and not based on evidence. So I would just be comfortable asserting that the Google Scholar agreement is high, but no so much that the Semantic Scholar agreement is. Note further we're not doing any statistical checks for significance on these values either._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 5\n",
    "\n",
    "Define a function called `precision_at_n` that calculates Precision@n as described in the lecture slides, which takes as input an assessment list and a value for _n_ and returns the respective Precision@n value. Run this function to calculate Precision@10 (that is, n=10) on all four assessments (two assessors and two search engines)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Precision@10 for asssesment1_google: 0.5\n",
      "Precision@10 for asssesment2_google: 0.4\n",
      "Precision@10 for asssesment1_semantic: 0.4\n",
      "Precision@10 for asssesment1_semantic: 0.7\n"
     ]
    }
   ],
   "source": [
    "# Add your code here:\n",
    "\n",
    "def precision_at_n(assessment_list, n):\n",
    "    relevant = 0\n",
    "    for assessment in assessment_list:\n",
    "        if assessment == 1:\n",
    "            relevant+=1\n",
    "    return (relevant/n)\n",
    "\n",
    "# Print out Precision@10 for all assessments here.\n",
    "print('Precision@10 for asssesment1_google:',precision_at_n(assessment1_google,10))\n",
    "print('Precision@10 for asssesment2_google:',precision_at_n(assessment2_google,10))\n",
    "print('Precision@10 for asssesment1_semantic:',precision_at_n(assessment1_semantic,10))\n",
    "print('Precision@10 for asssesment1_semantic:',precision_at_n(assessment2_semantic,10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Explain what these specific Precision@10 results tell us (or don't tell us) about the quality of the two search engines for your particular information need. You can also refer to the results of Task 4 if necessary."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Answer:** _We have no knowledge of total relevant documents, so cannot say anything about recall. We can however discuss the precision, as we have true positives and false positives. Thus why it's precision at 10. However, given there are multiple assessors, how to combine these values? For the particular information need, a simple average would give 0.45 for Google Scholar and 0.55 for Semantic Scholar, perhaps implying that Semantic Scholar is more precise. Or, you could take the lowest precision value from across the assessors for each search engine, giving both 0.4 precision and thus saying they're the same. Alternatively, you may  take the average and weight it by the level of agreement, hence 0.45*0.8=0.36 and 0.55*0.44=0.242 and hence say Google Scholar is more precise. Finally, Precision@10 doesn't say anything about the ordering of the results, as for Assesor 1 the first 5 Google Scholar gets 4 relevant out of 5, and Semantic only 2 out of 5 (albeit, you could reduce to precision@5). But staying with precision@10, this is information that may be relevant, but is not captured in the difference. Maybe the first 5 search results aren't relevant for one search engine, but 6-10 are. For another search engine, the first 5 results are all relevant, and none of 6-10 are. Both of these would have Precision@10 of 0.5 but there is a case to argue for the latter search engine being better. In summary, it tells us some things, that the search engines aren't returning total rubbish. But I would be hard pressed to choose a winner._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Submission"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Submit the answers to the assignment via Canvas as a modified version of this Notebook file (file with `.ipynb` extension) that includes your code and your answers.\n",
    "\n",
    "Before submitting, restart the kernel and re-run the complete code (**Kernel > Restart & Run All**), and then check whether your assignment code still works as expected.\n",
    "\n",
    "Don't forget to add your name, and remember that the assignments have to be done individually and group submissions are **not allowed**."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}