Bias grader.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bias grader"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook I will examin if it is possible to see if a grader is biased by either giving heigher or lower grades than others. This can later be used to infer the true value of report."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A grader to a question $q$ in handin $h$ might score lower than other graders. So given a score $S$ we may have the following\n",
    "\n",
    "$$ S_{g_1hq} \\neq S_{g_2hq} $$ where $$ g_1 \\neq g_2  $$\n",
    "\n",
    "so for a question $q$ in handin $h$ we can find a \"bias\" of the grader $g$ if comparing the score to the other graders score of same $h$ and $q$. Given enough data we could find a tendency of the grader giving a higher or lower score than the mean of the other graders."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The precision of the bias can vary greatly if there is not enough data to examine. Additionally the grader we are examining does also have a great influence of the score."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this implementation the code should take a grader id and return the bias of the given student.\n",
    "Optional a handin id can also be provided and the bias is found for only the given bias.\n",
    "to start out simple the bias is first found for a question."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "from collections import defaultdict\n",
    "import pymc as pm\n",
    "from pymc.Matplot import plot\n",
    "import numpy as np\n",
    "from matplotlib import pyplot as plt\n",
    "import json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "database dump files for testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "answeres = \"Data/sunes/answer.json\"\n",
    "questions = \"Data/sunes/question.json\"\n",
    "report_grade = \"Data/sunes/report_grade.json\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The max value of a question to normalize the score given by each grader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "question_max_value = dict()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def max_value(question_id):\n",
    "    if question_id in question_max_value:\n",
    "        return question_max_value[question_id]\n",
    "    else:\n",
    "        with open(questions,\"r\") as f:\n",
    "            for line in f:\n",
    "                data = json.loads(line)\n",
    "                q_id = data['_id']['$oid']\n",
    "                # Does not contain \n",
    "                if data[\"question_type\"] == \"boolean\":\n",
    "                    question_max_value[q_id] = 1\n",
    "                elif data[\"question_type\"] == \"numerical\":\n",
    "                    if 'numericalAnswers' in data:\n",
    "                        max_value = max(map(int,data['numericalAnswers'].keys()))\n",
    "                        question_max_value[q_id] = max_value\n",
    "                    else:\n",
    "                        # In the case of no numerical_answeres is provided but the question is numerical\n",
    "                        # then the max value is 5 as these are the default implementation\n",
    "                        question_max_value[q_id] = 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The bias of a grader $g$ for a handin $h$ and question $q$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "grader_handing = dict()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def graders_of_handing(handin):\n",
    "    # Find the graders that have answered the specific handin and question\n",
    "    graders_report = defaultdict(list)\n",
    "    with open(report_grade,\"r\") as f:\n",
    "        for line in f:\n",
    "            if data['status'] == 'ANSWERED': # Ensure that the handin is complete\n",
    "                data = json.loads(line)\n",
    "                q_id = data['_id']['$oid']\n",
    "                if data['handing'] == handin :\n",
    "                    graders_report.append(q_id,data['giver'])\n",
    "    return grader_report\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def bias(grader, handin, question):\n",
    "    \n",
    "    # Find the graders that have answered the specific handin and question\n",
    "    graders_report = graders_of_handing(handin)\n",
    "    \n",
    "    # find the value of all the \n",
    "    with open(answer,\"r\") as f:\n",
    "        for line in f:\n",
    "            data = json.loads(line)\n",
    "            q_id = data['_id']['$oid']\n",
    "\n",
    "    "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}