task02-abalone.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Machine Learning Lab - Hackathon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Summer Term 2021**\n",
    "\n",
    "- Julian Stier <julian.stier@uni-passau.de>\n",
    "- Sahib Julka <sahib.julka@uni-passau.de>\n",
    "- [StudIP Machine Learning Lab](https://studip.uni-passau.de/studip/dispatch.php/course/scm?cid=42befdd6822ee2029b26fa475cd02f60)\n",
    "- [FimGIT repositories](https://fimgit.fim.uni-passau.de/groups/padas/21ss-mllab/)\n",
    "\n",
    "**General Remarks**\n",
    "- You have time from 09:00 AM until 03:00 PM to work on the hackathon task.\n",
    "- Go through the notebook, answer questions, solve described tasks and fill out empty spaces or add cells based on your creativity.\n",
    "- Re-use previous implementations (of your own!) by either importing according python modules or copying it into the notebook.\n",
    "- Your overall git repository acts as the official submission. Put the hackathon notebook also into the git repository, alongside with any previous notebooks or python implementations you already uploaded.\n",
    "- If one of your implementation required for this notebook has not been working previously, you can now work on that specifically and try to solve it within the given time frame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 405,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 406,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "nb_dir = os.path.split(os.getcwd())[0]\n",
    "if nb_dir not in sys.path:\n",
    "    sys.path.append(nb_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step I: Prepare Your Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Download the two datasets.\n",
    "- Read it into memory.\n",
    "- Understand the feature shape and number of targets.\n",
    "- Split both datasets into three fixed train-validation-test sets with own chosen proportions. You can e.g. use 80% of the data for training, 10% of the data for the validation set and 10% for the test set. Make sure you shuffle the data in before once."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### UCI Dataset: Abalone\n",
    "> https://archive.ics.uci.edu/ml/machine-learning-databases/abalone/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 407,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2021-07-20 14:44:05--  https://archive.ics.uci.edu/ml/machine-learning-databases/abalone/abalone.data\n",
      "Resolving archive.ics.uci.edu (archive.ics.uci.edu)... 128.195.10.252\n",
      "Connecting to archive.ics.uci.edu (archive.ics.uci.edu)|128.195.10.252|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 191873 (187K) [application/x-httpd-php]\n",
      "Saving to: ‘./data/abalone/abalone.data.4’\n",
      "\n",
      "abalone.data.4      100%[===================>] 187.38K   354KB/s    in 0.5s    \n",
      "\n",
      "2021-07-20 14:44:06 (354 KB/s) - ‘./data/abalone/abalone.data.4’ saved [191873/191873]\n",
      "\n",
      "--2021-07-20 14:44:06--  https://archive.ics.uci.edu/ml/machine-learning-databases/abalone/abalone.names\n",
      "Resolving archive.ics.uci.edu (archive.ics.uci.edu)... 128.195.10.252\n",
      "Connecting to archive.ics.uci.edu (archive.ics.uci.edu)|128.195.10.252|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 4319 (4.2K) [application/x-httpd-php]\n",
      "Saving to: ‘./data/abalone/abalone.names.4’\n",
      "\n",
      "abalone.names.4     100%[===================>]   4.22K  --.-KB/s    in 0s      \n",
      "\n",
      "2021-07-20 14:44:07 (65.4 MB/s) - ‘./data/abalone/abalone.names.4’ saved [4319/4319]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget -P ./data/abalone/ https://archive.ics.uci.edu/ml/machine-learning-databases/abalone/abalone.data\n",
    "!wget -P ./data/abalone/ https://archive.ics.uci.edu/ml/machine-learning-databases/abalone/abalone.names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 408,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1. Title of Database: Abalone data\r\n",
      "\r\n",
      "2. Sources:\r\n",
      "\r\n",
      "   (a) Original owners of database:\r\n",
      "\tMarine Resources Division\r\n",
      "\tMarine Research Laboratories - Taroona\r\n",
      "\tDepartment of Primary Industry and Fisheries, Tasmania\r\n",
      "\tGPO Box 619F, Hobart, Tasmania 7001, Australia\r\n",
      "\t(contact: Warwick Nash +61 02 277277, wnash@dpi.tas.gov.au)\r\n",
      "\r\n",
      "   (b) Donor of database:\r\n",
      "\tSam Waugh (Sam.Waugh@cs.utas.edu.au)\r\n",
      "\tDepartment of Computer Science, University of Tasmania\r\n",
      "\tGPO Box 252C, Hobart, Tasmania 7001, Australia\r\n",
      "\r\n",
      "   (c) Date received: December 1995\r\n",
      "\r\n",
      "\r\n",
      "3. Past Usage:\r\n",
      "\r\n",
      "   Sam Waugh (1995) \"Extending and benchmarking Cascade-Correlation\", PhD\r\n",
      "   thesis, Computer Science Department, University of Tasmania.\r\n",
      "\r\n",
      "   -- Test set performance (final 1044 examples, first 3133 used for training):\r\n",
      "\t24.86% Cascade-Correlation (no hidden nodes)\r\n",
      "\t26.25% Cascade-Correlation (5 hidden nodes)\r\n",
      "\t21.5%  C4.5\r\n",
      "\t 0.0%  Linear Discriminate Analysis\r\n",
      "\t 3.57% k=5 Nearest Neighbour\r\n",
      "      (Problem encoded as a classification task)\r\n",
      "\r\n",
      "   -- Data set samples are highly overlapped.  Further information is required\r\n",
      "\tto separate completely using affine combinations.  Other restrictions\r\n",
      "\tto data set examined.\r\n",
      "\r\n",
      "   David Clark, Zoltan Schreter, Anthony Adams \"A Quantitative Comparison of\r\n",
      "   Dystal and Backpropagation\", submitted to the Australian Conference on\r\n",
      "   Neural Networks (ACNN'96). Data set treated as a 3-category classification\r\n",
      "   problem (grouping ring classes 1-8, 9 and 10, and 11 on).\r\n",
      "\r\n",
      "   -- Test set performance (3133 training, 1044 testing as above):\r\n",
      "\t64%    Backprop\r\n",
      "\t55%    Dystal\r\n",
      "   -- Previous work (Waugh, 1995) on same data set:\r\n",
      "\t61.40% Cascade-Correlation (no hidden nodes)\r\n",
      "\t65.61% Cascade-Correlation (5 hidden nodes)\r\n",
      "\t59.2%  C4.5\r\n",
      "\t32.57% Linear Discriminate Analysis\r\n",
      "\t62.46% k=5 Nearest Neighbour\r\n",
      "\r\n",
      "\r\n",
      "4. Relevant Information Paragraph:\r\n",
      "\r\n",
      "   Predicting the age of abalone from physical measurements.  The age of\r\n",
      "   abalone is determined by cutting the shell through the cone, staining it,\r\n",
      "   and counting the number of rings through a microscope -- a boring and\r\n",
      "   time-consuming task.  Other measurements, which are easier to obtain, are\r\n",
      "   used to predict the age.  Further information, such as weather patterns\r\n",
      "   and location (hence food availability) may be required to solve the problem.\r\n",
      "\r\n",
      "   From the original data examples with missing values were removed (the\r\n",
      "   majority having the predicted value missing), and the ranges of the\r\n",
      "   continuous values have been scaled for use with an ANN (by dividing by 200).\r\n",
      "\r\n",
      "   Data comes from an original (non-machine-learning) study:\r\n",
      "\r\n",
      "\tWarwick J Nash, Tracy L Sellers, Simon R Talbot, Andrew J Cawthorn and\r\n",
      "\tWes B Ford (1994) \"The Population Biology of Abalone (_Haliotis_\r\n",
      "\tspecies) in Tasmania. I. Blacklip Abalone (_H. rubra_) from the North\r\n",
      "\tCoast and Islands of Bass Strait\", Sea Fisheries Division, Technical\r\n",
      "\tReport No. 48 (ISSN 1034-3288)\r\n",
      "\r\n",
      "\r\n",
      "5. Number of Instances: 4177\r\n",
      "\r\n",
      "\r\n",
      "6. Number of Attributes: 8\r\n",
      "\r\n",
      "\r\n",
      "7. Attribute information:\r\n",
      "\r\n",
      "   Given is the attribute name, attribute type, the measurement unit and a\r\n",
      "   brief description.  The number of rings is the value to predict: either\r\n",
      "   as a continuous value or as a classification problem.\r\n",
      "\r\n",
      "\tName\t\tData Type\tMeas.\tDescription\r\n",
      "\t----\t\t---------\t-----\t-----------\r\n",
      "\tSex\t\tnominal\t\t\tM, F, and I (infant)\r\n",
      "\tLength\t\tcontinuous\tmm\tLongest shell measurement\r\n",
      "\tDiameter\tcontinuous\tmm\tperpendicular to length\r\n",
      "\tHeight\t\tcontinuous\tmm\twith meat in shell\r\n",
      "\tWhole weight\tcontinuous\tgrams\twhole abalone\r\n",
      "\tShucked weight\tcontinuous\tgrams\tweight of meat\r\n",
      "\tViscera weight\tcontinuous\tgrams\tgut weight (after bleeding)\r\n",
      "\tShell weight\tcontinuous\tgrams\tafter being dried\r\n",
      "\tRings\t\tinteger\t\t\t+1.5 gives the age in years\r\n",
      "\r\n",
      "   Statistics for numeric domains:\r\n",
      "\r\n",
      "\t\tLength\tDiam\tHeight\tWhole\tShucked\tViscera\tShell\tRings\r\n",
      "\tMin\t0.075\t0.055\t0.000\t0.002\t0.001\t0.001\t0.002\t    1\r\n",
      "\tMax\t0.815\t0.650\t1.130\t2.826\t1.488\t0.760\t1.005\t   29\r\n",
      "\tMean\t0.524\t0.408\t0.140\t0.829\t0.359\t0.181\t0.239\t9.934\r\n",
      "\tSD\t0.120\t0.099\t0.042\t0.490\t0.222\t0.110\t0.139\t3.224\r\n",
      "\tCorrel\t0.557\t0.575\t0.557\t0.540\t0.421\t0.504\t0.628\t  1.0\r\n",
      "\r\n",
      "\r\n",
      "8. Missing Attribute Values: None\r\n",
      "\r\n",
      "\r\n",
      "9. Class Distribution:\r\n",
      "\r\n",
      "\tClass\tExamples\r\n",
      "\t-----\t--------\r\n",
      "\t1\t1\r\n",
      "\t2\t1\r\n",
      "\t3\t15\r\n",
      "\t4\t57\r\n",
      "\t5\t115\r\n",
      "\t6\t259\r\n",
      "\t7\t391\r\n",
      "\t8\t568\r\n",
      "\t9\t689\r\n",
      "\t10\t634\r\n",
      "\t11\t487\r\n",
      "\t12\t267\r\n",
      "\t13\t203\r\n",
      "\t14\t126\r\n",
      "\t15\t103\r\n",
      "\t16\t67\r\n",
      "\t17\t58\r\n",
      "\t18\t42\r\n",
      "\t19\t32\r\n",
      "\t20\t26\r\n",
      "\t21\t14\r\n",
      "\t22\t6\r\n",
      "\t23\t9\r\n",
      "\t24\t2\r\n",
      "\t25\t1\r\n",
      "\t26\t1\r\n",
      "\t27\t2\r\n",
      "\t29\t1\r\n",
      "\t-----\t----\r\n",
      "\tTotal\t4177\r\n"
     ]
    }
   ],
   "source": [
    "!cat ./data/abalone/abalone.names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 409,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sex</th>\n",
       "      <th>Length</th>\n",
       "      <th>Diameter</th>\n",
       "      <th>Height</th>\n",
       "      <th>Whole weight</th>\n",
       "      <th>Shucked weight</th>\n",
       "      <th>Viscera weight</th>\n",
       "      <th>Shell weight</th>\n",
       "      <th>Rings</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>M</td>\n",
       "      <td>0.455</td>\n",
       "      <td>0.365</td>\n",
       "      <td>0.095</td>\n",
       "      <td>0.5140</td>\n",
       "      <td>0.2245</td>\n",
       "      <td>0.1010</td>\n",
       "      <td>0.150</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>M</td>\n",
       "      <td>0.350</td>\n",
       "      <td>0.265</td>\n",
       "      <td>0.090</td>\n",
       "      <td>0.2255</td>\n",
       "      <td>0.0995</td>\n",
       "      <td>0.0485</td>\n",
       "      <td>0.070</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>F</td>\n",
       "      <td>0.530</td>\n",
       "      <td>0.420</td>\n",
       "      <td>0.135</td>\n",
       "      <td>0.6770</td>\n",
       "      <td>0.2565</td>\n",
       "      <td>0.1415</td>\n",
       "      <td>0.210</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>M</td>\n",
       "      <td>0.440</td>\n",
       "      <td>0.365</td>\n",
       "      <td>0.125</td>\n",
       "      <td>0.5160</td>\n",
       "      <td>0.2155</td>\n",
       "      <td>0.1140</td>\n",
       "      <td>0.155</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>I</td>\n",
       "      <td>0.330</td>\n",
       "      <td>0.255</td>\n",
       "      <td>0.080</td>\n",
       "      <td>0.2050</td>\n",
       "      <td>0.0895</td>\n",
       "      <td>0.0395</td>\n",
       "      <td>0.055</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Sex  Length  Diameter  Height  Whole weight  Shucked weight  Viscera weight  \\\n",
       "0   M   0.455     0.365   0.095        0.5140          0.2245          0.1010   \n",
       "1   M   0.350     0.265   0.090        0.2255          0.0995          0.0485   \n",
       "2   F   0.530     0.420   0.135        0.6770          0.2565          0.1415   \n",
       "3   M   0.440     0.365   0.125        0.5160          0.2155          0.1140   \n",
       "4   I   0.330     0.255   0.080        0.2050          0.0895          0.0395   \n",
       "\n",
       "   Shell weight  Rings  \n",
       "0         0.150     15  \n",
       "1         0.070      7  \n",
       "2         0.210      9  \n",
       "3         0.155     10  \n",
       "4         0.055      7  "
      ]
     },
     "execution_count": 409,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "col_names = [\"Sex\", \"Length\", \"Diameter\", \"Height\", \"Whole weight\", \"Shucked weight\", \"Viscera weight\", \"Shell weight\", \"Rings\"]\n",
    "df = pd.read_csv(\"./data/abalone/abalone.data\", header=None, names=col_names)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 410,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Length</th>\n",
       "      <th>Diameter</th>\n",
       "      <th>Height</th>\n",
       "      <th>Whole weight</th>\n",
       "      <th>Shucked weight</th>\n",
       "      <th>Viscera weight</th>\n",
       "      <th>Shell weight</th>\n",
       "      <th>Rings</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>4177.000000</td>\n",
       "      <td>4177.000000</td>\n",
       "      <td>4177.000000</td>\n",
       "      <td>4177.000000</td>\n",
       "      <td>4177.000000</td>\n",
       "      <td>4177.000000</td>\n",
       "      <td>4177.000000</td>\n",
       "      <td>4177.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.523992</td>\n",
       "      <td>0.407881</td>\n",
       "      <td>0.139516</td>\n",
       "      <td>0.828742</td>\n",
       "      <td>0.359367</td>\n",
       "      <td>0.180594</td>\n",
       "      <td>0.238831</td>\n",
       "      <td>9.933684</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.120093</td>\n",
       "      <td>0.099240</td>\n",
       "      <td>0.041827</td>\n",
       "      <td>0.490389</td>\n",
       "      <td>0.221963</td>\n",
       "      <td>0.109614</td>\n",
       "      <td>0.139203</td>\n",
       "      <td>3.224169</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.075000</td>\n",
       "      <td>0.055000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.002000</td>\n",
       "      <td>0.001000</td>\n",
       "      <td>0.000500</td>\n",
       "      <td>0.001500</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>0.450000</td>\n",
       "      <td>0.350000</td>\n",
       "      <td>0.115000</td>\n",
       "      <td>0.441500</td>\n",
       "      <td>0.186000</td>\n",
       "      <td>0.093500</td>\n",
       "      <td>0.130000</td>\n",
       "      <td>8.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>0.545000</td>\n",
       "      <td>0.425000</td>\n",
       "      <td>0.140000</td>\n",
       "      <td>0.799500</td>\n",
       "      <td>0.336000</td>\n",
       "      <td>0.171000</td>\n",
       "      <td>0.234000</td>\n",
       "      <td>9.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>0.615000</td>\n",
       "      <td>0.480000</td>\n",
       "      <td>0.165000</td>\n",
       "      <td>1.153000</td>\n",
       "      <td>0.502000</td>\n",
       "      <td>0.253000</td>\n",
       "      <td>0.329000</td>\n",
       "      <td>11.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>0.815000</td>\n",
       "      <td>0.650000</td>\n",
       "      <td>1.130000</td>\n",
       "      <td>2.825500</td>\n",
       "      <td>1.488000</td>\n",
       "      <td>0.760000</td>\n",
       "      <td>1.005000</td>\n",
       "      <td>29.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Length     Diameter       Height  Whole weight  Shucked weight  \\\n",
       "count  4177.000000  4177.000000  4177.000000   4177.000000     4177.000000   \n",
       "mean      0.523992     0.407881     0.139516      0.828742        0.359367   \n",
       "std       0.120093     0.099240     0.041827      0.490389        0.221963   \n",
       "min       0.075000     0.055000     0.000000      0.002000        0.001000   \n",
       "25%       0.450000     0.350000     0.115000      0.441500        0.186000   \n",
       "50%       0.545000     0.425000     0.140000      0.799500        0.336000   \n",
       "75%       0.615000     0.480000     0.165000      1.153000        0.502000   \n",
       "max       0.815000     0.650000     1.130000      2.825500        1.488000   \n",
       "\n",
       "       Viscera weight  Shell weight        Rings  \n",
       "count     4177.000000   4177.000000  4177.000000  \n",
       "mean         0.180594      0.238831     9.933684  \n",
       "std          0.109614      0.139203     3.224169  \n",
       "min          0.000500      0.001500     1.000000  \n",
       "25%          0.093500      0.130000     8.000000  \n",
       "50%          0.171000      0.234000     9.000000  \n",
       "75%          0.253000      0.329000    11.000000  \n",
       "max          0.760000      1.005000    29.000000  "
      ]
     },
     "execution_count": 410,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fashion-MNIST"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 411,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2021-07-20 14:44:07--  https://github.com/zalandoresearch/fashion-mnist/raw/master/data/fashion/train-images-idx3-ubyte.gz\n",
      "Resolving github.com (github.com)... 140.82.121.3\n",
      "Connecting to github.com (github.com)|140.82.121.3|:443... connected.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: https://raw.githubusercontent.com/zalandoresearch/fashion-mnist/master/data/fashion/train-images-idx3-ubyte.gz [following]\n",
      "--2021-07-20 14:44:08--  https://raw.githubusercontent.com/zalandoresearch/fashion-mnist/master/data/fashion/train-images-idx3-ubyte.gz\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.111.133, 185.199.109.133, 185.199.108.133, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.111.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 26421880 (25M) [application/octet-stream]\n",
      "Saving to: ‘./data/fashion/raw/train-images-idx3-ubyte.gz.4’\n",
      "\n",
      "train-images-idx3-u 100%[===================>]  25.20M  8.72MB/s    in 2.9s    \n",
      "\n",
      "2021-07-20 14:44:11 (8.72 MB/s) - ‘./data/fashion/raw/train-images-idx3-ubyte.gz.4’ saved [26421880/26421880]\n",
      "\n",
      "--2021-07-20 14:44:11--  https://github.com/zalandoresearch/fashion-mnist/raw/master/data/fashion/train-labels-idx1-ubyte.gz\n",
      "Resolving github.com (github.com)... 140.82.121.3\n",
      "Connecting to github.com (github.com)|140.82.121.3|:443... connected.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: https://raw.githubusercontent.com/zalandoresearch/fashion-mnist/master/data/fashion/train-labels-idx1-ubyte.gz [following]\n",
      "--2021-07-20 14:44:11--  https://raw.githubusercontent.com/zalandoresearch/fashion-mnist/master/data/fashion/train-labels-idx1-ubyte.gz\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.111.133, 185.199.109.133, 185.199.108.133, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.111.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 29515 (29K) [application/octet-stream]\n",
      "Saving to: ‘./data/fashion/raw/train-labels-idx1-ubyte.gz.4’\n",
      "\n",
      "train-labels-idx1-u 100%[===================>]  28.82K  --.-KB/s    in 0.003s  \n",
      "\n",
      "2021-07-20 14:44:12 (11.0 MB/s) - ‘./data/fashion/raw/train-labels-idx1-ubyte.gz.4’ saved [29515/29515]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget -P ./data/fashion/raw/ https://github.com/zalandoresearch/fashion-mnist/raw/master/data/fashion/train-images-idx3-ubyte.gz\n",
    "!wget -P ./data/fashion/raw/ https://github.com/zalandoresearch/fashion-mnist/raw/master/data/fashion/train-labels-idx1-ubyte.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 412,
   "metadata": {},
   "outputs": [],
   "source": [
    "uci_test_size = 800\n",
    "label_col = 8\n",
    "\n",
    "uci_data = df.values.copy()\n",
    "np.random.shuffle(uci_data)\n",
    "uci_features = uci_data[:, :label_col]\n",
    "uci_labels = uci_data[:, label_col]\n",
    "\n",
    "uci_features_train = uci_features[2*uci_test_size:]\n",
    "uci_labels_train = uci_labels[2*uci_test_size:]\n",
    "\n",
    "uci_features_valid = uci_features[:uci_test_size]\n",
    "uci_labels_valid = uci_labels[:uci_test_size]\n",
    "\n",
    "uci_features_test = uci_features[uci_test_size:2*uci_test_size]\n",
    "uci_labels_test = uci_labels[uci_test_size:2*uci_test_size]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 413,
   "metadata": {},
   "outputs": [],
   "source": [
    "import urllib\n",
    "\n",
    "reader_lib_url = 'https://raw.githubusercontent.com/zalandoresearch/fashion-mnist/master/utils/mnist_reader.py'\n",
    "exec(urllib.request.urlopen(reader_lib_url).read(), globals())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 414,
   "metadata": {},
   "outputs": [],
   "source": [
    "fmnist_test_size = 2000\n",
    "\n",
    "fmnist_features, fmnist_labels = load_mnist('data/fashion/raw', kind='train')\n",
    "    \n",
    "fmnist_features_train = fmnist_features[2*fmnist_test_size:]\n",
    "fmnist_labels_train = fmnist_labels[2*fmnist_test_size:]\n",
    "\n",
    "fmnist_features_valid = fmnist_features[:fmnist_test_size]\n",
    "fmnist_labels_valid = fmnist_labels[:fmnist_test_size]\n",
    "\n",
    "fmnist_features_test = fmnist_features[fmnist_test_size:2*fmnist_test_size]\n",
    "fmnist_labels_test = fmnist_labels[fmnist_test_size:2*fmnist_test_size]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step II: Choose a Baseline Classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Choose a baseline classifier - except the neural network classifier - you have been working with over the semester and let it learn based on the **small** dataset\n",
    "* Provide some error measure or indicator whether your classifier learned, e.g. loss over multiple steps or the number of correctly classified samples on the training set or similar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "model_baseline = YourAlgorithm()\n",
    "model_baseline.learn(uci_features_train, uci_labels_train)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, I choose Logistic Regression as my Baseline Classifier \n",
    "\n",
    "P.S. the loss decrease, but does not get a good result of f1 score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 415,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "from interfaces.base_model import BaseModel\n",
    "\n",
    "def sigmoid(z):\n",
    "    return 1.0 / (1.0 + np.exp(-z))\n",
    "\n",
    "def softmax(z):\n",
    "    exps = np.exp(z)\n",
    "    return exps / np.sum(exps, axis=1, keepdims=True)\n",
    "\n",
    "\n",
    "def logistic_regression_eval(X, w):\n",
    "    return softmax(X@w)\n",
    "\n",
    "\n",
    "def logistic_regression_loss(y, y_hat):\n",
    "    m = y.shape[0]\n",
    "    loss = np.sum(-y*np.log(y_hat), axis=1)\n",
    "    return 1./m * np.sum(loss)\n",
    "\n",
    "def logistic_regression_train(X, Y, learning_rate=0.1, iteration_count=1000, batch_size=None):\n",
    "    m, n = X.shape\n",
    "    _, c = Y.shape\n",
    "    w = np.zeros((n, c))\n",
    "\n",
    "    for i in range(iteration_count):\n",
    "\n",
    "        X_chosen, Y_chosen = X, Y\n",
    "        if batch_size != None:\n",
    "            choices = np.random.choice(m, size=batch_size, replace=False)\n",
    "            X_chosen, Y_chosen = X[choices, :], Y[choices, :]\n",
    "\n",
    "        Y_hat = logistic_regression_eval(X_chosen, w)\n",
    "        gradient = X_chosen.T @ (Y_hat - Y_chosen)\n",
    "        w -= 1.0/(batch_size or m) * learning_rate * gradient\n",
    "        # print(logistic_regression_loss(Y_chosen, Y_hat))\n",
    "    return w\n",
    "\n",
    "\n",
    "def logistic_regression_predict(X, w):\n",
    "    Y_raw = logistic_regression_eval(X, w)\n",
    "    Y_label = np.argmax(Y_raw, axis=1)\n",
    "    Y_hat = (np.arange(Y_raw.shape[1]).reshape(1, -1) == Y_label.reshape(-1, 1)) * 1\n",
    "    return Y_hat\n",
    "\n",
    "\n",
    "class LogisticRegression (BaseModel):\n",
    "\n",
    "    def learn(self, X, Y, learning_rate=0.1, iteration_count=1000, batch_size=None):\n",
    "        self.w = logistic_regression_train(X, Y, learning_rate=learning_rate, iteration_count=iteration_count, batch_size=batch_size)\n",
    "\n",
    "    def infer(self, X):\n",
    "        return logistic_regression_predict(X, self.w)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 416,
   "metadata": {},
   "outputs": [],
   "source": [
    "baseline_model = LogisticRegression()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 417,
   "metadata": {},
   "outputs": [],
   "source": [
    "def features_to_vec(raw_X):\n",
    "    X = raw_X.copy()\n",
    "    column0_classes = np.unique(uci_features[:, 0])\n",
    "    X0_indicator = column0_classes.reshape(1, -1) == X[:, 0].reshape(-1, 1)\n",
    "    X[:, 0] = X0_indicator @ np.arange(column0_classes.shape[0])\n",
    "    return X.astype(float)\n",
    "\n",
    "def label_to_vec(raw_Y):\n",
    "    Y = raw_Y.copy()\n",
    "    Y_classes = np.unique(uci_labels)\n",
    "    Y_indicator = (Y_classes.reshape(1, -1) == Y.reshape(-1, 1)) * 1\n",
    "    return Y_indicator\n",
    "\n",
    "def z_score_normalizer(of_X): # m*n\n",
    "    mean = np.mean(of_X, axis=0)\n",
    "    scale_range = np.std(of_X, axis=0)\n",
    "    return lambda X: (X-mean) / scale_range"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 418,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_uci_train_raw = features_to_vec(uci_features_train)\n",
    "Y_uci_train = label_to_vec(uci_labels_train)\n",
    "\n",
    "normalizer =  z_score_normalizer(X_uci_train_raw)\n",
    "X_uci_train_normalized = normalizer(X_uci_train_raw)\n",
    "X_uci_train = np.insert(X_uci_train_normalized, 0, 1, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 419,
   "metadata": {},
   "outputs": [],
   "source": [
    "baseline_model.learn(X_uci_train, Y_uci_train, learning_rate=3, iteration_count=1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 420,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.        , 1.        , 0.        , 0.51282051, 0.1038961 ,\n",
       "       0.43361345, 0.01568627, 0.33948339, 0.29757785, 0.34173669,\n",
       "       0.        , 0.        , 0.09210526, 0.06122449, 0.04      ,\n",
       "       0.25531915, 0.09302326, 0.        , 0.        , 0.1       ,\n",
       "       0.23529412, 0.        , 0.        , 0.        , 0.        ,\n",
       "       0.        , 0.        , 1.        ])"
      ]
     },
     "execution_count": 420,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import f1_score\n",
    "\n",
    "Y_uci_train_hat = baseline_model.infer(X_uci_train)\n",
    "f1_score(Y_uci_train, Y_uci_train_hat, average=None, zero_division=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step III: Provide Evaluation Metrics for the Classifier Interface"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Given the class interface for machine learning models, use the predicted target from the result of an model.infer()-invocation to calculate precision, recall and f1-score given the actual test-set targets.\n",
    "* Do not use scikit-learn or similar libraries; but you can orientate on such interfaces or implementations.\n",
    "* Note, that a model can return two or multiple classes based on the problem it learned."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "baseline_predicted = model_baseline.infer(features_test)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> https://en.wikipedia.org/wiki/Precision_and_recall\n",
    "\n",
    "$precision = \\frac{\\text{true positives}}{\\text{true positives} + \\text{false positives}}$\n",
    "\n",
    "$recall = \\frac{\\text{true positives}}{\\text{true positives} + \\text{false negatives}}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing your implementation\n",
    "You can use below vectors as a reference for testing the output of infer() and the target vector of a 10-class-classifier. The *f1_score* method of scikit learn gives you a reference on how the values need to look like. Using the function is of course not a valid solution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 421,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_uci_valid_raw = features_to_vec(uci_features_valid)\n",
    "X_uci_valid_normalized = normalizer(X_uci_valid_raw)\n",
    "X_uci_valid = np.insert(X_uci_valid_normalized, 0, 1, axis=1)\n",
    "\n",
    "Y_uci_valid = label_to_vec(uci_labels_valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 422,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y_uci_valid_predicted = baseline_model.infer(X_uci_valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 436,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1.        , 1.        , 0.        , 0.27586207, 0.12903226,\n",
       "       0.34146341, 0.02816901, 0.35471698, 0.31386861, 0.31944444,\n",
       "       0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "       0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "       0.        , 1.        , 0.        , 1.        , 1.        ,\n",
       "       1.        , 0.        , 1.        ])"
      ]
     },
     "execution_count": 436,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import f1_score\n",
    "\n",
    "# from sklearn.metrics import f1_score\n",
    "f1_score(Y_uci_valid, Y_uci_valid_predicted, average=None, zero_division=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step IV: Experiment (1) Hyperparameter Choice of Baseline Classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Use one fixed train-validation-test split.\n",
    "* Choose a hyperparameter of your baseline classifier.\n",
    "* Conduct a grid search to find the best suitable value for it. Let the classifier learn on the training set and use an evaluation metric on the validation set (not the test set!) to find out which hyperparameter value works best for your classifier on the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "possible_hp_values = np.arange(1, 10, 0.1)\n",
    "best_hp_value = None\n",
    "best_f1_score = -np.infty\n",
    "for hp_value in possible_hp_values:\n",
    "    # 1. create a baseline classifier object with hp_value specified\n",
    "    current_model = YourAlgorithm(hyperparam=hp_value)\n",
    "    \n",
    "    # 2. learn the classifier on the training set\n",
    "    current_model.learn(uci_features_train, uci_labels_train)\n",
    "    \n",
    "    # 3. evaluate the model on the validation set\n",
    "    prediction = current_model.infer(uci_features_valid)\n",
    "    \n",
    "    f1_score = compute_f1(uci_labels_valid, prediction)\n",
    "    if f1_score > best_f1_score:\n",
    "        best_f1_score = f1_score\n",
    "        best_hp_value = hp_value\n",
    "\n",
    "print(\"Found hyperparameter value\", best_hp_value)\n",
    "print(\"Best f1-score on validation set\", best_f1_score)\n",
    "\n",
    "test_model = YourAlgorithm(hyperparam=best_hp_value)\n",
    "test_model.learn(uci_features_train, uci_labels_train)\n",
    "prediction = test_model.infer(uci_features_test)\n",
    "test_f1_score = compute_f1(uci_labels_test, prediction)\n",
    "print(\"F1-Score on test set\", test_f1_score)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step V: Use a Neural Network Classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Let a neural network learn on the training set and report its evaluation metric on the **validation** set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 449,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def ReLU(z):\n",
    "    return max(0, z)\n",
    "\n",
    "def ReLU_gradient(z):\n",
    "    return 1 if z > 0 else 0\n",
    "\n",
    "def sigmoid(z):\n",
    "    return 1 / (1 + np.exp(-z))\n",
    "\n",
    "def sigmoid_gradient(z):\n",
    "    return sigmoid(z) * (1-sigmoid(z))\n",
    "\n",
    "def softmax(z):\n",
    "    exps = np.exp(z)\n",
    "    return exps / np.sum(exps, axis=1, keepdims=True)\n",
    "\n",
    "def softmax_gradient(z):\n",
    "    return softmax(z)*(1-softmax(z))\n",
    "\n",
    "def tanh_gradient(z):\n",
    "    return 1 - np.tanh(z)**2\n",
    "\n",
    "\n",
    "def neural_network_loss(Y, Y_hat):\n",
    "    m = Y.shape[0]\n",
    "    err = -np.sum(Y*np.log(Y_hat))\n",
    "    J = 1./m * err\n",
    "    return J\n",
    "\n",
    "\n",
    "def neural_network_forward_propagation(X, Ws, bs, activation_functions):\n",
    "    no_layers = len(Ws)\n",
    "    Zs, As = [None]*no_layers, [None]*no_layers\n",
    "    Zs[0] = X@Ws[0] + bs[0]\n",
    "    As[0] = activation_functions[0](Zs[0])\n",
    "\n",
    "    for i in range(1, no_layers):\n",
    "        Zs[i] = As[i-1]@Ws[i] + bs[i]\n",
    "        As[i] = activation_functions[i](Zs[i])\n",
    "    return Zs, As\n",
    "\n",
    "\n",
    "def neural_network_backward_propagation(X, Y, Ws, bs, Zs, As, activation_gradient_functions):\n",
    "    no_layers = len(activation_gradient_functions)\n",
    "    n, m = X.shape\n",
    "    dZs, dWs, dbs = [None]*no_layers, [None]*no_layers, [None]*no_layers\n",
    "\n",
    "    current_layer = no_layers - 1\n",
    "    dZs[current_layer] = As[current_layer] - Y\n",
    "    dWs[current_layer] = 1./m * As[current_layer - 1].T @ dZs[current_layer]\n",
    "    dbs[current_layer] = 1./m * np.sum(dZs[current_layer], axis=0)\n",
    "\n",
    "    for i in range(1, no_layers - 1):\n",
    "        current_layer = no_layers - 1 - i\n",
    "        dZs[current_layer] = dZs[current_layer + 1] @ Ws[current_layer + 1].T \\\n",
    "                                * activation_gradient_functions[current_layer](Zs[current_layer])\n",
    "        dWs[current_layer] = 1./m * As[current_layer - 1].T @ dZs[current_layer]\n",
    "        dbs[current_layer] = 1./m * np.sum(dZs[current_layer], axis=0)\n",
    "\n",
    "    dZs[0] = dZs[1] @ Ws[1].T * activation_gradient_functions[0](Zs[0])\n",
    "    dWs[0] = 1./m * X.T @ dZs[0]\n",
    "    dbs[0] = 1./m * np.sum(dZs[0], axis=0)\n",
    "\n",
    "    return dZs, dWs, dbs\n",
    "\n",
    "\n",
    "def neural_network_train(X, Y, layers, batch_size=None, iteration_count=1000, learning_rate=0.1):\n",
    "    m, n = X.shape\n",
    "    no_layers = len(layers)\n",
    "    activation_functions = [l[1] for l in layers]\n",
    "    activation_gradient_functions = [l[2] for l in layers]\n",
    "\n",
    "    # initialize the parameter\n",
    "    Ws, bs = [None]*no_layers, [None]*no_layers\n",
    "    no_hidden_units, _, _ = layers[0]\n",
    "    Ws[0] = np.random.randn(n, no_hidden_units)\n",
    "    bs[0] = np.zeros((1, no_hidden_units))\n",
    "    for i in range(1, no_layers):\n",
    "        no_hidden_units, _, _ = layers[i]\n",
    "        no_prev_hidden_units, _, _ = layers[i-1]\n",
    "        Ws[i] = np.random.randn(no_prev_hidden_units, no_hidden_units)\n",
    "        bs[i] = np.zeros((1, no_hidden_units))\n",
    "\n",
    "    # gradient descent\n",
    "    for i in range(iteration_count):\n",
    "\n",
    "        X_chosen, Y_chosen = X, Y\n",
    "        if batch_size != None:\n",
    "            choices = np.random.choice(m, size=batch_size, replace=False)\n",
    "            X_chosen, Y_chosen = X[choices, :], Y[choices, :]\n",
    "\n",
    "        Zs, As = neural_network_forward_propagation(X_chosen, Ws, bs, activation_functions)\n",
    "        dZs, dWs, dbs = neural_network_backward_propagation(X_chosen, Y_chosen, Ws, bs, Zs, As, activation_gradient_functions)\n",
    "\n",
    "        for i in range(len(Ws)):\n",
    "            Ws[i] -= learning_rate * dWs[i]\n",
    "            bs[i] -= learning_rate * dbs[i]\n",
    "        # print('loss', neural_network_loss(Y_chosen, As[no_layers - 1]))\n",
    "    return Ws, bs\n",
    "\n",
    "\n",
    "def neural_network_predict(X, Ws, bs, layers):\n",
    "    activation_functions = [l[1] for l in layers]\n",
    "    _, As = neural_network_forward_propagation(X, Ws, bs, activation_functions)\n",
    "    return (As[-1] > 0.5) * 1\n",
    "\n",
    "\n",
    "class NeuralNetwork (BaseModel):\n",
    "    def learn(self, X, Y, layers, learning_rate=0.1, iteration_count=1000, batch_size=None):\n",
    "        self.layers = layers\n",
    "        Ws, bs = neural_network_train(X, Y, layers, learning_rate=learning_rate, iteration_count=iteration_count, batch_size=batch_size)\n",
    "        self.Ws = Ws\n",
    "        self.bs = bs\n",
    "\n",
    "    def infer(self, X):\n",
    "        return neural_network_predict(X, self.Ws, self.bs, self.layers)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NN for uci dataset\n",
    "\n",
    "the loss decrease, but does not get a good result in f1 score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 450,
   "metadata": {},
   "outputs": [],
   "source": [
    "nn_model = NeuralNetwork()\n",
    "\n",
    "layers_uci = [(4, np.tanh, tanh_gradient), (28, softmax, softmax_gradient)]\n",
    "np.random.seed(0)\n",
    "\n",
    "nn_model.learn(X_uci_train, Y_uci_train, layers_uci, learning_rate=0.01, iteration_count=1000)\n",
    "Y_uci_valid_hat = nn_model.infer(X_uci_valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 451,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 1., 0., 1., 1., 1., 1., 1.])"
      ]
     },
     "execution_count": 451,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f1_score(Y_uci_valid, Y_uci_valid_hat, average=None, zero_division=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NN for fmnist dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 442,
   "metadata": {},
   "outputs": [],
   "source": [
    "def lable_to_vec_fmnist(raw_Y):\n",
    "    Y = raw_Y.copy()\n",
    "    Y_classes = np.unique(fmnist_labels)\n",
    "    Y_indicator = (Y_classes.reshape(1, -1) == Y.reshape(-1, 1)) * 1\n",
    "    return Y_indicator\n",
    "\n",
    "normalizer =  z_score_normalizer(fmnist_features_train)\n",
    "X_fmnist_train = normalizer(fmnist_features_train)\n",
    "X_fmnist_test = normalizer(fmnist_features_test)\n",
    "X_fmnist_valid = normalizer(fmnist_features_valid)\n",
    "\n",
    "Y_fmnist_train = lable_to_vec_fmnist(fmnist_labels_train)\n",
    "Y_fmnist_test = lable_to_vec_fmnist(fmnist_labels_test)\n",
    "Y_fmnist_valid = lable_to_vec_fmnist(fmnist_labels_valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 452,
   "metadata": {},
   "outputs": [],
   "source": [
    "layers = [(10, np.tanh, tanh_gradient), (10, softmax, softmax_gradient)]\n",
    "np.random.seed(0)\n",
    "\n",
    "Ws_fmnist, bs_fmnist = neural_network_train(X_fmnist_train, Y_fmnist_train, layers, learning_rate=0.03, iteration_count=1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 453,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.74151436, 0.95508274, 0.64739884, 0.81643836, 0.71060172,\n",
       "       0.85213033, 0.37090909, 0.89552239, 0.88831169, 0.88943489])"
      ]
     },
     "execution_count": 453,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y_fmnist_valid_hat = neural_network_predict(X_fmnist_valid, Ws_fmnist, bs_fmnist, layers)\n",
    "f1_score(Y_fmnist_valid, Y_fmnist_valid_hat, average=None, zero_division=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step VI: Experiment (2) Hyperparameter Choice of Neural Net"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- choose a hyperparameter of your neural net, e.g. the number of neurons in the first hidden layer or the learning rate for SGD\n",
    "- (iteratively) create models for each hyperparameter setting, e.g. the number of neurons h=10,20,30,40,50,60,70,80,90,100\n",
    "- train the neural net on the train set and evaluate it over your validation set\n",
    "- keep all models with each hyperparameter setting and determine which is the best performing model on the validation set\n",
    "- evaluate them also on the test set. is the best model on the validation set with its hyperparameter also the best model on the test set?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step VII: Experiment (3) Neural Net Stability on Shuffled Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- over multiple runs $r \\geq 5, r\\in\\mathbb{N}$ randomly shuffle your training data\n",
    "- split it into a train-test, e.g. 90% of the data is for training, 10% for testing\n",
    "- what are the mean and standard deviation of your models over multiple runs?\n",
    "- plot a boxplot with matplotlib/seaborn of the stability of your model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 441,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "\n",
    "def get_k_folds(n, k, random=True):\n",
    "    fold_size = n//k\n",
    "    m = fold_size*k\n",
    "\n",
    "    indices = np.random.permutation(m) if random else np.arange(m)\n",
    "    indices_splits = indices.reshape(k, -1)\n",
    "\n",
    "    folds = []\n",
    "    fold_indices = np.arange(k)\n",
    "    for i in range(k):\n",
    "        train_indices = indices_splits[fold_indices != i].flatten()\n",
    "        test_indices = indices_splits[fold_indices == i].flatten()\n",
    "        folds.append((train_indices, test_indices))\n",
    "\n",
    "    return folds\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "example_results = np.minimum(np.random.normal(0.8, 0.1, (100,)), 1)\n",
    "sns.boxplot(data=example_results)\n",
    "plt.title(\"Stability of My Model over 100 runs on test set\")\n",
    "plt.ylabel(\"Accuracy on test set\")\n",
    "plt.xlabel(\"My Model\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step VIII: Bonus: implement Momentum SGD / ADAM / .."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- this task is optional if you have time at the end\n",
    "- inspect your stochastic gradient descent implementation\n",
    "- have a look at online examples such as [wiseodd.github.com](https://wiseodd.github.io/techblog/2016/06/22/nn-optimization/) for implementations of variants on stochastic gradient such as with Nesterov Momentum or ADAM\n",
    "- change your implementation of SGD to one or multiple of these variants and try a simple run of your neural net and compare it with previous results\n",
    "- sketch a first design of an optimizer-class which is fed with parameters of your model and performs the update step "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}