HW3_511212133.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ac7744d7",
   "metadata": {},
   "source": [
    "Şıhmehmet Yıldız\n",
    "\n",
    "$511212133$\n",
    "\n",
    "yildizsih@itu.edu.tr\n",
    "\n",
    "MAT555E - Homework 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "2c043675",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from scipy.io.matlab import loadmat\n",
    "from scipy.optimize import linear_sum_assignment"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9de11cfc",
   "metadata": {},
   "source": [
    "# HW3\n",
    "\n",
    "In this homework we are going to look at a hyperspectral image dataset called [Salinas](https://rslab.ut.ac.ir/data) dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "b807fa36",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(512, 217, 204)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "salinas = loadmat('data/Salinas_corrected.mat')['salinas_corrected']\n",
    "salinas.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "fe4b1aaf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(512, 217)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "salinas_gt = loadmat('data/Salinas_gt.mat')['salinas_gt']\n",
    "salinas_gt.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3263f646",
   "metadata": {},
   "source": [
    "## Q1 \n",
    "\n",
    "1. Reshape and split the data into train (80%) and test (20%) data sets.\n",
    "2. Apply all of the supervised ML algorithms we learned on the data and compare the results."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df2d57b1",
   "metadata": {},
   "source": [
    "## Answer 1,"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6ebd735",
   "metadata": {},
   "source": [
    "Import the necessary python packages for question 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "548cd6cd",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import confusion_matrix, accuracy_score\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.cluster import KMeans, DBSCAN, AgglomerativeClustering\n",
    "from scipy.stats import chisquare\n",
    "import time"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad991ff9",
   "metadata": {},
   "source": [
    "Upload the dataset and reshape it to the appropriate format for use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "7a84fba0",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = salinas.reshape((512*217,204))\n",
    "y = salinas_gt.reshape(512*217)\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X,y,train_size=0.80)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "abdde6b5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16],\n",
       "      dtype=uint8)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.unique(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c66b040",
   "metadata": {},
   "source": [
    "### K-Nearest Neighbors (K-NN)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "6e6ceb36",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training time is  2.607513189315796\n"
     ]
    }
   ],
   "source": [
    "model = KNeighborsClassifier(n_neighbors=20,metric='minkowski')\n",
    "t1 = time.time()\n",
    "model.fit(X_train,y_train)\n",
    "t2 = time.time()\n",
    "print('Training time is ',t2-t1)\n",
    "predicted = model.predict(X_test)\n",
    "cm = confusion_matrix(y_test,predicted)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "e79c3b22",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[10350,    68,   115,    51,    32,    87,    88,    65,   101,\n",
       "           68,    85,    40,    49,    27,    37,    39,    36],\n",
       "       [   11,   389,     4,     0,     0,     0,     0,     0,     0,\n",
       "            0,     0,     0,     0,     0,     0,     0,     0],\n",
       "       [    9,     0,   693,     0,     0,     0,     0,     0,     0,\n",
       "            0,     0,     0,     0,     0,     0,     0,     0],\n",
       "       [   77,     0,     0,   353,     0,     2,     0,     0,     0,\n",
       "            0,     0,     0,     0,     0,     0,     0,     0],\n",
       "       [   39,     0,     0,     0,   260,     1,     0,     0,     0,\n",
       "            0,     0,     0,     0,     0,     0,     0,     0],\n",
       "       [   23,     0,     0,     5,     2,   474,     0,     0,     0,\n",
       "            0,     0,     0,     0,     0,     0,     0,     0],\n",
       "       [   10,     0,     0,     0,     0,     0,   796,     0,     0,\n",
       "            0,     0,     0,     0,     0,     0,     0,     0],\n",
       "       [    9,     0,     0,     0,     0,     0,     0,   726,     0,\n",
       "            0,     0,     0,     0,     0,     0,     0,     0],\n",
       "       [   53,     0,     0,     0,     0,     0,     0,     0,  1979,\n",
       "            0,    36,     0,     0,     0,     1,   227,     0],\n",
       "       [  112,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "         1129,     0,     0,     0,     0,     0,     0,     0],\n",
       "       [   60,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "            3,   590,     2,     6,     0,     0,     0,     0],\n",
       "       [   22,     0,     0,     1,     0,     0,     0,     0,     0,\n",
       "            0,     0,   178,     1,     0,     0,     0,     0],\n",
       "       [   17,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "            0,     0,     0,   391,     0,     0,     0,     0],\n",
       "       [   25,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "            0,     0,     0,     0,   152,     0,     0,     0],\n",
       "       [   35,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "            0,     0,     0,     0,     3,   171,     0,     0],\n",
       "       [   20,     0,     0,     0,     0,     0,     0,     0,   494,\n",
       "            0,    11,     0,     0,     0,     0,   929,     0],\n",
       "       [    8,     0,     2,     0,     0,     0,     0,     0,     0,\n",
       "            0,     1,     0,     0,     0,     0,     0,   341]])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rows, cols = linear_sum_assignment(cm,maximize=True)\n",
    "cm = cm[rows,:]\n",
    "cm = cm[:,cols]\n",
    "cm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "3b73a0f4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8955942576841727"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.score(X_test,y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "4e75d9bc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Power_divergenceResult(statistic=1492769.7641870303, pvalue=0.0)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chisquare(cm,axis=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e855b41",
   "metadata": {},
   "source": [
    "### Support Vector Machine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9179acc2",
   "metadata": {},
   "outputs": [],
   "source": [
    "#SVM\n",
    "#Create the SVM model\n",
    "from sklearn.svm import SVC\n",
    "#svc_model = SVC(kernel = 'linear', random_state = 0)\n",
    "svc_model = SVC(C=12.0,\n",
    "             kernel='rbf',\n",
    "             max_iter=1000)\n",
    "#Fit the model for the data\n",
    "t1 = time.time()\n",
    "svc_model.fit(X_train, y_train)\n",
    "t2 = time.time()\n",
    "print('Training time is ',t2-t1)\n",
    "#Make the prediction\n",
    "predicted = svc_model.predict(X_test)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "0c49c1f3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.44845471012454574\n",
      "0.4465145583007065\n",
      "0.4465145583007065\n",
      "[[1673   82  111 3952  434  442  179   77  191 2930  614   62  166  122\n",
      "   104  140   41]\n",
      " [   2  401    9    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0]\n",
      " [  11    0  730    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0]\n",
      " [   0    0    0  405    0    1    0    0    0    1    0    0    0    0\n",
      "     0    0    0]\n",
      " [  51    0    0    0  212    2    0    0    0    0    0    0    0    0\n",
      "     0    0    0]\n",
      " [   5    0    0    7    1  548    1    0    0    1    1    0    0    0\n",
      "     0    0    0]\n",
      " [   3    0    0    0    0    0  785    0    0    0    0    0    0    0\n",
      "     0    0    0]\n",
      " [   9    0    0    0    0    0    0  710    0    0    0    0    0    0\n",
      "     0    0    0]\n",
      " [  11    0    0    0    0    0    0    0   32    4   25    0    0    0\n",
      "     0 2189    0]\n",
      " [   0    0    0    0    0    0    0    0    1 1244    0    0    0    0\n",
      "     0    0    0]\n",
      " [  15    0    0   12    0    0    0    0    3   17  620    0    2    0\n",
      "     0    0    0]\n",
      " [   0    0    0    9    0    0    0    0    0  150   45   12    5    0\n",
      "     0    0    0]\n",
      " [   0    0    0    1    0    0    0    0    0    0    0    0  381    0\n",
      "     0    0    0]\n",
      " [   2    0    0    0    0    0    0    0    0    0    0    0    0  176\n",
      "     2    0    0]\n",
      " [  14    0    0    0    0    0    0    0    0    0    2    0    0    7\n",
      "   199    0    0]\n",
      " [   4    0    0    2    0    0    0    0    4    0   14    0    0    0\n",
      "     0 1434    0]\n",
      " [   3    0    0    0    0    0    0    0    0    0    4    0    0    0\n",
      "     0    0  360]]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Power_divergenceResult(statistic=490387.628999595, pvalue=0.0)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm = confusion_matrix(y_test,predicted)\n",
    "print(cm)\n",
    "print(svc_model.score(X_train, y_train))\n",
    "print(svc_model.score(X_test, y_test))\n",
    "print(accuracy_score(predicted,y_test))\n",
    "print(chisquare(cm,axis=None))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9eb827eb",
   "metadata": {},
   "source": [
    "SVM with linear model,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "871639c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.svm import LinearSVC\n",
    "Lsvc_model = LinearSVC(multi_class='ovr',max_iter=3500)\n",
    "Lsvc_model.fit(X_train,y_train)\n",
    "predicted = Lsvc_model.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c7ae98d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "cm = confusion_matrix(y_test,predicted)\n",
    "chisquare(cm,axis=None)\n",
    "print(cm)\n",
    "print(Lsvc_model.score(X_train, y_train))\n",
    "print(Lsvc_model.score(X_test, y_test))\n",
    "print(accuracy_score(predicted,y_test))\n",
    "print(chisquare)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3e942e2",
   "metadata": {},
   "source": [
    "### Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3a81d575",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Logistic Regression\n",
    "from sklearn.linear_model import LogisticRegression, LogisticRegressionCV\n",
    "\n",
    "LR_model = LogisticRegression(C=100,\n",
    "                           tol=1e-1,\n",
    "                           penalty='l2',\n",
    "                           solver='newton-cg',\n",
    "                           fit_intercept=False,\n",
    "                           max_iter=8500)\n",
    "t1= time.time()\n",
    "LR_model.fit(X_train,y_train)\n",
    "t2 = time.time()\n",
    "print('Training time is ',t2-t1)\n",
    "\n",
    "predicted = LR_model.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b3ae25b0",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(LR_model.score(X_train, y_train))\n",
    "print(LR_model.score(X_test, y_test))\n",
    "print(accuracy_score(predicted,y_test))\n",
    "cm = confusion_matrix(y_test,predicted)\n",
    "print(cm)\n",
    "chisquare(cm,axis=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f578abae",
   "metadata": {},
   "source": [
    "### Multinomial Regression,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6ba38e01",
   "metadata": {},
   "outputs": [],
   "source": [
    "MR_model = LogisticRegression(solver='lbfgs',multi_class='multinomial')\n",
    "t1 = time.time()\n",
    "MR_model.fit(train_x, train_y)\n",
    "t2 = time.time()\n",
    "print('Training time is ',t2-t1)\n",
    "MR_model.score(test_x, test_y)\n",
    "\n",
    "predicted = MR_model.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a0210995",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(MR_model.score(X_train, y_train))\n",
    "print(MR_model.score(X_test, y_test))\n",
    "print(accuracy_score(predicted,y_test))\n",
    "cm = confusion_matrix(y_test,predicted)\n",
    "print(cm)\n",
    "chisquare(cm,axis=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9769df7e",
   "metadata": {},
   "source": [
    "### Linear Discriminant Analysis,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "24a7f30b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.rcParams['figure.figsize'] = (9,9)\n",
    "\n",
    "LDA_model = LDA(n_components=2)\n",
    "t1 = time.time()\n",
    "data = LDA_model.fit_transform(X_train, y_train)\n",
    "t2 = time.time()\n",
    "print('Training time is ',t2-t1)\n",
    "plt.rcParams['figure.figsize'] = (9,9)\n",
    "plt.scatter(data[:,0],data[:,1],c=y_train,cmap='jet',s=100,alpha=0.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "0c43d255",
   "metadata": {},
   "outputs": [],
   "source": [
    "predicted = LDA_model.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "9d96ebaa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8109537256843266\n",
      "0.813194725709914\n",
      "0.813194725709914\n",
      "[[8585  104  133  398  199  482  130  124   64  409  286   91   93   95\n",
      "    74  105   70]\n",
      " [   0  390    5    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0]\n",
      " [   3    1  700    0    0    0    0    1    0    0    0    0    0    0\n",
      "     1    0    0]\n",
      " [  16    0    0  356    0    9    0    0    0    0    0    0    0    0\n",
      "     0    0    0]\n",
      " [   5    0    0    0  264   30    0    0    0    0    0    0    0    0\n",
      "     0    0    0]\n",
      " [   3    0    0    2    8  524    0    0    0    0    0    0    0    0\n",
      "     0    0    0]\n",
      " [   6    0    0    0    0    2  793    0    0    0    0    0    0    0\n",
      "     0    0    0]\n",
      " [   2    0    0    0    0    0    0  762    0    0    0    0    0    0\n",
      "     0    0    0]\n",
      " [  31    0    0    0    0    0    0    0 1943    0   10    0    0    0\n",
      "     0  314    0]\n",
      " [ 104    0    0    0    0    0    0    0    0 1112    0    0    1    0\n",
      "     0    0    0]\n",
      " [  53    0    0    0    0    0    0    0    0    5  575    3    0    1\n",
      "     2    3    0]\n",
      " [  15    0    0    0    0    0    0    0    0    2    0  181    1    0\n",
      "     0    0    0]\n",
      " [  70    0    0    0    0    0    0    0    0    7    0    0  287    0\n",
      "     0    0    0]\n",
      " [   0    0    0    0    0    0    0    0    0    0    0    0    0  176\n",
      "     2    0    0]\n",
      " [   7    0    0    0    0    0    0    0    0    0    0    1    0    8\n",
      "   171    0    0]\n",
      " [  12    0    0    0    0    4    0    0  526    0    0    0    0    0\n",
      "     0  886    0]\n",
      " [   6    0    0    0    0    0    0    0    0    0    4    0    0    0\n",
      "     8    0  365]]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Power_divergenceResult(statistic=1065699.945907025, pvalue=0.0)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(LDA_model.score(X_train, y_train))\n",
    "print(LDA_model.score(X_test, y_test))\n",
    "print(accuracy_score(predicted,y_test))\n",
    "cm = confusion_matrix(y_test,predicted)\n",
    "print(cm)\n",
    "chisquare(cm,axis=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aab0bba5",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "1. [Geeksforgeeks - SVM](https://www.geeksforgeeks.org/support-vector-machine-algorithm/)\n",
    "2. [MAT-555E Lecture Notes](https://github.com/kaygun/2022-MAT555E)\n",
    "3. [Scikit-Learn](https://scikit-learn.org/stable/)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3007fa6",
   "metadata": {},
   "source": [
    "## Q2\n",
    "\n",
    "Now, apply all of the unsupervised algorithms we have learned so far and compare the results. In order to align the unsupervised labels and true labels use the `linear_sum_assignment` function from `scipy.optimize`. For example, if a confusion matrix looks like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "8973877b",
   "metadata": {},
   "outputs": [],
   "source": [
    "cm = np.array(\n",
    "      [[1,10,2],\n",
    "       [9,2,1],\n",
    "       [3,4,10]])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3a24d62",
   "metadata": {},
   "source": [
    "we get"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "448f1fd2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[10,  1,  2],\n",
       "       [ 2,  9,  1],\n",
       "       [ 4,  3, 10]])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rows, cols = linear_sum_assignment(cm,maximize=True)\n",
    "cm = cm[rows,:]\n",
    "cm = cm[:,cols]\n",
    "cm"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c720c444",
   "metadata": {},
   "source": [
    "## Answer 2,"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "842d24b0",
   "metadata": {},
   "source": [
    "Unsupervised machine algortihms we have learned are,\n",
    "\n",
    "* K-means\n",
    "\n",
    "* Hierarchical Cluster\n",
    "\n",
    "* DSCAN \n",
    "\n",
    "The application of these 3 methods for the data set, respectively, is given below."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7382a407",
   "metadata": {},
   "source": [
    "Import the necessary python packages for question 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "91959db5",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from scipy.io.matlab import loadmat\n",
    "from scipy.optimize import linear_sum_assignment\n",
    "from sklearn.cluster import KMeans, DBSCAN, AgglomerativeClustering\n",
    "from sklearn.metrics import confusion_matrix, accuracy_score\n",
    "from scipy.stats import chisquare\n",
    "import time"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9208fadd",
   "metadata": {},
   "source": [
    "Upload the dataset and reshape it to the appropriate format for use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "0b540220",
   "metadata": {},
   "outputs": [],
   "source": [
    "salinas = loadmat('data/Salinas_corrected.mat')['salinas_corrected']\n",
    "salinas.shape\n",
    "salinas_gt = loadmat('data/Salinas_gt.mat')['salinas_gt']\n",
    "salinas_gt.shape\n",
    "X = salinas.reshape((512*217,204))\n",
    "y = salinas_gt.reshape(512*217)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b97ab741",
   "metadata": {},
   "source": [
    "### K-means\n",
    "\n",
    "K-means is unsupervised clustering algorithm using K number of clusters. The algorithm assigns data points to one of the clusters depending on lenght of the cluster center.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "122da583",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training time is  10.78682804107666\n"
     ]
    }
   ],
   "source": [
    "#K-means\n",
    "k = len(np.unique(y)) # define cluster number\n",
    "Kmeans_model = KMeans(n_clusters=k) #create model\n",
    "t1 = time.time()\n",
    "predicted = Kmeans_model.fit_predict(X) # train model\n",
    "t2 = time.time()\n",
    "\n",
    "print('Training time is ', t2-t1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "714b967c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[12177   697  2792   902  4109  9543   604   652   789 13109  2732  5366\n",
      "     66  1588   884   666   299]\n",
      " [    0  2004     5     0     0     0     0     0     0     0     0     0\n",
      "      0     0     0     0     0]\n",
      " [    0  2136  1475     0     0     0     0    96     2     0     0     0\n",
      "      0     0    12     0     5]\n",
      " [ 1420     0     0     0     2   316     0     0     0   238     0     0\n",
      "      0     0     0     0     0]\n",
      " [    0     0     0     0  1384    10     0     0     0     0     0     0\n",
      "      0     0     0     0     0]\n",
      " [   98     0     0     1    31  2543     0     0     0     0     0     5\n",
      "      0     0     0     0     0]\n",
      " [    1     0     0     0     0     0  3747     0     0     3    91    95\n",
      "      0     2    20     0     0]\n",
      " [    1     0     6     0     0     1     0  3471     6     0     0     0\n",
      "      0     3     0     0    91]\n",
      " [    1     0     0     0     1     0     0     6  7670    10     9    64\n",
      "      0   389    12  2971   138]\n",
      " [    0     0     0     0     0     0     0     0     0  5606     2   586\n",
      "      0     9     0     0     0]\n",
      " [  122     0     0     0     3    14     0     1    87  1578    16   575\n",
      "      0   845    32     5     0]\n",
      " [   41     0     0     0     0     3     0     0     0   226     0   798\n",
      "      0     0     0     0     0]\n",
      " [ 1877     0     0     0     0     5     0     0     0    37     0     0\n",
      "      0     8     0     0     0]\n",
      " [    0     0     0     0     0     0     0     0     0     0     0     0\n",
      "      0   903    13     0     0]\n",
      " [    0     0     0     0     0     0     0     0    30     1     0     6\n",
      "      0    78   951     3     1]\n",
      " [   16     0     0     0     3    22     0     0  3460    13     0     0\n",
      "      0   122     1  3570    61]\n",
      " [    6     0     0     0    40     6     0   255   323     4     0     0\n",
      "      0     1     0   322   850]]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Power_divergenceResult(statistic=1608165.94281034, pvalue=0.0)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm = confusion_matrix(y,predicted) #calculate confusion matrix\n",
    "rows, cols = linear_sum_assignment(cm,maximize=True)\n",
    "cm = cm[rows,:]\n",
    "cm = cm[:,cols]\n",
    "print(cm)\n",
    "chisquare(cm,axis=None) #chisquare test"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2908e666",
   "metadata": {},
   "source": [
    "### Hierarchical Clustering\n",
    "Hierarchical clustering is an unsupervised learning method based on hierarchical ordering. The hierarchical clustering can be classified into the two different type of clustering:\n",
    "\n",
    "* Agglomerative (bottom-up) hierarchical clustering\n",
    "* Divisive (top-down) hierarchical clustering\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "51fa8e13",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training time is  1663.1189179420471\n"
     ]
    }
   ],
   "source": [
    "# Hierarchical Cluster\n",
    "k = len(np.unique(y))\n",
    "HC_model = AgglomerativeClustering(n_clusters=k, \n",
    "                                linkage=\"complete\", \n",
    "                                affinity='manhattan')\n",
    "\n",
    "t1 = time.time()\n",
    "predicted = HC_model.fit_predict(X)\n",
    "t2 = time.time()\n",
    "print('Training time is ', t2-t1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "d2d59fe8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[30708   919  3957  1047  3933     1   629    21  1556  6887  5584    32\n",
      "     10    22   367   924   378]\n",
      " [    0     0  2009     0     0     0     0     0     0     0     0     0\n",
      "      0     0     0     0     0]\n",
      " [    0     0  3699     0     0     0     0     0    26     0     0     0\n",
      "      0     0     0     1     0]\n",
      " [ 1927     0     0     0     2     0     0     0     0    47     0     0\n",
      "      0     0     0     0     0]\n",
      " [   17     0     0    69  1308     0     0     0     0     0     0     0\n",
      "      0     0     0     0     0]\n",
      " [ 2638     0     0     0    32     1     0     0     0     4     3     0\n",
      "      0     0     0     0     0]\n",
      " [    6     0     0     0     0     0  3818     0     5     0   129     0\n",
      "      0     0     1     0     0]\n",
      " [    1     0  2800     0     0     0     0     0   772     0     3     0\n",
      "      0     0     0     3     0]\n",
      " [    1     4     4     0     2     0     0     0  8228    12   256     0\n",
      "      0     0     0  2764     0]\n",
      " [ 2407     0     0     0     0     0     0     0     0  3694   102     0\n",
      "      0     0     0     0     0]\n",
      " [  245     0     0     0     2     0     0     0   103  1090  1611     0\n",
      "      0     0     2   222     3]\n",
      " [   80     0     0     0     0     0     0     0     0   911    77     0\n",
      "      0     0     0     0     0]\n",
      " [ 1702     0     0     0     0     0     0     0     0     2   223     0\n",
      "      0     0     0     0     0]\n",
      " [    0     0     0     0     0     0     0     0    18     0   898     0\n",
      "      0     0     0     0     0]\n",
      " [    0     0     0     0     0     0     0     0   251     2   169     0\n",
      "      0     0   623    25     0]\n",
      " [   33     0     0     1     5     0     0     0  4024     0    34     0\n",
      "      0     0     0  3156    15]\n",
      " [    5     0   116     0     5     0     0     0  1518     3     5     0\n",
      "      0     0     0    52   103]]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Power_divergenceResult(statistic=3124934.310825893, pvalue=0.0)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm = confusion_matrix(y,predicted) #calculate confusion matrix\n",
    "rows, cols = linear_sum_assignment(cm,maximize=True)\n",
    "cm = cm[rows,:]\n",
    "cm = cm[:,cols]\n",
    "print(cm)\n",
    "chisquare(cm,axis=None) #chisquare test"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61103e3d",
   "metadata": {},
   "source": [
    "### DBSCAN\n",
    "\n",
    "Density-based spatial clustering of noisy applications (DBSCAN) is a density-based data clustering algorithm used in data mining and machine learning. The algorithm groups 'densely grouped' data points into a cluster. It can identify clusters in large spatial datasets by looking at the local density of the data points. One of the most important features of DBSCAN clustering against other clustering algorithms is that it is robust against outliers. Also, DBSCAN does not require the number of clusters unlike K-means and Hierarchical clustering."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "7ad95cee",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training time is  15.369641065597534\n"
     ]
    }
   ],
   "source": [
    "DBSCAN_model = DBSCAN()\n",
    "\n",
    "t1 = time.time()\n",
    "DBSCAN_model.fit(X)\n",
    "t2 = time.time()\n",
    "print('Training time is ', t2-t1)\n",
    "predicted = DBSCAN_model.fit_predict(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "2db7c001",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[    0     0     0     0     0     0     0     0     0     0     0     0\n",
      "      0     0     0     0     0     0]\n",
      " [    6 56963     6     0     0     0     0     0     0     0     0     0\n",
      "      0     0     0     0     0     0]\n",
      " [    0  2009     0     0     0     0     0     0     0     0     0     0\n",
      "      0     0     0     0     0     0]\n",
      " [    0  3726     0     0     0     0     0     0     0     0     0     0\n",
      "      0     0     0     0     0     0]\n",
      " [    0  1976     0     0     0     0     0     0     0     0     0     0\n",
      "      0     0     0     0     0     0]\n",
      " [    0  1394     0     0     0     0     0     0     0     0     0     0\n",
      "      0     0     0     0     0     0]\n",
      " [    0  2678     0     0     0     0     0     0     0     0     0     0\n",
      "      0     0     0     0     0     0]\n",
      " [    0  3959     0     0     0     0     0     0     0     0     0     0\n",
      "      0     0     0     0     0     0]\n",
      " [    0  3579     0     0     0     0     0     0     0     0     0     0\n",
      "      0     0     0     0     0     0]\n",
      " [    0 11271     0     0     0     0     0     0     0     0     0     0\n",
      "      0     0     0     0     0     0]\n",
      " [    0  6203     0     0     0     0     0     0     0     0     0     0\n",
      "      0     0     0     0     0     0]\n",
      " [    0  3278     0     0     0     0     0     0     0     0     0     0\n",
      "      0     0     0     0     0     0]\n",
      " [    0  1062     0     0     0     0     0     0     0     0     0     0\n",
      "      6     0     0     0     0     0]\n",
      " [    0  1927     0     0     0     0     0     0     0     0     0     0\n",
      "      0     0     0     0     0     0]\n",
      " [    0   916     0     0     0     0     0     0     0     0     0     0\n",
      "      0     0     0     0     0     0]\n",
      " [    0  1070     0     0     0     0     0     0     0     0     0     0\n",
      "      0     0     0     0     0     0]\n",
      " [    0  7268     0     0     0     0     0     0     0     0     0     0\n",
      "      0     0     0     0     0     0]\n",
      " [    0  1807     0     0     0     0     0     0     0     0     0     0\n",
      "      0     0     0     0     0     0]]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Power_divergenceResult(statistic=10222045.201756911, pvalue=0.0)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm = confusion_matrix(y,predicted) #calculate confusion matrix\n",
    "rows, cols = linear_sum_assignment(cm,maximize=True)\n",
    "cm = cm[rows,:]\n",
    "cm = cm[:,cols]\n",
    "print(cm)\n",
    "chisquare(cm,axis=None) #chisquare test"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "adacaf16",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "1. [How to Master the Popular DBSCAN Clustering Algorithm for Machine Learning](https://www.analyticsvidhya.com/blog/2020/09/how-dbscan-clustering-works/)\n",
    "2. [Scikit-Learn web-site](https://scikit-learn.org/stable/modules/clustering.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4acf3e3",
   "metadata": {},
   "source": [
    "## Q3\n",
    "\n",
    "Now compare supervised and unsupervised algorithm results."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "969c1f8f",
   "metadata": {},
   "source": [
    "## Answer 3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99cbc402",
   "metadata": {},
   "source": [
    "#### Comparison of unsupervised machine learning algorithms (K-means, Hierarchical Clustering, DBSCAN)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf0fea21",
   "metadata": {},
   "source": [
    "Comparison of algorithm from application results,\n",
    "\n",
    "* When the results are examined, it is seen that K-means makes the clustering as soon as possible. Hierarchical clustering is the slowest method. ($t_{k-means} \\approx 10 sec,t_{HC} \\approx 1663 sec, t_{DBSCAN} \\approx 15 sec $)\n",
    "\n",
    "* If the Chi-Square test results are examined, it is seen that the most successful clustering DBSCAN method does. It is seen that the worst clustering is done by the K-means method.\n",
    "\n",
    "* Not specifying the number of clusters in the DBSCAN method may play a role in the success of DBSCAN. The chi-square test result can be improved by determining the number of best clusters in K-means and Hierarchical clustering methods. However, it probably gave the best results because of the DBSCAN algorithm.\n",
    "\n",
    "* In the hierarchical clustering method, a memory error has been received on the personal computer and it has been observed that it uses too much memory. The other two methods could be used on the personal computer without getting a memory error. Hierarchical Clustering method has high memory usage and is a disadvantageous method in this respect.\n",
    "\n",
    "\n",
    "Comparison of algorithm from algorithm's general procedure,\n",
    "\n",
    "* DBSCAN can model outliers in data. Therefore, it is robust to outliers. On the other hand, K-means and Hierarchical clustering methods cannot handle outliers. \n",
    "\n",
    "* K-means is a distance-based lazy algorithm, so it is quite fast. On the contrary, Hierarchical clustering is based on hierarchical alignment, so it is a rather cumbersome (slow) method.\n",
    "\n",
    "* K-means and Hierarchical clustering methods must specify how many clusters there will be as inputs. However, there is no need to specify the number of clusters in the DBSCAN method.\n",
    "\n",
    "* Since DBSCAN is a density-based method, it can handle outliers as well as clusters of different shapes and sizes. However, K-means, Hierarchical clustering methods may not be able to handle clusterings of complex shapes and sizes.\n",
    "\n",
    "* In the Hierarchical Clustering method, memory usage is proportional to $O(n^2)$ and time usage is proportional to $O(n^3)$. Therefore, the use of memory and time in high-dimensional data is increasing. In the other two methods, memory and time usage is quite low compared to hierarchial clustering.\n",
    "\n",
    "* Determining the best number of clusters in the K means method can be complicated. It has a disadvantage in this respect. In the DBSCAN method, not specifying the number of clusters stands out as an advantage.\n",
    "\n",
    "* The K-means algorithm is suitable for large datasets due to its simple clustering algorithm. The DBSCAN algorithm, on the other hand, may not be able to handle the clustering of large datasets. Hierarchical clustering is not suitable for large data sets in terms of memory and time.\n",
    "\n",
    "* In the DBSCAN algorithm, on the other hand, the variance in density can negatively affect clustering. Large variance in Density can negatively affect proper clustering. On the other hand, since the method is sensitive to eps (The maximum distance between two samples for one to be considered as in the neighbourhood of the other) and min point (minimum data in a cluster) parameters, it is necessary to determine the appropriate eps and min point for the data. This can also be considered a disadvantage."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81684647",
   "metadata": {},
   "source": [
    "#### Comparison of supervised machine learning algorithms,\n",
    "\n",
    "Comparison of algorithm from application results,\n",
    "\n",
    "* When the analyzes were repeated in some of the methods used (KNN, SVM), different accuracy metrics were obtained. This situation shows that the methods are not deterministic methods, they are stochastic methods.\n",
    "\n",
    "* KNN and LDA are very fast methods, and when accuracy metrics are compared, it is observed that the KNN method gives better results than the LDA method.\n",
    "\n",
    "* SVR, Logistic and Multinomial Regression methods take longer computation time than KNN and LDA methods.\n",
    "\n",
    "* The accuracy of the SVR method for the considered data set is lower than KNN and LDA."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83c7d764",
   "metadata": {},
   "source": [
    "## References\n",
    "1. [K-means, DBSCAN, GMM, Agglomerative clustering — Mastering the popular models in a segmentation problem](https://towardsdatascience.com/k-means-dbscan-gmm-agglomerative-clustering-mastering-the-popular-models-in-a-segmentation-c891a3818e29)\n",
    "2. [Scikit-Learn web-site](https://scikit-learn.org/stable/modules/clustering.html)\n",
    "3. [Difference between K-Means and DBScan Clustering](https://www.geeksforgeeks.org/difference-between-k-means-and-dbscan-clustering/)\n",
    "4. [How to cluster large amounts of data with minimal memory usage](https://stackoverflow.com/questions/58396826/how-to-cluster-large-amounts-of-data-with-minimal-memory-usage)\n",
    "5. [Comparing supervised learning algorithms](https://www.dataschool.io/comparing-supervised-learning-algorithms/)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f73bf479",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}