Loan_Default_K.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/hannahsophieMWB/machine_learning_MiT/blob/main/Loan_Default_K.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "9dce5b10",
      "metadata": {
        "id": "9dce5b10"
      },
      "source": [
        "# **Loan Default Prediction**"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1e46a80e",
      "metadata": {
        "id": "1e46a80e"
      },
      "source": [
        "----------------------------------------\n",
        "## **Context and Objective**\n",
        "----------------------------------------\n",
        "\n",
        "- It is very critical to be able to identify the loan defaults in the financial domain. Given a set of features about a person, the goal is to predict if the person will default or not.\n",
        "- In this notebook, we are continuing to cover the concepts of LVC 3 and try to create a model that predicts if a particular person will default or not.\n",
        "\n",
        "--------------------\n",
        "## **Dataset**\n",
        "--------------------\n",
        "\n",
        "The dataset used has the below features:\n",
        "\n",
        "- **Default** - Yes/No, categorical dependent variable suggesting the state of default.\n",
        "- **Student** - Yes/No, categorical independent variable depicting if the applicant is a student or not.\n",
        "- **Balance** - The account balance of the applicant.\n",
        "- **Income** - The income of the applicant."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "9fa7be61",
      "metadata": {
        "id": "9fa7be61"
      },
      "source": [
        "## **Importing the libraries and overview of the dataset**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "id": "4ff42574",
      "metadata": {
        "id": "4ff42574"
      },
      "outputs": [],
      "source": [
        "# Used for data manipulation\n",
        "import pandas as pd\n",
        "\n",
        "# Used for numerical computations\n",
        "import numpy as np\n",
        "\n",
        "from sklearn import linear_model\n",
        "\n",
        "# Used for visualizations\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "\n",
        "# To get rid of warning messages\n",
        "import warnings\n",
        "warnings.filterwarnings('ignore')\n",
        "\n",
        "# Remove scientific notations and display numbers with 2 decimal points instead\n",
        "pd.options.display.float_format = '{:,.2f}'.format\n",
        "\n",
        "# Update default background style of plots\n",
        "sns.set_style(style='darkgrid')"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# mount GDrive\n",
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RSDtgYjJfyIS",
        "outputId": "a2b37de8-0e4f-424b-ab88-150f4692ab14"
      },
      "id": "RSDtgYjJfyIS",
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/drive\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "id": "0a09483c",
      "metadata": {
        "id": "0a09483c"
      },
      "source": [
        "### **Loading the data**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "id": "f8700fb5",
      "metadata": {
        "id": "f8700fb5"
      },
      "outputs": [],
      "source": [
        "# Let us import the files\n",
        "df = pd.read_csv('/content/drive/MyDrive/AI and Machine Learning/machine_learning_MiT/MiT Course/Week 3/GL 3 - Classification /Loan Default.csv')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "id": "d2c2ecb2",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "d2c2ecb2",
        "outputId": "1425e018-9cd5-447e-b2ea-9771c83e0df8"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "  default student  balance    income\n",
              "0      No      No   729.53 44,361.63\n",
              "1      No     Yes   817.18 12,106.13\n",
              "2      No      No 1,073.55 31,767.14\n",
              "3      No      No   529.25 35,704.49\n",
              "4      No      No   785.66 38,463.50"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-8956c8e9-7fb7-46f4-a668-201675a05cf8\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>default</th>\n",
              "      <th>student</th>\n",
              "      <th>balance</th>\n",
              "      <th>income</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>No</td>\n",
              "      <td>No</td>\n",
              "      <td>729.53</td>\n",
              "      <td>44,361.63</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>No</td>\n",
              "      <td>Yes</td>\n",
              "      <td>817.18</td>\n",
              "      <td>12,106.13</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>No</td>\n",
              "      <td>No</td>\n",
              "      <td>1,073.55</td>\n",
              "      <td>31,767.14</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>No</td>\n",
              "      <td>No</td>\n",
              "      <td>529.25</td>\n",
              "      <td>35,704.49</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>No</td>\n",
              "      <td>No</td>\n",
              "      <td>785.66</td>\n",
              "      <td>38,463.50</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-8956c8e9-7fb7-46f4-a668-201675a05cf8')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-8956c8e9-7fb7-46f4-a668-201675a05cf8 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-8956c8e9-7fb7-46f4-a668-201675a05cf8');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-6a04116a-bb78-44f3-833f-68a197518539\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-6a04116a-bb78-44f3-833f-68a197518539')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-6a04116a-bb78-44f3-833f-68a197518539 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ],
      "source": [
        "# Dropping unnecessary column from the dataframe\n",
        "df.drop(columns = ['Unnamed: 0'], inplace = True)\n",
        "\n",
        "# View the head of the data frame\n",
        "df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ed6e44c7",
      "metadata": {
        "id": "ed6e44c7"
      },
      "source": [
        "### **Encoding categorical variables**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "id": "e3b5aed7",
      "metadata": {
        "id": "e3b5aed7"
      },
      "outputs": [],
      "source": [
        "# Let us define a dictionary to replace the Yes/No with 1/0\n",
        "encoding_dict = {'Yes': 1, 'No': 0}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "id": "c32a9651",
      "metadata": {
        "id": "c32a9651"
      },
      "outputs": [],
      "source": [
        "# Encoding the features\n",
        "df['default'] = df['default'].map(encoding_dict)\n",
        "\n",
        "df['student'] = df['student'].map(encoding_dict)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "id": "01905ed5",
      "metadata": {
        "id": "01905ed5"
      },
      "outputs": [],
      "source": [
        "# Creating the independent variables\n",
        "X = df.drop('default', axis = 1)\n",
        "\n",
        "# Target variable\n",
        "Y = df['default']"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "id": "9688fdd2",
      "metadata": {
        "id": "9688fdd2"
      },
      "outputs": [],
      "source": [
        "# Instantiating the scaler\n",
        "ss = StandardScaler()\n",
        "\n",
        "X = ss.fit_transform(X)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "0e47d4af",
      "metadata": {
        "id": "0e47d4af"
      },
      "source": [
        "## **K-Nearest Neighbors from Scratch**"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "3de9a821",
      "metadata": {
        "id": "3de9a821"
      },
      "source": [
        "When we implement KNN algorithm using sklearn library, we find its K-Nearest Neighbors for each record in the training set, **including the point itself**, and classify by majority vote. In particular, when K=1, the record only looks at itself and always gets it right, and train error is zero. For the records in the test set, we find the K-Nearest Neighbors in the training set (thus, these are different than the record being tested), and classify by majority vote.\n",
        "\n",
        "Due to this difference between classification in training and testing data, we can try to do the same thing on the entire  dataset, without leaving out a test set, i.e., for each record in the entire data set, we find the K-Nearest Neighbors (**other than itself**), and classify by majority vote.\n",
        "\n",
        "To do this, we will need to code the algorithm from scratch. The below code creates a class called KNearestNeighbor with different methods like train(), predict(). We have shown different ways to computer distances between data points. Invoking the predict() by specifying the data and num_loops will compute the distances and return respective predictions.\n",
        "\n",
        "**Step-wise implementation of KNN from scratch**\n",
        "\n",
        "1. Instantiating the K-Nearest Neighbor\n",
        "\n",
        "2. Setting the training data using the train() method\n",
        "\n",
        "3. Invoking the predict() by specifying the data and num_loops. It will compute the distances and return respective predictions."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "id": "315098bb",
      "metadata": {
        "id": "315098bb"
      },
      "outputs": [],
      "source": [
        "\"\"\"\n",
        "Implementation of K-nearest neighbor (KNN) from scratch\n",
        "where you can either use 2-loops (inefficient), 1-loop (better)\n",
        "or a heavily vectorized zero-loop implementation.\n",
        "\n",
        "\"\"\"\n",
        "\n",
        "import numpy as np\n",
        "\n",
        "\n",
        "class KNearestNeighbor:\n",
        "    def __init__(self, k):\n",
        "        self.k = k\n",
        "\n",
        "    def train(self, X, y):\n",
        "        self.X_train = X\n",
        "        self.y_train = y\n",
        "\n",
        "    def predict(self, X_test, num_loops=0, train=True):\n",
        "        if num_loops == 0:\n",
        "            distances = self.compute_distance_vectorized(X_test)\n",
        "\n",
        "        elif num_loops == 1:\n",
        "            distances = self.compute_distance_one_loop(X_test)\n",
        "\n",
        "        else:\n",
        "            distances = self.compute_distance_two_loops(X_test)\n",
        "\n",
        "        return self.predict_labels(distances, train)\n",
        "\n",
        "    def compute_distance_two_loops(self, X_test):\n",
        "        \"\"\"\n",
        "        Inefficient naive implementation, use only\n",
        "        as a way of understanding what kNN is doing\n",
        "        \"\"\"\n",
        "\n",
        "        num_test = X_test.shape[0]\n",
        "        num_train = self.X_train.shape[0]\n",
        "        distances = np.zeros((num_test, num_train))\n",
        "\n",
        "        for i in range(num_test):\n",
        "            for j in range(num_train):\n",
        "                distances[i, j] = np.sqrt(np.sum((X_test[i, :] - self.X_train[j, :])**2))\n",
        "\n",
        "\n",
        "        return distances\n",
        "\n",
        "    def compute_distance_one_loop(self, X_test):\n",
        "        \"\"\"\n",
        "        Much better than two-loops but not as fast as fully vectorized version.\n",
        "        Utilize Numpy broadcasting in X_train - X_test[i,:]\n",
        "        \"\"\"\n",
        "        num_test = X_test.shape[0]\n",
        "        num_train = self.X_train.shape[0]\n",
        "        distances = np.zeros((num_test, num_train))\n",
        "\n",
        "        for i in range(num_test):\n",
        "            distances[i, :] = np.sqrt(np.sum((self.X_train - X_test[i, :])** 2, axis=1))\n",
        "\n",
        "        return distances\n",
        "\n",
        "    def compute_distance_vectorized(self, X_test):\n",
        "        \"\"\"\n",
        "        Can be tricky to understand this, we utilize heavy\n",
        "        vecotorization as well as numpy broadcasting.\n",
        "        Idea: if we have two vectors a, b (two examples)\n",
        "        and for vectors we can compute (a-b)^2 = a^2 - 2a (dot) b + b^2\n",
        "        expanding on this and doing so for every vector lends to the\n",
        "        heavy vectorized formula for all examples at the same time.\n",
        "        \"\"\"\n",
        "        X_test_squared = np.sum(X_test ** 2, axis=1, keepdims=True)\n",
        "        X_train_squared = np.sum(self.X_train ** 2, axis=1, keepdims=True)\n",
        "        two_X_test_X_train = np.dot(X_test, self.X_train.T)\n",
        "\n",
        "        return np.sqrt(\n",
        "            self.eps + X_test_squared - 2 * two_X_test_X_train + X_train_squared.T\n",
        "        )\n",
        "\n",
        "    def predict_labels(self, distances, train):\n",
        "        num_test = distances.shape[0]\n",
        "        y_pred = np.zeros(num_test)\n",
        "\n",
        "        for i in range(num_test):\n",
        "            y_indices = np.argsort(distances[i, :])\n",
        "            if train==True:\n",
        "                k_closest_classes = self.y_train[y_indices[1:self.k+1]].astype(int)\n",
        "            elif train==False:\n",
        "                k_closest_classes = self.y_train[y_indices[:self.k]].astype(int)\n",
        "            y_pred[i] = np.argmax(np.bincount(k_closest_classes))\n",
        "\n",
        "        return y_pred"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "id": "07df6cd0",
      "metadata": {
        "id": "07df6cd0"
      },
      "outputs": [],
      "source": [
        "# Converting into an array\n",
        "X_array = np.array(X)\n",
        "\n",
        "y_array = np.array(Y)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "8a3e264a",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8a3e264a",
        "outputId": "981c3375-7e8b-4fe9-f23f-3faefd266fe1",
        "scrolled": false
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "1\n",
            "[0.04190000000000005]\n",
            "2\n",
            "[0.03380000000000005]\n",
            "3\n",
            "[0.03390000000000004]\n",
            "4\n",
            "[0.03049999999999997]\n",
            "5\n",
            "[0.032200000000000006]\n",
            "6\n",
            "[0.02939999999999998]\n",
            "7\n",
            "[0.029000000000000026]\n",
            "8\n",
            "[0.027800000000000047]\n",
            "9\n",
            "[0.028800000000000048]\n",
            "10\n",
            "[0.027699999999999947]\n",
            "11\n",
            "[0.028100000000000014]\n",
            "12\n",
            "[0.028100000000000014]\n",
            "13\n",
            "[0.027699999999999947]\n",
            "14\n",
            "[0.02749999999999997]\n",
            "15\n",
            "[0.027800000000000047]\n",
            "16\n",
            "[0.027800000000000047]\n",
            "17\n",
            "[0.027599999999999958]\n",
            "18\n",
            "[0.028100000000000014]\n",
            "19\n",
            "[0.027599999999999958]\n",
            "20\n",
            "[0.027900000000000036]\n",
            "21\n",
            "[0.027599999999999958]\n",
            "22\n",
            "[0.027800000000000047]\n",
            "23\n",
            "[0.02739999999999998]\n",
            "24\n",
            "[0.027800000000000047]\n",
            "25\n",
            "[0.02739999999999998]\n",
            "26\n",
            "[0.027800000000000047]\n",
            "27\n",
            "[0.027800000000000047]\n",
            "28\n",
            "[0.028100000000000014]\n",
            "29\n",
            "[0.02739999999999998]\n",
            "30\n",
            "[0.027800000000000047]\n",
            "31\n",
            "[0.02749999999999997]\n",
            "32\n",
            "[0.027599999999999958]\n",
            "33\n",
            "[0.02739999999999998]\n",
            "34\n",
            "[0.028000000000000025]\n",
            "35\n",
            "[0.027599999999999958]\n",
            "36\n",
            "[0.027900000000000036]\n",
            "37\n",
            "[0.02729999999999999]\n",
            "38\n",
            "[0.027699999999999947]\n",
            "39\n",
            "[0.02729999999999999]\n",
            "40\n",
            "[0.02739999999999998]\n",
            "41\n",
            "[0.02729999999999999]\n",
            "42\n",
            "[0.027599999999999958]\n",
            "43\n",
            "[0.02729999999999999]\n",
            "44\n",
            "[0.027699999999999947]\n",
            "45\n",
            "[0.027599999999999958]\n",
            "46\n",
            "[0.028200000000000003]\n",
            "47\n",
            "[0.027699999999999947]\n",
            "48\n",
            "[0.028100000000000014]\n",
            "49\n",
            "[0.027800000000000047]\n",
            "50\n",
            "[0.028200000000000003]\n",
            "51\n",
            "[0.027699999999999947]\n",
            "52\n",
            "[0.028000000000000025]\n",
            "53\n",
            "[0.027599999999999958]\n",
            "54\n",
            "[0.027800000000000047]\n",
            "55\n",
            "[0.027800000000000047]\n",
            "56\n",
            "[0.027900000000000036]\n",
            "57\n",
            "[0.027800000000000047]\n",
            "58\n",
            "[0.027699999999999947]\n",
            "59\n"
          ]
        }
      ],
      "source": [
        "# We select the best value of K for which the error rate is the least\n",
        "# Let us loop over a few values of K to determine the best value\n",
        "\n",
        "knn_many_split = {}\n",
        "\n",
        "for n_neighbors in range(1, 200):\n",
        "    lista = []\n",
        "    print(n_neighbors)\n",
        "\n",
        "    # Fitting the KNN with k = n_neighbors\n",
        "    KNN = KNearestNeighbor(k = n_neighbors)\n",
        "\n",
        "    # Fitting the model on the data\n",
        "    KNN.train(X_array, y_array)\n",
        "\n",
        "    # Making predictions on the data using trained model\n",
        "    y_pred = KNN.predict(X_array, num_loops = 1, train = True)\n",
        "\n",
        "    lista.append(1 - (sum(y_pred == y_array) / y_array.shape[0]))\n",
        "\n",
        "    knn_many_split[n_neighbors] = lista\n",
        "\n",
        "    print(knn_many_split[n_neighbors])\n",
        "\n",
        "knn_many_split"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "e5c5c03c",
      "metadata": {
        "id": "e5c5c03c"
      },
      "outputs": [],
      "source": [
        "# Extracting k, the error in different lists for easy plotting\n",
        "\n",
        "kl = []\n",
        "vl = []\n",
        "for n_neighbors, v in knn_many_split.items():\n",
        "    kl.append(n_neighbors)\n",
        "    vl.append(knn_many_split[n_neighbors][0])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "5a40b8e8",
      "metadata": {
        "id": "5a40b8e8"
      },
      "outputs": [],
      "source": [
        "# Plotting k vs error\n",
        "plt.figure(figsize=(8,8))\n",
        "plt.xlabel(\"number of neighbors (k)\")\n",
        "plt.ylabel(\"Test Error\")\n",
        "plt.plot(kl,vl)\n",
        "plt.legend()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "bcc38949",
      "metadata": {
        "id": "bcc38949"
      },
      "source": [
        "## **Conclusion**"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "244cc7cc",
      "metadata": {
        "id": "244cc7cc"
      },
      "source": [
        "- The test error is changing with changing the k values.\n",
        "- The value of K in the KNN method is related to the model's error rate. A small value of K may result in overfitting, whereas a large value of k may result in underfitting. Overfitting implies that the model performs well on the training data but performs poorly when additional data is introduced.\n",
        "- We can observe that the test error increased with the increase in the value of K, it started growing after K crosses 70. The larger the value of K is, the smoother the classification boundary. So, we have to pick the best suitable K value by evaluating with different values of K."
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.7"
    },
    "colab": {
      "provenance": [],
      "include_colab_link": true
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}