Lab06_1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YSWhGiSwpnRE"
   },
   "source": [
    "# Lab 6-1: Softmax Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TeIFf07epnRJ"
   },
   "source": [
    "Author: Seungjae Lee (이승재)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ec-nhEcHpnRK"
   },
   "source": [
    "<div class=\"alert alert-warning\">\n",
    "    We use elemental PyTorch to implement linear regression here. However, in most actual applications, abstractions such as <code>nn.Module</code> or <code>nn.Linear</code> are used.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5oVx3tG5pnRM"
   },
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "XFbew_sIpnRM"
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "sPCgvRxwpnRO",
    "outputId": "0d484dcd-5591-44f0-ac78-982d6c30cdf1"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch._C.Generator at 0x23135a4c0d0>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# For reproducibility\n",
    "torch.manual_seed(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LYiwp7gdpnRP"
   },
   "source": [
    "## Softmax"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KJur853UpnRQ"
   },
   "source": [
    "Convert numbers to probabilities with softmax."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Q3Vz1MN7pnRQ"
   },
   "source": [
    "$$ P(class=i) = \\frac{e^i}{\\sum e^i} $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "3xcvLsDdpnRR"
   },
   "outputs": [],
   "source": [
    "z = torch.FloatTensor([1, 2, 3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6xAjeEkFpnRR"
   },
   "source": [
    "PyTorch has a `softmax` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "-R3KruSEpnRS",
    "outputId": "ff1c0299-0f23-41f7-e12c-da16e8ea4bac"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([0.0900, 0.2447, 0.6652])\n"
     ]
    }
   ],
   "source": [
    "hypothesis = F.softmax(z, dim=0)\n",
    "print(hypothesis)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-rmQK9sFpnRS"
   },
   "source": [
    "Since they are probabilities, they should add up to 1. Let's do a sanity check."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Zq5kq3ylpnRT",
    "outputId": "cf2f33f4-971b-49e7-c3ca-cb649fa5047d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(1.)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hypothesis.sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fLjsxrp4pnRT"
   },
   "source": [
    "## Cross Entropy Loss (Low-level)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qJWF6ODqpnRT"
   },
   "source": [
    "For multi-class classification, we use the cross entropy loss."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dAdYZDo-pnRU"
   },
   "source": [
    "$$ L = \\frac{1}{N} \\sum - y \\log(\\hat{y}) $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "y70dsSjMpnRU"
   },
   "source": [
    "where $\\hat{y}$ is the predicted probability and $y$ is the correct probability (0 or 1)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "URCjBmj7pnRU",
    "outputId": "44a2ea26-7b7e-44e9-a377-3e011e4d5888",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0.2645, 0.1639, 0.1855, 0.2585, 0.1277],\n",
      "        [0.2430, 0.1624, 0.2322, 0.1930, 0.1694],\n",
      "        [0.2226, 0.1986, 0.2326, 0.1594, 0.1868]], grad_fn=<SoftmaxBackward0>)\n"
     ]
    }
   ],
   "source": [
    "z = torch.rand(3, 5, requires_grad=True)\n",
    "hypothesis = F.softmax(z, dim=1)\n",
    "print(hypothesis)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "SV7GUNmSpnRV",
    "outputId": "0aaa0731-e3f7-4fae-ed43-75e289f9e076"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([0, 2, 1])\n"
     ]
    }
   ],
   "source": [
    "y = torch.randint(5, (3,)).long()\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "le1w95aNpnRV",
    "outputId": "667ef8a2-7f22-4ac9-ffea-081a22407ef7"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 0., 0., 0., 0.],\n",
       "        [0., 0., 1., 0., 0.],\n",
       "        [0., 1., 0., 0., 0.]])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_one_hot = torch.zeros_like(hypothesis)\n",
    "y_one_hot.scatter_(1, y.unsqueeze(1), 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "bzY9GrzypnRV",
    "outputId": "a77485e6-c51a-4564-d0c4-79e56e094cbe"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(1.4689, grad_fn=<MeanBackward0>)\n"
     ]
    }
   ],
   "source": [
    "cost = (y_one_hot * -torch.log(hypothesis)).sum(dim=1).mean()\n",
    "print(cost)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fH5GxzSJpnRW"
   },
   "source": [
    "## Cross-entropy Loss with `torch.nn.functional`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Q8CjNMK0pnRW"
   },
   "source": [
    "PyTorch has `F.log_softmax()` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "bKn97pKCpnRW",
    "outputId": "9b89e799-a5c5-4211-aeab-a52e144f722c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-1.3301, -1.8084, -1.6846, -1.3530, -2.0584],\n",
       "        [-1.4147, -1.8174, -1.4602, -1.6450, -1.7758],\n",
       "        [-1.5025, -1.6165, -1.4586, -1.8360, -1.6776]], grad_fn=<LogBackward0>)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Low level\n",
    "torch.log(F.softmax(z, dim=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "yH5RZXWOpnRX",
    "outputId": "e142ebd2-c26e-405e-dcfe-0c001888420d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-1.3301, -1.8084, -1.6846, -1.3530, -2.0584],\n",
       "        [-1.4147, -1.8174, -1.4602, -1.6450, -1.7758],\n",
       "        [-1.5025, -1.6165, -1.4586, -1.8360, -1.6776]],\n",
       "       grad_fn=<LogSoftmaxBackward0>)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# High level\n",
    "F.log_softmax(z, dim=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EEZdnuclpnRX"
   },
   "source": [
    "PyTorch also has `F.nll_loss()` function that computes the negative loss likelihood."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "t8uCmj5NpnRX",
    "outputId": "207ae241-9ff3-491a-98a8-eb33f2155bde"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(1.4689, grad_fn=<MeanBackward0>)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Low level\n",
    "(y_one_hot * -torch.log(F.softmax(z, dim=1))).sum(dim=1).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UhArcV6JsqLZ"
   },
   "source": [
    "Q1. 위에서 배운 torch.log(F.softmax(z, dim=1)) => F.log_softmax(z,dim=1)을 통해 비용함수를 수정해보세요!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "F-unuKlqsNSn",
    "outputId": "8e2389d0-7354-4580-8364-0c864c15a1bf"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(1.4689, grad_fn=<MeanBackward0>)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(y_one_hot * - F.log_softmax(z,dim=1)).sum(dim=1).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "HlrCGMAwpnRY",
    "outputId": "be9f217c-c52a-4f97-a1c4-a92974c6879c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(1.4689, grad_fn=<NllLossBackward0>)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# High level\n",
    "F.nll_loss(F.log_softmax(z, dim=1), y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6C0br2TWpnRY"
   },
   "source": [
    "PyTorch also has `F.cross_entropy` that combines `F.log_softmax()` and `F.nll_loss()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "KRDiKbYopnRY",
    "outputId": "f57a1364-2ecf-4381-cef0-de9b794a1c1a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(1.4689, grad_fn=<NllLossBackward0>)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "F.cross_entropy(z, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IkokQKF4pnRZ"
   },
   "source": [
    "## Training with Low-level Cross Entropy Loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "id": "PV5b_p-GpnRZ"
   },
   "outputs": [],
   "source": [
    "x_train = [[1, 2, 1, 1],\n",
    "           [2, 1, 3, 2],\n",
    "           [3, 1, 3, 4],\n",
    "           [4, 1, 5, 5],\n",
    "           [1, 7, 5, 5],\n",
    "           [1, 2, 5, 6],\n",
    "           [1, 6, 6, 6],\n",
    "           [1, 7, 7, 7]]\n",
    "y_train = [2, 2, 2, 1, 1, 1, 0, 0] # 총 3개의 클래스\n",
    "x_train = torch.FloatTensor(x_train)\n",
    "y_train = torch.LongTensor(y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "id": "ysLsPyr0pnRZ",
    "outputId": "57d6afbc-37ce-42f1-d39f-8ca293e715ef"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch    0/1000 Cost: 1.098612\n",
      "Epoch  100/1000 Cost: 0.761050\n",
      "Epoch  200/1000 Cost: 0.689991\n",
      "Epoch  300/1000 Cost: 0.643229\n",
      "Epoch  400/1000 Cost: 0.604117\n",
      "Epoch  500/1000 Cost: 0.568255\n",
      "Epoch  600/1000 Cost: 0.533922\n",
      "Epoch  700/1000 Cost: 0.500291\n",
      "Epoch  800/1000 Cost: 0.466908\n",
      "Epoch  900/1000 Cost: 0.433507\n",
      "Epoch 1000/1000 Cost: 0.399962\n"
     ]
    }
   ],
   "source": [
    "# 모델 초기화\n",
    "W = torch.zeros((4, 3), requires_grad=True) #가중치 W를 0으로 초기화하고 학습을 통해 값이 변경될 수 있도록 하세요.\n",
    "b = torch.zeros(1, requires_grad=True) #편향 b를 0으로 초기화하고 학습을 통해 값이 변경될 수 있도록 하세요.\n",
    "# optimizer 설정\n",
    "optimizer = optim.SGD([W, b], lr = 0.1) #SGD optimizer를 사용하고 learning rate는 0.1로 적용하세요.\n",
    "\n",
    "nb_epochs = 1000\n",
    "for epoch in range(nb_epochs + 1):\n",
    "\n",
    "    # Cost 계산 (1)\n",
    "    hypothesis = F.softmax(x_train.matmul(W) + b, dim = 1) # 입력은 x_train과 W의 행렬 곱셈이고 두번째 차원에 대해서 소프트맥스 함수를 적용하세요. \n",
    "    y_one_hot = torch.zeros_like(hypothesis)\n",
    "    y_one_hot.scatter_(1, y_train.unsqueeze(1), 1)\n",
    "    cost =  (y_one_hot * -torch.log(hypothesis)).sum(dim=1).mean() #low-level의 cross entropy를 사용해주세요\n",
    "\n",
    "    # cost로 H(x) 개선\n",
    "    optimizer.zero_grad()\n",
    "    cost.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "    # 100번마다 로그 출력\n",
    "    if epoch % 100 == 0:\n",
    "        print('Epoch {:4d}/{} Cost: {:.6f}'.format(\n",
    "            epoch, nb_epochs, cost.item()\n",
    "        ))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jwHS3xTzpnRd"
   },
   "source": [
    "## Training with `F.cross_entropy`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "id": "LqE-NgwlpnRe",
    "outputId": "b9a3b975-cf0a-4e00-bdee-0e40b29bd790"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch    0/1000 Cost: 1.098612\n",
      "Epoch  100/1000 Cost: 0.761050\n",
      "Epoch  200/1000 Cost: 0.689991\n",
      "Epoch  300/1000 Cost: 0.643229\n",
      "Epoch  400/1000 Cost: 0.604117\n",
      "Epoch  500/1000 Cost: 0.568255\n",
      "Epoch  600/1000 Cost: 0.533922\n",
      "Epoch  700/1000 Cost: 0.500291\n",
      "Epoch  800/1000 Cost: 0.466908\n",
      "Epoch  900/1000 Cost: 0.433507\n",
      "Epoch 1000/1000 Cost: 0.399962\n"
     ]
    }
   ],
   "source": [
    "# 모델 초기화\n",
    "W = torch.zeros((4, 3), requires_grad=True) #가중치 W를 0으로 초기화하고 학습을 통해 값이 변경될 수 있도록 하세요.\n",
    "b = torch.zeros(1, requires_grad=True) #편향 b를 0으로 초기화하고 학습을 통해 값이 변경될 수 있도록 하세요.\n",
    "# optimizer 설정\n",
    "optimizer = optim.SGD([W, b], lr = 0.1) #SGD optimizer를 사용하고 learning rate는 0.1로 적용하세요.\n",
    "\n",
    "nb_epochs = 1000\n",
    "for epoch in range(nb_epochs + 1):\n",
    "\n",
    "    # Cost 계산 (2)\n",
    "    z = x_train.matmul(W) + b # or .mm or @\n",
    "    cost = F.cross_entropy(z, y_train) #PyTorch에서 제공하는 F.cross_entropy함수를 사용해 cost를 구현해보세요.\n",
    "\n",
    "    # cost로 H(x) 개선\n",
    "    optimizer.zero_grad()\n",
    "    cost.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "    # 100번마다 로그 출력\n",
    "    if epoch % 100 == 0:\n",
    "        print('Epoch {:4d}/{} Cost: {:.6f}'.format(\n",
    "            epoch, nb_epochs, cost.item()\n",
    "        ))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "h8IEcPKLpnRe"
   },
   "source": [
    "## High-level Implementation with `nn.Module`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "id": "DOE776ujpnRf"
   },
   "outputs": [],
   "source": [
    "class SoftmaxClassifierModel(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.linear = nn.Linear(4, 3) #x_train의 shape과 y_train의 class의 갯수를 고려해보세요.\n",
    "\n",
    "    def forward(self, x):\n",
    "        return self.linear(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "id": "oIiAVoe2pnRf"
   },
   "outputs": [],
   "source": [
    "model = SoftmaxClassifierModel() #SoftmaxClassifierModel()로 모델을 초기화하세요."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "w78KcGSBpnRf"
   },
   "source": [
    "Let's try another new dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "id": "v4xDQefApnRf",
    "outputId": "667a213c-6848-4868-d6a5-67d6aea30b67"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch    0/1000 Cost: 1.849513\n",
      "Epoch  100/1000 Cost: 0.689894\n",
      "Epoch  200/1000 Cost: 0.609259\n",
      "Epoch  300/1000 Cost: 0.551218\n",
      "Epoch  400/1000 Cost: 0.500141\n",
      "Epoch  500/1000 Cost: 0.451947\n",
      "Epoch  600/1000 Cost: 0.405051\n",
      "Epoch  700/1000 Cost: 0.358733\n",
      "Epoch  800/1000 Cost: 0.312912\n",
      "Epoch  900/1000 Cost: 0.269522\n",
      "Epoch 1000/1000 Cost: 0.241922\n"
     ]
    }
   ],
   "source": [
    "# optimizer 설정\n",
    "optimizer = optim.SGD(model.parameters(), lr = 0.1) #SGD optimizer를 사용하고 learning rate는 0.1로 적용하세요.\n",
    "\n",
    "nb_epochs = 1000\n",
    "for epoch in range(nb_epochs + 1):\n",
    "\n",
    "    # H(x) 계산\n",
    "    prediction = model(x_train) #모델을 생성해서 예측값을 구해보세요.\n",
    "\n",
    "    # cost 계산\n",
    "    cost = F.cross_entropy(prediction, y_train) #PyTorch에서 제공하는 F.cross_entropy함수를 사용해 cost를 구현해보세요.\n",
    "\n",
    "    # cost로 H(x) 개선\n",
    "    optimizer.zero_grad()\n",
    "    cost.backward()\n",
    "    optimizer.step()\n",
    "    \n",
    "    # 20번마다 로그 출력\n",
    "    if epoch % 100 == 0:\n",
    "        print('Epoch {:4d}/{} Cost: {:.6f}'.format(\n",
    "            epoch, nb_epochs, cost.item()\n",
    "        ))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "Lab06_1.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "PyTorch",
   "language": "python",
   "name": "pytorch"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}