Lab05.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OWoEjnmfd1lG"
   },
   "source": [
    "# Lab 5: Logistic Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jYgrJaSBd1lN"
   },
   "source": [
    "Author: Seungjae Lee (이승재)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fX12twYRd1lO"
   },
   "source": [
    "<div class=\"alert alert-warning\">\n",
    "    We use elemental PyTorch to implement linear regression here. However, in most actual applications, abstractions such as <code>nn.Module</code> or <code>nn.Linear</code> are used. You can see those implementations near the end of this notebook.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bao6VR5Pd1lP"
   },
   "source": [
    "## Reminder: Logistic Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lKvotCYed1lP"
   },
   "source": [
    "### Hypothesis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HD-S293ed1lQ"
   },
   "source": [
    "$$ H(X) = \\frac{1}{1+e^{-W^T X}} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DQLeTc5hd1lQ"
   },
   "source": [
    "### Cost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PBIIU057d1lR"
   },
   "source": [
    "$$ cost(W) = -\\frac{1}{m} \\sum y \\log\\left(H(x)\\right) + (1-y) \\left( \\log(1-H(x) \\right) $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xpTGq6kld1lS"
   },
   "source": [
    " - If $y \\simeq H(x)$, cost is near 0.\n",
    " - If $y \\neq H(x)$, cost is high."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zVdnlofcd1lW"
   },
   "source": [
    "### Weight Update via Gradient Descent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-Ok5jeK3d1lY"
   },
   "source": [
    "$$ W := W - \\alpha \\frac{\\partial}{\\partial W} cost(W) $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jnNit8did1lY"
   },
   "source": [
    " - $\\alpha$: Learning rate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "r0i_bbozd1lZ"
   },
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "6GNFdC7Kd1la"
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "XiHiXkg0d1lb",
    "outputId": "ac26c7a4-4ba8-4a28-9631-b6d342675854"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch._C.Generator at 0x22a0c72ba08>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# For reproducibility\n",
    "torch.manual_seed(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-hbwRxSFd1lc"
   },
   "source": [
    "## Training Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "S6sdFKP5d1ld"
   },
   "outputs": [],
   "source": [
    "x_data = [[1, 2], [2, 3], [3, 1], [4, 3], [5, 3], [6, 2]]\n",
    "y_data = [[0], [0], [0], [1], [1], [1]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ISorVWoSd1ld"
   },
   "source": [
    "Consider the following classification problem: given the number of hours each student spent watching the lecture and working in the code lab, predict whether the student passed or failed a course. For example, the first (index 0) student watched the lecture for 1 hour and spent 2 hours in the lab session ([1, 2]), and ended up failing the course ([0])."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "lallaxkPd1le"
   },
   "outputs": [],
   "source": [
    "x_train = torch.FloatTensor(x_data)\n",
    "y_train = torch.FloatTensor(y_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rM89auFWd1le"
   },
   "source": [
    "As always, we need these data to be in `torch.Tensor` format, so we convert them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "N9HM9pUGd1lf",
    "outputId": "e677cc40-a0da-4382-de4d-6b15084f5212"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([6, 2])\n",
      "torch.Size([6, 1])\n"
     ]
    }
   ],
   "source": [
    "print(x_train.shape)\n",
    "print(y_train.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rTdVp92Dd1lf"
   },
   "source": [
    "## Computing the Hypothesis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "n4RqCyp1d1lf"
   },
   "source": [
    "$$ H(X) = \\frac{1}{1+e^{-W^T X}} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Bo1a4R8sd1lg"
   },
   "source": [
    "PyTorch has a `torch.exp()` function that resembles the exponential function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "9iDhEXmVd1lg",
    "outputId": "adf75f61-a2e7-4cc8-d3ea-bfce459a05cc"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "e^1 equals:  tensor([2.7183])\n"
     ]
    }
   ],
   "source": [
    "print('e^1 equals: ', torch.exp(torch.FloatTensor([1])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_1I9zt17d1lh"
   },
   "source": [
    "We can use it to compute the hypothesis function conveniently."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "aQqWIqoLd1lh"
   },
   "outputs": [],
   "source": [
    "W = torch.zeros((2, 1), requires_grad=True)\n",
    "b = torch.zeros(1, requires_grad=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "A7r3kFFvd1lh"
   },
   "outputs": [],
   "source": [
    "hypothesis = 1 / (1 + torch.exp(-(x_train.matmul(W) + b)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "Z1-oWTp5d1lh",
    "outputId": "e91811d7-d3e3-48b2-a2a7-551f3efeceb4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0.5000],\n",
      "        [0.5000],\n",
      "        [0.5000],\n",
      "        [0.5000],\n",
      "        [0.5000],\n",
      "        [0.5000]], grad_fn=<MulBackward0>)\n",
      "torch.Size([6, 1])\n"
     ]
    }
   ],
   "source": [
    "print(hypothesis)\n",
    "print(hypothesis.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hxJO6JfKd1li"
   },
   "source": [
    "Or, we could use `torch.sigmoid()` function! This resembles the sigmoid function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "XSyZNfPhd1li",
    "outputId": "0f32c457-0503-46ff-bbb3-cc0d60216c3c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/(1+e^{-1}) equals:  tensor([0.7311])\n"
     ]
    }
   ],
   "source": [
    "print('1/(1+e^{-1}) equals: ', torch.sigmoid(torch.FloatTensor([1])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "D_ZfiY7cd1lj"
   },
   "source": [
    "Now, the code for hypothesis function is cleaner."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "B9Gvs6phd1lj"
   },
   "outputs": [],
   "source": [
    "hypothesis = torch.sigmoid(x_train.matmul(W) + b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "vE8rzSZxd1lj",
    "outputId": "6ab5987a-2041-4214-b070-cc96617ea0ff"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0.5000],\n",
      "        [0.5000],\n",
      "        [0.5000],\n",
      "        [0.5000],\n",
      "        [0.5000],\n",
      "        [0.5000]], grad_fn=<SigmoidBackward0>)\n",
      "torch.Size([6, 1])\n"
     ]
    }
   ],
   "source": [
    "print(hypothesis)\n",
    "print(hypothesis.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Vac6S29-LLqP"
   },
   "source": [
    "Q1. Logistic Regression의 Hypothesis에서 시그모이드 함수를 쓰는 이유는?\n",
    "\n",
    "=> 시그모이드는 0과 1 사이의 출력값을 갖기 때문에 이러한 특성을 이용해 classification을 할 수 있다. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "L8GQ25rwd1lk"
   },
   "source": [
    "## Computing the Cost Function (Low-level)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dGbHEzted1lk"
   },
   "source": [
    "$$ cost(W) = -\\frac{1}{m} \\sum y \\log\\left(H(x)\\right) + (1-y) \\left( \\log(1-H(x) \\right) $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IhR4AyY_d1lk"
   },
   "source": [
    "We want to measure the difference between `hypothesis` and `y_train`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "W8aeyBW-d1ll",
    "outputId": "bb9dc5c8-8947-4995-9cfd-2f3c601f6a1a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0.5000],\n",
      "        [0.5000],\n",
      "        [0.5000],\n",
      "        [0.5000],\n",
      "        [0.5000],\n",
      "        [0.5000]], grad_fn=<SigmoidBackward0>)\n",
      "tensor([[0.],\n",
      "        [0.],\n",
      "        [0.],\n",
      "        [1.],\n",
      "        [1.],\n",
      "        [1.]])\n"
     ]
    }
   ],
   "source": [
    "print(hypothesis)\n",
    "print(y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6qDvTFgWd1ln"
   },
   "source": [
    "For one element, the loss can be computed as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "8FkNyGYcd1lp",
    "outputId": "bc8787ff-a9e2-424b-c7f1-e003df0bc56a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0.6931], grad_fn=<NegBackward0>)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "-(y_train[0] * torch.log(hypothesis[0]) + \n",
    "  (1 - y_train[0]) * torch.log(1 - hypothesis[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eMr8Dgj4d1lr"
   },
   "source": [
    "To compute the losses for the entire batch, we can simply input the entire vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "i2TcW9Hxd1lt",
    "outputId": "fb55e647-ba96-4f3f-ea74-f8c65481916a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0.6931],\n",
      "        [0.6931],\n",
      "        [0.6931],\n",
      "        [0.6931],\n",
      "        [0.6931],\n",
      "        [0.6931]], grad_fn=<NegBackward0>)\n"
     ]
    }
   ],
   "source": [
    "losses = -(y_train * torch.log(hypothesis) + \n",
    "           (1 - y_train) * torch.log(1 - hypothesis))\n",
    "print(losses)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wq8A2ydcd1lv"
   },
   "source": [
    "Then, we just `.mean()` to take the mean of these individual losses."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "id": "v2QFnEZ3d1lw",
    "outputId": "3479e39d-2016-448d-a9ed-aa4eced6313a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(0.6931, grad_fn=<MeanBackward0>)\n"
     ]
    }
   ],
   "source": [
    "cost = losses.mean()\n",
    "print(cost)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LsWAiGcGMNTx"
   },
   "source": [
    "Q2. Logistic Regression의 cost function에서 MSE를 사용하지 못하는 이유는?\n",
    "\n",
    "=> global minimum 값이 아닌 local minimum값을 이용하게 되면 모델이 최적화가 되지 않기 때문."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yQgZi6vZd1lx"
   },
   "source": [
    "## Computing the Cost Function with `F.binary_cross_entropy`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YUJ7C4RMd1ly"
   },
   "source": [
    "In reality, binary classification is used so often that PyTorch has a simple function called `F.binary_cross_entropy` implemented to lighten the burden."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "id": "WYNPWmgpd1ly",
    "outputId": "f998441f-14eb-4049-9e52-039c8a138481"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(0.6931, grad_fn=<BinaryCrossEntropyBackward0>)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "F.binary_cross_entropy(hypothesis, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "u0C1m3skd1l7"
   },
   "source": [
    "## Training with Low-level Binary Cross Entropy Loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "id": "5AksFyHxd1l7"
   },
   "outputs": [],
   "source": [
    "x_data = [[1, 2], [2, 3], [3, 1], [4, 3], [5, 3], [6, 2]]\n",
    "y_data = [[0], [0], [0], [1], [1], [1]]\n",
    "x_train = torch.FloatTensor(x_data)\n",
    "y_train = torch.FloatTensor(y_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "rP1PhJ5xd1l8",
    "outputId": "66015548-a51c-43b7-eab4-15421a93a0db"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch    0/1000 Cost: 0.693147\n",
      "Epoch  100/1000 Cost: 0.692560\n",
      "Epoch  200/1000 Cost: 0.691977\n",
      "Epoch  300/1000 Cost: 0.691400\n",
      "Epoch  400/1000 Cost: 0.690829\n",
      "Epoch  500/1000 Cost: 0.690262\n",
      "Epoch  600/1000 Cost: 0.689701\n",
      "Epoch  700/1000 Cost: 0.689144\n",
      "Epoch  800/1000 Cost: 0.688593\n",
      "Epoch  900/1000 Cost: 0.688046\n",
      "Epoch 1000/1000 Cost: 0.687505\n"
     ]
    }
   ],
   "source": [
    "# 모델 초기화\n",
    "W = torch.zeros((2,1), requires_grad = True)#이 부분을 채워넣으세요.# #가중치 W를 0으로 초기화하고 학습을 통해 값이 변경될 수 있도록 하세요. x_train의 shape을 잘 생각해보세요!!\n",
    "b = torch.zeros(1, requires_grad = True)#이 부분을 채워넣으세요.# #가중치 W를 0으로 초기화하고 학습을 통해 값이 변경될 수 있도록 하세요.\n",
    "# optimizer 설정\n",
    "optimizer = optim.SGD([W,b], lr = 1e-5) #이 부분을 채워넣으세요.# #SGD optimizer를 사용하고 learning rate는 1로 적용하세요.\n",
    "\n",
    "nb_epochs = 1000\n",
    "for epoch in range(nb_epochs + 1):\n",
    "\n",
    "    # Cost 계산\n",
    "    hypothesis =torch.sigmoid(x_train.matmul(W) + b) \n",
    "    # or .mm or @ #torch.sigmoid를 사용하고 x_train의 shape이 달라졌기 때문에 행렬곱셉을 합니다!!\n",
    "    cost = -(y_train * torch.log(hypothesis) + (1 - y_train) * torch.log(1 - hypothesis)).mean()\n",
    "    # cost(W)=−1/m∑ylog(H(x))+(1−y)(log(1−H(x)))을 이용하세요.\n",
    "\n",
    "    # cost로 H(x) 개선\n",
    "    #이 부분을 채워넣으세요.#  # gradient를 0으로 초기화\n",
    "    #이 부분을 채워넣으세요.#  # 비용 함수를 미분하여 gradient 계산\n",
    "    #이 부분을 채워넣으세요.#  # W와 b를 업데이트\n",
    "    optimizer.zero_grad()\n",
    "    cost.backward()\n",
    "    optimizer.step()\n",
    "    \n",
    "\n",
    "    # 100번마다 로그 출력\n",
    "    if epoch % 100 == 0:\n",
    "        print('Epoch {:4d}/{} Cost: {:.6f}'.format(\n",
    "            epoch, nb_epochs, cost.item()\n",
    "        ))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OziCLRLkd1l-"
   },
   "source": [
    "## Training with `F.binary_cross_entropy`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "id": "zJdnMYKRd1l-",
    "outputId": "6831cccf-b130-43f0-ef55-3f57577923e2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch    0/1000 Cost: 0.693147\n",
      "Epoch  100/1000 Cost: 0.134722\n",
      "Epoch  200/1000 Cost: 0.080643\n",
      "Epoch  300/1000 Cost: 0.057900\n",
      "Epoch  400/1000 Cost: 0.045300\n",
      "Epoch  500/1000 Cost: 0.037261\n",
      "Epoch  600/1000 Cost: 0.031672\n",
      "Epoch  700/1000 Cost: 0.027556\n",
      "Epoch  800/1000 Cost: 0.024394\n",
      "Epoch  900/1000 Cost: 0.021888\n",
      "Epoch 1000/1000 Cost: 0.019852\n"
     ]
    }
   ],
   "source": [
    "# 모델 초기화\n",
    "W = torch.zeros((2,1), requires_grad = True)#이 부분을 채워넣으세요.# #가중치 W를 0으로 초기화하고 학습을 통해 값이 변경될 수 있도록 하세요. x_train의 shape을 잘 생각해보세요!!\n",
    "b = torch.zeros(1, requires_grad = True)#이 부분을 채워넣으세요.# #가중치 W를 0으로 초기화하고 학습을 통해 값이 변경될 수 있도록 하세요.\n",
    "# optimizer 설정\n",
    "optimizer = optim.SGD([W,b], lr = 1) #이 부분을 채워넣으세요.# #SGD optimizer를 사용하고 learning rate는 1로 적용하세요.\n",
    "\n",
    "nb_epochs = 1000\n",
    "for epoch in range(nb_epochs + 1):\n",
    "\n",
    "    # Cost 계산\n",
    "    hypothesis = torch.sigmoid(x_train.matmul(W) + b)#이 부분을 채워넣으세요.# # or .mm or @ #torch.sigmoid를 사용하고 x_train의 shape이 달라졌기 때문에 행렬곱셉을 합니다!!\n",
    "    cost = F.binary_cross_entropy(hypothesis, y_train) #이 부분을 채워넣으세요.# #PyTorch에서 제공하는 binary_cross_entropy 함수를 사용하세요.\n",
    "\n",
    "    # cost로 H(x) 개선\n",
    "    #이 부분을 채워넣으세요.#  # gradient를 0으로 초기화\n",
    "    #이 부분을 채워넣으세요.#  # 비용 함수를 미분하여 gradient 계산\n",
    "    #이 부분을 채워넣으세요.#  # W와 b를 업데이트\n",
    "    \n",
    "    optimizer.zero_grad()\n",
    "    cost.backward()\n",
    "    optimizer.step()\n",
    "    \n",
    "    # 100번마다 로그 출력\n",
    "    if epoch % 100 == 0:\n",
    "        print('Epoch {:4d}/{} Cost: {:.6f}'.format(\n",
    "            epoch, nb_epochs, cost.item()\n",
    "        ))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CGxZzc-id1l_"
   },
   "source": [
    "## Loading Real Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qymF9ZiIYhKv"
   },
   "source": [
    "data-03-diabetes.csv 파일 사용!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "id": "RVvbZBiWd1l_"
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "C1cp2Hy_gT3w"
   },
   "outputs": [],
   "source": [
    "from google.colab import drive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "9kBU6ttYgT5W",
    "outputId": "fde353d7-9aa2-4ff3-e9a5-d67bdf70c568"
   },
   "outputs": [],
   "source": [
    "drive.mount('/content/drive')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "id": "kfvVhaDvd1mA"
   },
   "outputs": [],
   "source": [
    "xy = np.loadtxt('data-03-diabetes.csv', delimiter=',', dtype=np.float32)\n",
    "x_data = xy[:, 0:-1]\n",
    "y_data = xy[:, [-1]]\n",
    "x_train = torch.FloatTensor(x_data)\n",
    "y_train = torch.FloatTensor(y_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "9MNmGRG4d1mA",
    "outputId": "5b0e1f2b-182d-43a2-82d9-8bfb4b1e96dd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-0.2941,  0.4874,  0.1803, -0.2929,  0.0000,  0.0015, -0.5312, -0.0333],\n",
      "        [-0.8824, -0.1457,  0.0820, -0.4141,  0.0000, -0.2072, -0.7669, -0.6667],\n",
      "        [-0.0588,  0.8392,  0.0492,  0.0000,  0.0000, -0.3055, -0.4927, -0.6333],\n",
      "        [-0.8824, -0.1055,  0.0820, -0.5354, -0.7778, -0.1624, -0.9240,  0.0000],\n",
      "        [ 0.0000,  0.3769, -0.3443, -0.2929, -0.6028,  0.2846,  0.8873, -0.6000]])\n",
      "tensor([[0.],\n",
      "        [1.],\n",
      "        [0.],\n",
      "        [1.],\n",
      "        [0.]])\n"
     ]
    }
   ],
   "source": [
    "print(x_train[0:5])\n",
    "print(y_train[0:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vX7BmUzLd1mB"
   },
   "source": [
    "## Training with Real Data using low-level Binary Cross Entropy Loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "id": "mFDd3uB0d1mB",
    "outputId": "bad5e35c-d39b-4644-f31d-2f15404aad22"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch    0/100 Cost: 0.693147\n",
      "Epoch   10/100 Cost: 0.572727\n",
      "Epoch   20/100 Cost: 0.539493\n",
      "Epoch   30/100 Cost: 0.519708\n",
      "Epoch   40/100 Cost: 0.507066\n",
      "Epoch   50/100 Cost: 0.498539\n",
      "Epoch   60/100 Cost: 0.492549\n",
      "Epoch   70/100 Cost: 0.488209\n",
      "Epoch   80/100 Cost: 0.484985\n",
      "Epoch   90/100 Cost: 0.482543\n",
      "Epoch  100/100 Cost: 0.480661\n"
     ]
    }
   ],
   "source": [
    "# 모델 초기화\n",
    "W = torch.zeros((8,1), requires_grad = True) #이 부분을 채워넣으세요.# #가중치 W를 0으로 초기화하고 학습을 통해 값이 변경될 수 있도록 하세요. x_train의 shape을 잘 생각해보세요!!\n",
    "b = torch.zeros(1, requires_grad = True)#이 부분을 채워넣으세요.# #가중치 W를 0으로 초기화하고 학습을 통해 값이 변경될 수 있도록 하세요.\n",
    "# optimizer 설정\n",
    "optimizer = optim.SGD([W,b], lr = 1)#이 부분을 채워넣으세요.# #SGD optimizer를 사용하고 learning rate는 1로 적용하세요.\n",
    "\n",
    "nb_epochs = 100\n",
    "for epoch in range(nb_epochs + 1):\n",
    "\n",
    "    # Cost 계산\n",
    "    hypothesis = torch.sigmoid(x_train.matmul(W) + b)#이 부분을 채워넣으세요.# # or .mm or @ #torch.sigmoid를 사용하고 x_train의 shape이 달라졌기 때문에 행렬곱셉을 합니다!!\n",
    "    cost = -(y_train * torch.log(hypothesis) + (1 - y_train) * torch.log(1 - hypothesis)).mean()#이 부분을 채워넣으세요.# # cost(W)=−1/m∑ylog(H(x))+(1−y)(log(1−H(x)))을 이용하세요.\n",
    "\n",
    "    # cost로 H(x) 개선\n",
    "    #이 부분을 채워넣으세요.#  # gradient를 0으로 초기화\n",
    "    #이 부분을 채워넣으세요.#  # 비용 함수를 미분하여 gradient 계산\n",
    "    #이 부분을 채워넣으세요.#  # W와 b를 업데이트\n",
    "    optimizer.zero_grad()\n",
    "    cost.backward()\n",
    "    optimizer.step()\n",
    "    \n",
    "    # 10번마다 로그 출력\n",
    "    if epoch % 10 == 0:\n",
    "        print('Epoch {:4d}/{} Cost: {:.6f}'.format(\n",
    "            epoch, nb_epochs, cost.item()\n",
    "        ))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_IqYdBJKd1mC"
   },
   "source": [
    "## Training with Real Data using `F.binary_cross_entropy`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "id": "kS4yfIf_d1mC",
    "outputId": "30d1982e-447e-427e-ebaf-7c5c94f589c0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch    0/100 Cost: 0.693147\n",
      "Epoch   10/100 Cost: 0.572727\n",
      "Epoch   20/100 Cost: 0.539493\n",
      "Epoch   30/100 Cost: 0.519708\n",
      "Epoch   40/100 Cost: 0.507066\n",
      "Epoch   50/100 Cost: 0.498539\n",
      "Epoch   60/100 Cost: 0.492549\n",
      "Epoch   70/100 Cost: 0.488209\n",
      "Epoch   80/100 Cost: 0.484985\n",
      "Epoch   90/100 Cost: 0.482543\n",
      "Epoch  100/100 Cost: 0.480661\n"
     ]
    }
   ],
   "source": [
    "# 모델 초기화\n",
    "W = torch.zeros((8,1), requires_grad = True) #이 부분을 채워넣으세요.# #가중치 W를 0으로 초기화하고 학습을 통해 값이 변경될 수 있도록 하세요. x_train의 shape을 잘 생각해보세요!!\n",
    "b = torch.zeros(1, requires_grad =True) #이 부분을 채워넣으세요.# #가중치 W를 0으로 초기화하고 학습을 통해 값이 변경될 수 있도록 하세요.\n",
    "# optimizer 설정\n",
    "optimizer = optim.SGD([W, b], lr = 1) #SGD optimizer를 사용하고 learning rate는 1로 적용하세요.\n",
    "\n",
    "nb_epochs = 100\n",
    "for epoch in range(nb_epochs + 1):\n",
    "\n",
    "    # Cost 계산\n",
    "    hypothesis = torch.sigmoid(x_train.matmul(W) + b)#이 부분을 채워넣으세요.# # or .mm or @ #torch.sigmoid를 사용하고 x_train의 shape이 달라졌기 때문에 행렬곱셉을 합니다!!\n",
    "    cost = F.binary_cross_entropy(hypothesis, y_train)#PyTorch에서 제공하는 binary_cross_entropy 함수를 사용하세요.\n",
    "\n",
    "    # cost로 H(x) 개선\n",
    "    #이 부분을 채워넣으세요.#  # gradient를 0으로 초기화\n",
    "    #이 부분을 채워넣으세요.#  # 비용 함수를 미분하여 gradient 계산\n",
    "    #이 부분을 채워넣으세요.#  # W와 b를 업데이트\n",
    "    optimizer.zero_grad()\n",
    "    cost.backward()\n",
    "    optimizer.step()\n",
    "    \n",
    "    # 10번마다 로그 출력\n",
    "    if epoch % 10 == 0:\n",
    "        print('Epoch {:4d}/{} Cost: {:.6f}'.format(\n",
    "            epoch, nb_epochs, cost.item()\n",
    "        ))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HuqdMlhjd1mD"
   },
   "source": [
    "## Checking the Accuracy our our Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "X_YgqECVd1mD"
   },
   "source": [
    "After we finish training the model, we want to check how well our model fits the training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "id": "ETC0bLT4d1mE",
    "outputId": "d27aefd7-7cac-40cb-ff57-7724ebc26c9e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0.4103],\n",
      "        [0.9242],\n",
      "        [0.2300],\n",
      "        [0.9411],\n",
      "        [0.1772]], grad_fn=<SliceBackward0>)\n"
     ]
    }
   ],
   "source": [
    "hypothesis = torch.sigmoid(x_train.matmul(W) + b)\n",
    "print(hypothesis[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Oc6V0StDd1mE"
   },
   "source": [
    "We can change **hypothesis** (real number from 0 to 1) to **binary predictions** (either 0 or 1) by comparing them to 0.5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "id": "hg0Ow208d1mE",
    "outputId": "da327c43-95d6-4d02-d2d4-6e3e23945bfc"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[False],\n",
      "        [ True],\n",
      "        [False],\n",
      "        [ True],\n",
      "        [False]])\n"
     ]
    }
   ],
   "source": [
    "prediction = hypothesis >= torch.FloatTensor([0.5])\n",
    "print(prediction[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xFDxcdtUd1mF"
   },
   "source": [
    "Then, we compare it with the correct labels `y_train`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "id": "JdSHsInid1mF",
    "outputId": "12302be5-08dc-4c0d-bbdc-5a02464d8e6d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[False],\n",
      "        [ True],\n",
      "        [False],\n",
      "        [ True],\n",
      "        [False]])\n",
      "tensor([[0.],\n",
      "        [1.],\n",
      "        [0.],\n",
      "        [1.],\n",
      "        [0.]])\n"
     ]
    }
   ],
   "source": [
    "print(prediction[:5])\n",
    "print(y_train[:5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "id": "hZGytd8Gd1mF",
    "outputId": "caa6535e-e618-4a52-e91a-439cb0f4932d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[True],\n",
      "        [True],\n",
      "        [True],\n",
      "        [True],\n",
      "        [True]])\n"
     ]
    }
   ],
   "source": [
    "correct_prediction = prediction.float() == y_train\n",
    "print(correct_prediction[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IyybcRe4d1mH"
   },
   "source": [
    "Finally, we can calculate the accuracy by counting the number of correct predictions and dividng by total number of predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "id": "IczKlmbZd1mH",
    "outputId": "9c4e5f43-27d2-4f26-9c19-950ed53c4fb0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The model has an accuracy of 76.68% for the training set.\n"
     ]
    }
   ],
   "source": [
    "accuracy = correct_prediction.sum().item() / len(correct_prediction)\n",
    "print('The model has an accuracy of {:2.2f}% for the training set.'.format(accuracy * 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RmnNUiVDd1mI"
   },
   "source": [
    "## Optional: High-level Implementation with `nn.Module`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "id": "Pej07LDBd1mI"
   },
   "outputs": [],
   "source": [
    "class BinaryClassifier(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.linear = nn.Linear(8, 1)\n",
    "        self.sigmoid = nn.Sigmoid()\n",
    "\n",
    "    def forward(self, x):\n",
    "        return self.sigmoid(self.linear(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "id": "5WcFoEPFd1mJ"
   },
   "outputs": [],
   "source": [
    "model = BinaryClassifier() #이 부분을 채워넣으세요.# #BinaryClassifier()로 모델을 초기화하세요."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "id": "gyNYPiond1mK",
    "outputId": "6f080fed-bc3c-441f-a2de-2eb3fdaa04fe"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch    0/100 Cost: 0.704829 Accuracy 45.72%\n",
      "Epoch   10/100 Cost: 0.572392 Accuracy 67.59%\n",
      "Epoch   20/100 Cost: 0.539564 Accuracy 73.25%\n",
      "Epoch   30/100 Cost: 0.520041 Accuracy 75.89%\n",
      "Epoch   40/100 Cost: 0.507561 Accuracy 76.15%\n",
      "Epoch   50/100 Cost: 0.499125 Accuracy 76.42%\n",
      "Epoch   60/100 Cost: 0.493177 Accuracy 77.21%\n",
      "Epoch   70/100 Cost: 0.488846 Accuracy 76.81%\n",
      "Epoch   80/100 Cost: 0.485612 Accuracy 76.28%\n",
      "Epoch   90/100 Cost: 0.483146 Accuracy 76.55%\n",
      "Epoch  100/100 Cost: 0.481234 Accuracy 76.81%\n"
     ]
    }
   ],
   "source": [
    "# optimizer 설정\n",
    "optimizer = optim.SGD(model.parameters(), lr = 1) #이 부분을 채워넣으세요.# #SGD optimizer를 사용하고 learning rate는 1로 적용하세요.\n",
    "\n",
    "nb_epochs = 100\n",
    "for epoch in range(nb_epochs + 1):\n",
    "\n",
    "    # H(x) 계산\n",
    "    hypothesis = model(x_train) #이 부분을 채워넣으세요.# #모델을 생성해서 예측값을 구해보세요.\n",
    "\n",
    "    # cost 계산\n",
    "    cost = F.binary_cross_entropy(hypothesis, y_train) #이 부분을 채워넣으세요.# #PyTorch에서 제공하는 binary_cross_entropy 함수를 사용하세요.\n",
    "\n",
    "    # cost로 H(x) 개선\n",
    "    #이 부분을 채워넣으세요.#  # gradient를 0으로 초기화\n",
    "    #이 부분을 채워넣으세요.#  # 비용 함수를 미분하여 gradient 계산\n",
    "    #이 부분을 채워넣으세요.#  # W와 b를 업데이트\n",
    "    optimizer.zero_grad()\n",
    "    cost.backward()\n",
    "    optimizer.step()\n",
    "    \n",
    "    # 20번마다 로그 출력\n",
    "    if epoch % 10 == 0:\n",
    "        prediction = hypothesis >= torch.FloatTensor([0.5])\n",
    "        correct_prediction = prediction.float() == y_train\n",
    "        accuracy = correct_prediction.sum().item() / len(correct_prediction)\n",
    "        print('Epoch {:4d}/{} Cost: {:.6f} Accuracy {:2.2f}%'.format(\n",
    "            epoch, nb_epochs, cost.item(), accuracy * 100,\n",
    "        ))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [
    "u0C1m3skd1l7",
    "OziCLRLkd1l-"
   ],
   "name": "Lab05.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}