Lab05_final.ipynb

{"cells":[{"cell_type":"markdown","metadata":{"id":"OWoEjnmfd1lG"},"source":["# Lab 5: Logistic Classification"]},{"cell_type":"markdown","metadata":{"id":"jYgrJaSBd1lN"},"source":["Author: Seungjae Lee (이승재)"]},{"cell_type":"markdown","metadata":{"id":"fX12twYRd1lO"},"source":["<div class=\"alert alert-warning\">\n","    We use elemental PyTorch to implement linear regression here. However, in most actual applications, abstractions such as <code>nn.Module</code> or <code>nn.Linear</code> are used. You can see those implementations near the end of this notebook.\n","</div>"]},{"cell_type":"markdown","metadata":{"id":"bao6VR5Pd1lP"},"source":["## Reminder: Logistic Regression"]},{"cell_type":"markdown","metadata":{"id":"lKvotCYed1lP"},"source":["### Hypothesis"]},{"cell_type":"markdown","metadata":{"id":"HD-S293ed1lQ"},"source":["$$ H(X) = \\frac{1}{1+e^{-W^T X}} $$"]},{"cell_type":"markdown","metadata":{"id":"DQLeTc5hd1lQ"},"source":["### Cost"]},{"cell_type":"markdown","metadata":{"id":"PBIIU057d1lR"},"source":["$$ cost(W) = -\\frac{1}{m} \\sum y \\log\\left(H(x)\\right) + (1-y) \\left( \\log(1-H(x) \\right) $$"]},{"cell_type":"markdown","metadata":{"id":"xpTGq6kld1lS"},"source":[" - If $y \\simeq H(x)$, cost is near 0.\n"," - If $y \\neq H(x)$, cost is high."]},{"cell_type":"markdown","metadata":{"id":"zVdnlofcd1lW"},"source":["### Weight Update via Gradient Descent"]},{"cell_type":"markdown","metadata":{"id":"-Ok5jeK3d1lY"},"source":["$$ W := W - \\alpha \\frac{\\partial}{\\partial W} cost(W) $$"]},{"cell_type":"markdown","metadata":{"id":"jnNit8did1lY"},"source":[" - $\\alpha$: Learning rate"]},{"cell_type":"markdown","metadata":{"id":"r0i_bbozd1lZ"},"source":["## Imports"]},{"cell_type":"code","execution_count":1,"metadata":{"id":"6GNFdC7Kd1la","executionInfo":{"status":"ok","timestamp":1648042583582,"user_tz":-540,"elapsed":7080,"user":{"displayName":"‍홍소연[학생](전자정보대학 전자공학과)","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"03502202636524032613"}}},"outputs":[],"source":["import torch\n","import torch.nn as nn\n","import torch.nn.functional as F\n","import torch.optim as optim"]},{"cell_type":"code","execution_count":2,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"XiHiXkg0d1lb","outputId":"5bdcf5c7-6684-41e0-8622-7cf67a96c132","executionInfo":{"status":"ok","timestamp":1648042583583,"user_tz":-540,"elapsed":10,"user":{"displayName":"‍홍소연[학생](전자정보대학 전자공학과)","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"03502202636524032613"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["<torch._C.Generator at 0x7f1743f4ceb0>"]},"metadata":{},"execution_count":2}],"source":["# For reproducibility\n","torch.manual_seed(1)"]},{"cell_type":"markdown","metadata":{"id":"-hbwRxSFd1lc"},"source":["## Training Data"]},{"cell_type":"code","execution_count":3,"metadata":{"id":"S6sdFKP5d1ld","executionInfo":{"status":"ok","timestamp":1648042583584,"user_tz":-540,"elapsed":8,"user":{"displayName":"‍홍소연[학생](전자정보대학 전자공학과)","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"03502202636524032613"}}},"outputs":[],"source":["x_data = [[1, 2], [2, 3], [3, 1], [4, 3], [5, 3], [6, 2]]\n","y_data = [[0], [0], [0], [1], [1], [1]]"]},{"cell_type":"markdown","metadata":{"id":"ISorVWoSd1ld"},"source":["Consider the following classification problem: given the number of hours each student spent watching the lecture and working in the code lab, predict whether the student passed or failed a course. For example, the first (index 0) student watched the lecture for 1 hour and spent 2 hours in the lab session ([1, 2]), and ended up failing the course ([0])."]},{"cell_type":"code","execution_count":4,"metadata":{"id":"lallaxkPd1le","executionInfo":{"status":"ok","timestamp":1648042583584,"user_tz":-540,"elapsed":8,"user":{"displayName":"‍홍소연[학생](전자정보대학 전자공학과)","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"03502202636524032613"}}},"outputs":[],"source":["x_train = torch.FloatTensor(x_data)\n","y_train = torch.FloatTensor(y_data)"]},{"cell_type":"markdown","metadata":{"id":"rM89auFWd1le"},"source":["As always, we need these data to be in `torch.Tensor` format, so we convert them."]},{"cell_type":"code","execution_count":5,"metadata":{"id":"N9HM9pUGd1lf","outputId":"5559d9d7-f77f-4dfc-9628-2c85d594f165","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1648042583584,"user_tz":-540,"elapsed":8,"user":{"displayName":"‍홍소연[학생](전자정보대학 전자공학과)","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"03502202636524032613"}}},"outputs":[{"output_type":"stream","name":"stdout","text":["torch.Size([6, 2])\n","torch.Size([6, 1])\n"]}],"source":["print(x_train.shape)\n","print(y_train.shape)"]},{"cell_type":"markdown","metadata":{"id":"rTdVp92Dd1lf"},"source":["## Computing the Hypothesis"]},{"cell_type":"markdown","metadata":{"id":"n4RqCyp1d1lf"},"source":["$$ H(X) = \\frac{1}{1+e^{-W^T X}} $$"]},{"cell_type":"markdown","metadata":{"id":"Bo1a4R8sd1lg"},"source":["PyTorch has a `torch.exp()` function that resembles the exponential function."]},{"cell_type":"code","execution_count":6,"metadata":{"id":"9iDhEXmVd1lg","outputId":"2d87e525-ea6a-4e6c-c746-98cb37f302b4","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1648042583585,"user_tz":-540,"elapsed":6,"user":{"displayName":"‍홍소연[학생](전자정보대학 전자공학과)","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"03502202636524032613"}}},"outputs":[{"output_type":"stream","name":"stdout","text":["e^1 equals:  tensor([2.7183])\n"]}],"source":["print('e^1 equals: ', torch.exp(torch.FloatTensor([1])))"]},{"cell_type":"markdown","metadata":{"id":"_1I9zt17d1lh"},"source":["We can use it to compute the hypothesis function conveniently."]},{"cell_type":"code","execution_count":7,"metadata":{"id":"aQqWIqoLd1lh","executionInfo":{"status":"ok","timestamp":1648042584040,"user_tz":-540,"elapsed":460,"user":{"displayName":"‍홍소연[학생](전자정보대학 전자공학과)","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"03502202636524032613"}}},"outputs":[],"source":["W = torch.zeros((2, 1), requires_grad=True)\n","b = torch.zeros(1, requires_grad=True)"]},{"cell_type":"code","execution_count":8,"metadata":{"id":"A7r3kFFvd1lh","executionInfo":{"status":"ok","timestamp":1648042584041,"user_tz":-540,"elapsed":20,"user":{"displayName":"‍홍소연[학생](전자정보대학 전자공학과)","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"03502202636524032613"}}},"outputs":[],"source":["hypothesis = 1 / (1 + torch.exp(-(x_train.matmul(W) + b)))"]},{"cell_type":"code","execution_count":9,"metadata":{"id":"Z1-oWTp5d1lh","outputId":"f29dcac5-5d33-4d45-cffd-bf5099614862","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1648042584041,"user_tz":-540,"elapsed":19,"user":{"displayName":"‍홍소연[학생](전자정보대학 전자공학과)","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"03502202636524032613"}}},"outputs":[{"output_type":"stream","name":"stdout","text":["tensor([[0.5000],\n","        [0.5000],\n","        [0.5000],\n","        [0.5000],\n","        [0.5000],\n","        [0.5000]], grad_fn=<MulBackward0>)\n","torch.Size([6, 1])\n"]}],"source":["print(hypothesis)\n","print(hypothesis.shape)"]},{"cell_type":"markdown","metadata":{"id":"hxJO6JfKd1li"},"source":["Or, we could use `torch.sigmoid()` function! This resembles the sigmoid function:"]},{"cell_type":"code","execution_count":10,"metadata":{"id":"XSyZNfPhd1li","outputId":"779f7b53-8998-4b17-ff48-e80e2b3cf636","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1648042584041,"user_tz":-540,"elapsed":18,"user":{"displayName":"‍홍소연[학생](전자정보대학 전자공학과)","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"03502202636524032613"}}},"outputs":[{"output_type":"stream","name":"stdout","text":["1/(1+e^{-1}) equals:  tensor([0.7311])\n"]}],"source":["print('1/(1+e^{-1}) equals: ', torch.sigmoid(torch.FloatTensor([1])))"]},{"cell_type":"markdown","metadata":{"id":"D_ZfiY7cd1lj"},"source":["Now, the code for hypothesis function is cleaner."]},{"cell_type":"code","execution_count":11,"metadata":{"id":"B9Gvs6phd1lj","executionInfo":{"status":"ok","timestamp":1648042584042,"user_tz":-540,"elapsed":17,"user":{"displayName":"‍홍소연[학생](전자정보대학 전자공학과)","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"03502202636524032613"}}},"outputs":[],"source":["hypothesis = torch.sigmoid(x_train.matmul(W) + b)"]},{"cell_type":"code","execution_count":12,"metadata":{"id":"vE8rzSZxd1lj","outputId":"12e835a1-60c1-474f-d2e1-a30684f3eef3","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1648042584042,"user_tz":-540,"elapsed":17,"user":{"displayName":"‍홍소연[학생](전자정보대학 전자공학과)","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"03502202636524032613"}}},"outputs":[{"output_type":"stream","name":"stdout","text":["tensor([[0.5000],\n","        [0.5000],\n","        [0.5000],\n","        [0.5000],\n","        [0.5000],\n","        [0.5000]], grad_fn=<SigmoidBackward0>)\n","torch.Size([6, 1])\n"]}],"source":["print(hypothesis)\n","print(hypothesis.shape)"]},{"cell_type":"markdown","metadata":{"id":"Vac6S29-LLqP"},"source":["Q1. Logistic Regression의 Hypothesis에서 시그모이드 함수를 쓰는 이유는?\n","\n","=> 이진 분류에서는 일차함수보다 S자 형태인 시그모이드 함수를 통해 classification이 수월하게 이뤄지기 때문"]},{"cell_type":"markdown","metadata":{"id":"L8GQ25rwd1lk"},"source":["## Computing the Cost Function (Low-level)"]},{"cell_type":"markdown","metadata":{"id":"dGbHEzted1lk"},"source":["$$ cost(W) = -\\frac{1}{m} \\sum y \\log\\left(H(x)\\right) + (1-y) \\left( \\log(1-H(x) \\right) $$"]},{"cell_type":"markdown","metadata":{"id":"IhR4AyY_d1lk"},"source":["We want to measure the difference between `hypothesis` and `y_train`."]},{"cell_type":"code","execution_count":13,"metadata":{"id":"W8aeyBW-d1ll","outputId":"3b512fb2-96ee-4c60-8bb0-b93ec23c703e","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1648042584042,"user_tz":-540,"elapsed":15,"user":{"displayName":"‍홍소연[학생](전자정보대학 전자공학과)","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"03502202636524032613"}}},"outputs":[{"output_type":"stream","name":"stdout","text":["tensor([[0.5000],\n","        [0.5000],\n","        [0.5000],\n","        [0.5000],\n","        [0.5000],\n","        [0.5000]], grad_fn=<SigmoidBackward0>)\n","tensor([[0.],\n","        [0.],\n","        [0.],\n","        [1.],\n","        [1.],\n","        [1.]])\n"]}],"source":["print(hypothesis)\n","print(y_train)"]},{"cell_type":"markdown","metadata":{"id":"6qDvTFgWd1ln"},"source":["For one element, the loss can be computed as follows:"]},{"cell_type":"code","execution_count":14,"metadata":{"id":"8FkNyGYcd1lp","outputId":"7c732e69-1487-4a29-91b2-b609306649c0","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1648042584043,"user_tz":-540,"elapsed":15,"user":{"displayName":"‍홍소연[학생](전자정보대학 전자공학과)","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"03502202636524032613"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["tensor([0.6931], grad_fn=<NegBackward0>)"]},"metadata":{},"execution_count":14}],"source":["-(y_train[0] * torch.log(hypothesis[0]) + \n","  (1 - y_train[0]) * torch.log(1 - hypothesis[0]))"]},{"cell_type":"markdown","metadata":{"id":"eMr8Dgj4d1lr"},"source":["To compute the losses for the entire batch, we can simply input the entire vector."]},{"cell_type":"code","execution_count":15,"metadata":{"id":"i2TcW9Hxd1lt","outputId":"8d9fd39c-0675-49bb-f7ed-7fa6d4544f90","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1648042584043,"user_tz":-540,"elapsed":14,"user":{"displayName":"‍홍소연[학생](전자정보대학 전자공학과)","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"03502202636524032613"}}},"outputs":[{"output_type":"stream","name":"stdout","text":["tensor([[0.6931],\n","        [0.6931],\n","        [0.6931],\n","        [0.6931],\n","        [0.6931],\n","        [0.6931]], grad_fn=<NegBackward0>)\n"]}],"source":["losses = -(y_train * torch.log(hypothesis) + \n","           (1 - y_train) * torch.log(1 - hypothesis))\n","print(losses)"]},{"cell_type":"markdown","metadata":{"id":"wq8A2ydcd1lv"},"source":["Then, we just `.mean()` to take the mean of these individual losses."]},{"cell_type":"code","execution_count":16,"metadata":{"id":"v2QFnEZ3d1lw","outputId":"453e187d-02ee-43dc-db26-f08fe4faad97","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1648042584043,"user_tz":-540,"elapsed":12,"user":{"displayName":"‍홍소연[학생](전자정보대학 전자공학과)","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"03502202636524032613"}}},"outputs":[{"output_type":"stream","name":"stdout","text":["tensor(0.6931, grad_fn=<MeanBackward0>)\n"]}],"source":["cost = losses.mean()\n","print(cost)"]},{"cell_type":"markdown","metadata":{"id":"LsWAiGcGMNTx"},"source":["Q2. Logistic Regression의 cost function에서 MSE를 사용하지 못하는 이유는?\n","\n","=> 심한 비볼록 형태의 그래프가 나오며 오차를 잘못 판단할 가능성이 커짐"]},{"cell_type":"markdown","metadata":{"id":"yQgZi6vZd1lx"},"source":["## Computing the Cost Function with `F.binary_cross_entropy`"]},{"cell_type":"markdown","metadata":{"id":"YUJ7C4RMd1ly"},"source":["In reality, binary classification is used so often that PyTorch has a simple function called `F.binary_cross_entropy` implemented to lighten the burden."]},{"cell_type":"code","execution_count":17,"metadata":{"id":"WYNPWmgpd1ly","outputId":"981624a3-f7cf-47ce-adef-b61ed4199951","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1648042584043,"user_tz":-540,"elapsed":9,"user":{"displayName":"‍홍소연[학생](전자정보대학 전자공학과)","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"03502202636524032613"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["tensor(0.6931, grad_fn=<BinaryCrossEntropyBackward0>)"]},"metadata":{},"execution_count":17}],"source":["F.binary_cross_entropy(hypothesis, y_train)"]},{"cell_type":"markdown","metadata":{"id":"u0C1m3skd1l7"},"source":["## Training with Low-level Binary Cross Entropy Loss"]},{"cell_type":"code","execution_count":18,"metadata":{"id":"5AksFyHxd1l7","executionInfo":{"status":"ok","timestamp":1648042584044,"user_tz":-540,"elapsed":9,"user":{"displayName":"‍홍소연[학생](전자정보대학 전자공학과)","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"03502202636524032613"}}},"outputs":[],"source":["x_data = [[1, 2], [2, 3], [3, 1], [4, 3], [5, 3], [6, 2]]\n","y_data = [[0], [0], [0], [1], [1], [1]]\n","x_train = torch.FloatTensor(x_data)\n","y_train = torch.FloatTensor(y_data)"]},{"cell_type":"code","execution_count":19,"metadata":{"id":"rP1PhJ5xd1l8","outputId":"8d578073-d146-4da3-f18f-47cd5ed36b15","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1648042584629,"user_tz":-540,"elapsed":594,"user":{"displayName":"‍홍소연[학생](전자정보대학 전자공학과)","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"03502202636524032613"}}},"outputs":[{"output_type":"stream","name":"stdout","text":["Epoch    0/1000 Cost: 0.693147\n","Epoch  100/1000 Cost: 0.134722\n","Epoch  200/1000 Cost: 0.080643\n","Epoch  300/1000 Cost: 0.057900\n","Epoch  400/1000 Cost: 0.045300\n","Epoch  500/1000 Cost: 0.037261\n","Epoch  600/1000 Cost: 0.031673\n","Epoch  700/1000 Cost: 0.027556\n","Epoch  800/1000 Cost: 0.024394\n","Epoch  900/1000 Cost: 0.021888\n","Epoch 1000/1000 Cost: 0.019852\n"]}],"source":["# 모델 초기화\n","W = torch.zeros((2,1),requires_grad=True) #가중치 W를 0으로 초기화하고 학습을 통해 값이 변경될 수 있도록 하세요. x_train의 shape을 잘 생각해보세요!!\n","b = torch.zeros(1,requires_grad=True) #가중치 W를 0으로 초기화하고 학습을 통해 값이 변경될 수 있도록 하세요.\n","# optimizer 설정\n","optimizer = optim.SGD([W,b],lr=1) #SGD optimizer를 사용하고 learning rate는 1로 적용하세요.\n","\n","nb_epochs = 1000\n","for epoch in range(nb_epochs + 1):\n","\n","    # Cost 계산\n","    hypothesis =torch.sigmoid(x_train.matmul(W)+b) # or .mm or @ #torch.sigmoid를 사용하고 x_train의 shape이 달라졌기 때문에 행렬곱셉을 합니다!!\n","    cost = -(y_train*torch.log(hypothesis)+(1-y_train)*torch.log(1-hypothesis)).mean()\n","             # cost(W)=−1/m∑ylog(H(x))+(1−y)(log(1−H(x)))을 이용하세요.\n","\n","    # cost로 H(x) 개선\n","    optimizer.zero_grad()  # gradient를 0으로 초기화\n","    cost.backward()  # 비용 함수를 미분하여 gradient 계산\n","    optimizer.step()  # W와 b를 업데이트\n","\n","\n","    # 100번마다 로그 출력\n","    if epoch % 100 == 0:\n","        print('Epoch {:4d}/{} Cost: {:.6f}'.format(\n","            epoch, nb_epochs, cost.item()\n","        ))"]},{"cell_type":"markdown","metadata":{"id":"OziCLRLkd1l-"},"source":["## Training with `F.binary_cross_entropy`"]},{"cell_type":"code","execution_count":20,"metadata":{"id":"zJdnMYKRd1l-","outputId":"941b638a-c173-4c96-ad89-87c8d8d9e30a","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1648042585082,"user_tz":-540,"elapsed":458,"user":{"displayName":"‍홍소연[학생](전자정보대학 전자공학과)","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"03502202636524032613"}}},"outputs":[{"output_type":"stream","name":"stdout","text":["Epoch    0/1000 Cost: 0.693147\n","Epoch  100/1000 Cost: 0.134722\n","Epoch  200/1000 Cost: 0.080643\n","Epoch  300/1000 Cost: 0.057900\n","Epoch  400/1000 Cost: 0.045300\n","Epoch  500/1000 Cost: 0.037261\n","Epoch  600/1000 Cost: 0.031672\n","Epoch  700/1000 Cost: 0.027556\n","Epoch  800/1000 Cost: 0.024394\n","Epoch  900/1000 Cost: 0.021888\n","Epoch 1000/1000 Cost: 0.019852\n"]}],"source":["# 모델 초기화\n","W = torch.zeros((2,1),requires_grad=True) #가중치 W를 0으로 초기화하고 학습을 통해 값이 변경될 수 있도록 하세요. x_train의 shape을 잘 생각해보세요!!\n","b = torch.zeros(1,requires_grad=True) #가중치 W를 0으로 초기화하고 학습을 통해 값이 변경될 수 있도록 하세요.\n","# optimizer 설정\n","optimizer = optim.SGD([W,b],lr=1) #SGD optimizer를 사용하고 learning rate는 1로 적용하세요.\n","\n","nb_epochs = 1000\n","for epoch in range(nb_epochs + 1):\n","\n","    # Cost 계산\n","    hypothesis = torch.sigmoid(x_train.matmul(W)+b) # or .mm or @ #torch.sigmoid를 사용하고 x_train의 shape이 달라졌기 때문에 행렬곱셉을 합니다!!\n","    cost = F.binary_cross_entropy(hypothesis,y_train) #PyTorch에서 제공하는 binary_cross_entropy 함수를 사용하세요.\n","\n","    # cost로 H(x) 개선\n","    optimizer.zero_grad()  # gradient를 0으로 초기화\n","    cost.backward()  # 비용 함수를 미분하여 gradient 계산\n","    optimizer.step()  # W와 b를 업데이트\n","\n","    # 100번마다 로그 출력\n","    if epoch % 100 == 0:\n","        print('Epoch {:4d}/{} Cost: {:.6f}'.format(\n","            epoch, nb_epochs, cost.item()\n","        ))"]},{"cell_type":"markdown","metadata":{"id":"CGxZzc-id1l_"},"source":["## Loading Real Data"]},{"cell_type":"markdown","metadata":{"id":"qymF9ZiIYhKv"},"source":["data-03-diabetes.csv 파일 사용!"]},{"cell_type":"code","execution_count":21,"metadata":{"id":"RVvbZBiWd1l_","executionInfo":{"status":"ok","timestamp":1648042585083,"user_tz":-540,"elapsed":4,"user":{"displayName":"‍홍소연[학생](전자정보대학 전자공학과)","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"03502202636524032613"}}},"outputs":[],"source":["import numpy as np"]},{"cell_type":"code","execution_count":23,"metadata":{"id":"C1cp2Hy_gT3w","executionInfo":{"status":"ok","timestamp":1648042637208,"user_tz":-540,"elapsed":275,"user":{"displayName":"‍홍소연[학생](전자정보대학 전자공학과)","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"03502202636524032613"}}},"outputs":[],"source":["from google.colab import drive"]},{"cell_type":"code","execution_count":24,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"9kBU6ttYgT5W","outputId":"7786eb90-6892-4068-f782-f7f7f46a1f1d","executionInfo":{"status":"ok","timestamp":1648042654261,"user_tz":-540,"elapsed":16681,"user":{"displayName":"‍홍소연[학생](전자정보대학 전자공학과)","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"03502202636524032613"}}},"outputs":[{"output_type":"stream","name":"stdout","text":["Mounted at /content/drive\n"]}],"source":["drive.mount('/content/drive')"]},{"cell_type":"code","execution_count":35,"metadata":{"id":"kfvVhaDvd1mA","executionInfo":{"status":"ok","timestamp":1648043242581,"user_tz":-540,"elapsed":273,"user":{"displayName":"‍홍소연[학생](전자정보대학 전자공학과)","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"03502202636524032613"}}},"outputs":[],"source":["xy = np.loadtxt('/content/drive/MyDrive/data-03-diabetes.csv', delimiter=',', dtype=np.float32)\n","x_data = xy[:, 0:-1]\n","y_data = xy[:, [-1]]\n","x_train = torch.FloatTensor(x_data)\n","y_train = torch.FloatTensor(y_data)"]},{"cell_type":"code","execution_count":36,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"9MNmGRG4d1mA","outputId":"b47be933-38af-42d5-a33f-cc4bcda8bd94","executionInfo":{"status":"ok","timestamp":1648043243844,"user_tz":-540,"elapsed":2,"user":{"displayName":"‍홍소연[학생](전자정보대학 전자공학과)","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"03502202636524032613"}}},"outputs":[{"output_type":"stream","name":"stdout","text":["tensor([[-0.2941,  0.4874,  0.1803, -0.2929,  0.0000,  0.0015, -0.5312, -0.0333],\n","        [-0.8824, -0.1457,  0.0820, -0.4141,  0.0000, -0.2072, -0.7669, -0.6667],\n","        [-0.0588,  0.8392,  0.0492,  0.0000,  0.0000, -0.3055, -0.4927, -0.6333],\n","        [-0.8824, -0.1055,  0.0820, -0.5354, -0.7778, -0.1624, -0.9240,  0.0000],\n","        [ 0.0000,  0.3769, -0.3443, -0.2929, -0.6028,  0.2846,  0.8873, -0.6000]])\n","tensor([[0.],\n","        [1.],\n","        [0.],\n","        [1.],\n","        [0.]])\n"]}],"source":["print(x_train[0:5])\n","print(y_train[0:5])"]},{"cell_type":"markdown","metadata":{"id":"vX7BmUzLd1mB"},"source":["## Training with Real Data using low-level Binary Cross Entropy Loss"]},{"cell_type":"code","execution_count":37,"metadata":{"id":"mFDd3uB0d1mB","outputId":"4782851a-9fd8-4ec5-c3c8-e9745c94d05d","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1648043253625,"user_tz":-540,"elapsed":3,"user":{"displayName":"‍홍소연[학생](전자정보대학 전자공학과)","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"03502202636524032613"}}},"outputs":[{"output_type":"stream","name":"stdout","text":["Epoch    0/100 Cost: 0.693147\n","Epoch   10/100 Cost: 0.572727\n","Epoch   20/100 Cost: 0.539493\n","Epoch   30/100 Cost: 0.519708\n","Epoch   40/100 Cost: 0.507066\n","Epoch   50/100 Cost: 0.498539\n","Epoch   60/100 Cost: 0.492549\n","Epoch   70/100 Cost: 0.488209\n","Epoch   80/100 Cost: 0.484985\n","Epoch   90/100 Cost: 0.482543\n","Epoch  100/100 Cost: 0.480661\n"]}],"source":["# 모델 초기화\n","W = torch.zeros([8,1],requires_grad=True) #가중치 W를 0으로 초기화하고 학습을 통해 값이 변경될 수 있도록 하세요. x_train의 shape을 잘 생각해보세요!!\n","b = torch.zeros(1,requires_grad=True) #가중치 W를 0으로 초기화하고 학습을 통해 값이 변경될 수 있도록 하세요.\n","# optimizer 설정\n","optimizer = optim.SGD([W,b],lr=1) #SGD optimizer를 사용하고 learning rate는 1로 적용하세요.\n","\n","nb_epochs = 100\n","for epoch in range(nb_epochs + 1):\n","\n","    # Cost 계산\n","    hypothesis = torch.sigmoid(x_train.matmul(W)+b) # or .mm or @ #torch.sigmoid를 사용하고 x_train의 shape이 달라졌기 때문에 행렬곱셉을 합니다!!\n","    cost = -(y_train*torch.log(hypothesis)+(1-y_train)*torch.log(1-hypothesis)).mean() \n","    # cost(W)=−1/m∑ylog(H(x))+(1−y)(log(1−H(x)))을 이용하세요.\n","\n","    # cost로 H(x) 개선\n","    optimizer.zero_grad()  # gradient를 0으로 초기화\n","    cost.backward()  # 비용 함수를 미분하여 gradient 계산\n","    optimizer.step()  # W와 b를 업데이트\n","\n","    # 10번마다 로그 출력\n","    if epoch % 10 == 0:\n","        print('Epoch {:4d}/{} Cost: {:.6f}'.format(\n","            epoch, nb_epochs, cost.item()\n","        ))"]},{"cell_type":"markdown","metadata":{"id":"_IqYdBJKd1mC"},"source":["## Training with Real Data using `F.binary_cross_entropy`"]},{"cell_type":"code","execution_count":38,"metadata":{"id":"kS4yfIf_d1mC","outputId":"ab749deb-bd3f-4664-aa39-b99c1ad09ef6","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1648043318882,"user_tz":-540,"elapsed":290,"user":{"displayName":"‍홍소연[학생](전자정보대학 전자공학과)","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"03502202636524032613"}}},"outputs":[{"output_type":"stream","name":"stdout","text":["Epoch    0/100 Cost: 0.693147\n","Epoch   10/100 Cost: 0.572727\n","Epoch   20/100 Cost: 0.539493\n","Epoch   30/100 Cost: 0.519708\n","Epoch   40/100 Cost: 0.507066\n","Epoch   50/100 Cost: 0.498539\n","Epoch   60/100 Cost: 0.492549\n","Epoch   70/100 Cost: 0.488209\n","Epoch   80/100 Cost: 0.484985\n","Epoch   90/100 Cost: 0.482543\n","Epoch  100/100 Cost: 0.480661\n"]}],"source":["# 모델 초기화\n","W = torch.zeros([8,1],requires_grad=True) #가중치 W를 0으로 초기화하고 학습을 통해 값이 변경될 수 있도록 하세요. x_train의 shape을 잘 생각해보세요!!\n","b = torch.zeros(1,requires_grad=True) #가중치 W를 0으로 초기화하고 학습을 통해 값이 변경될 수 있도록 하세요.\n","# optimizer 설정\n","optimizer = optim.SGD([W,b],lr=1) #SGD optimizer를 사용하고 learning rate는 1로 적용하세요.\n","\n","nb_epochs = 100\n","for epoch in range(nb_epochs + 1):\n","\n","    # Cost 계산\n","    hypothesis = torch.sigmoid(x_train.matmul(W)+b)# or .mm or @ #torch.sigmoid를 사용하고 x_train의 shape이 달라졌기 때문에 행렬곱셉을 합니다!!\n","    cost = F.binary_cross_entropy(hypothesis,y_train) #PyTorch에서 제공하는 binary_cross_entropy 함수를 사용하세요.\n","\n","    # cost로 H(x) 개선\n","    optimizer.zero_grad()  # gradient를 0으로 초기화\n","    cost.backward()  # 비용 함수를 미분하여 gradient 계산\n","    optimizer.step()  # W와 b를 업데이트\n","\n","    # 10번마다 로그 출력\n","    if epoch % 10 == 0:\n","        print('Epoch {:4d}/{} Cost: {:.6f}'.format(\n","            epoch, nb_epochs, cost.item()\n","        ))"]},{"cell_type":"markdown","metadata":{"id":"HuqdMlhjd1mD"},"source":["## Checking the Accuracy our our Model"]},{"cell_type":"markdown","metadata":{"id":"X_YgqECVd1mD"},"source":["After we finish training the model, we want to check how well our model fits the training set."]},{"cell_type":"code","execution_count":39,"metadata":{"id":"ETC0bLT4d1mE","outputId":"94f1e271-5fa1-4c3f-c653-bac1c44f44e7","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1648043323972,"user_tz":-540,"elapsed":261,"user":{"displayName":"‍홍소연[학생](전자정보대학 전자공학과)","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"03502202636524032613"}}},"outputs":[{"output_type":"stream","name":"stdout","text":["tensor([[0.4103],\n","        [0.9242],\n","        [0.2300],\n","        [0.9411],\n","        [0.1772]], grad_fn=<SliceBackward0>)\n"]}],"source":["hypothesis = torch.sigmoid(x_train.matmul(W) + b)\n","print(hypothesis[:5])"]},{"cell_type":"markdown","metadata":{"id":"Oc6V0StDd1mE"},"source":["We can change **hypothesis** (real number from 0 to 1) to **binary predictions** (either 0 or 1) by comparing them to 0.5."]},{"cell_type":"code","execution_count":40,"metadata":{"id":"hg0Ow208d1mE","outputId":"29c9d8fd-1e57-482d-93e0-a23ea52c18f3","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1648043326092,"user_tz":-540,"elapsed":277,"user":{"displayName":"‍홍소연[학생](전자정보대학 전자공학과)","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"03502202636524032613"}}},"outputs":[{"output_type":"stream","name":"stdout","text":["tensor([[False],\n","        [ True],\n","        [False],\n","        [ True],\n","        [False]])\n"]}],"source":["prediction = hypothesis >= torch.FloatTensor([0.5])\n","print(prediction[:5])"]},{"cell_type":"markdown","metadata":{"id":"xFDxcdtUd1mF"},"source":["Then, we compare it with the correct labels `y_train`."]},{"cell_type":"code","execution_count":41,"metadata":{"id":"JdSHsInid1mF","outputId":"630d6f6b-062e-4567-80b9-1c76e170d5f2","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1648043327247,"user_tz":-540,"elapsed":2,"user":{"displayName":"‍홍소연[학생](전자정보대학 전자공학과)","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"03502202636524032613"}}},"outputs":[{"output_type":"stream","name":"stdout","text":["tensor([[False],\n","        [ True],\n","        [False],\n","        [ True],\n","        [False]])\n","tensor([[0.],\n","        [1.],\n","        [0.],\n","        [1.],\n","        [0.]])\n"]}],"source":["print(prediction[:5])\n","print(y_train[:5])"]},{"cell_type":"code","execution_count":42,"metadata":{"id":"hZGytd8Gd1mF","outputId":"525f61fc-032d-418f-d311-4b50d084e54e","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1648043328324,"user_tz":-540,"elapsed":8,"user":{"displayName":"‍홍소연[학생](전자정보대학 전자공학과)","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"03502202636524032613"}}},"outputs":[{"output_type":"stream","name":"stdout","text":["tensor([[True],\n","        [True],\n","        [True],\n","        [True],\n","        [True]])\n"]}],"source":["correct_prediction = prediction.float() == y_train\n","print(correct_prediction[:5])"]},{"cell_type":"markdown","metadata":{"id":"IyybcRe4d1mH"},"source":["Finally, we can calculate the accuracy by counting the number of correct predictions and dividng by total number of predictions."]},{"cell_type":"code","execution_count":43,"metadata":{"id":"IczKlmbZd1mH","outputId":"4abaa3aa-8e6f-4717-9c88-168279efb180","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1648043329983,"user_tz":-540,"elapsed":4,"user":{"displayName":"‍홍소연[학생](전자정보대학 전자공학과)","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"03502202636524032613"}}},"outputs":[{"output_type":"stream","name":"stdout","text":["The model has an accuracy of 76.68% for the training set.\n"]}],"source":["accuracy = correct_prediction.sum().item() / len(correct_prediction)\n","print('The model has an accuracy of {:2.2f}% for the training set.'.format(accuracy * 100))"]},{"cell_type":"markdown","metadata":{"id":"RmnNUiVDd1mI"},"source":["## Optional: High-level Implementation with `nn.Module`"]},{"cell_type":"code","execution_count":44,"metadata":{"id":"Pej07LDBd1mI","executionInfo":{"status":"ok","timestamp":1648043331254,"user_tz":-540,"elapsed":7,"user":{"displayName":"‍홍소연[학생](전자정보대학 전자공학과)","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"03502202636524032613"}}},"outputs":[],"source":["class BinaryClassifier(nn.Module):\n","    def __init__(self):\n","        super().__init__()\n","        self.linear = nn.Linear(8, 1)\n","        self.sigmoid = nn.Sigmoid()\n","\n","    def forward(self, x):\n","        return self.sigmoid(self.linear(x))"]},{"cell_type":"code","execution_count":47,"metadata":{"id":"5WcFoEPFd1mJ","executionInfo":{"status":"ok","timestamp":1648043375614,"user_tz":-540,"elapsed":267,"user":{"displayName":"‍홍소연[학생](전자정보대학 전자공학과)","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"03502202636524032613"}}},"outputs":[],"source":["model =BinaryClassifier() #BinaryClassifier()로 모델을 초기화하세요."]},{"cell_type":"code","execution_count":50,"metadata":{"id":"gyNYPiond1mK","outputId":"94291f9d-3420-4722-9771-6a18b3b3f03e","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1648043484240,"user_tz":-540,"elapsed":269,"user":{"displayName":"‍홍소연[학생](전자정보대학 전자공학과)","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"03502202636524032613"}}},"outputs":[{"output_type":"stream","name":"stdout","text":["Epoch    0/100 Cost: 0.704829 Accuracy 45.72%\n","Epoch   10/100 Cost: 0.572392 Accuracy 67.59%\n","Epoch   20/100 Cost: 0.539563 Accuracy 73.25%\n","Epoch   30/100 Cost: 0.520041 Accuracy 75.89%\n","Epoch   40/100 Cost: 0.507561 Accuracy 76.15%\n","Epoch   50/100 Cost: 0.499125 Accuracy 76.42%\n","Epoch   60/100 Cost: 0.493177 Accuracy 77.21%\n","Epoch   70/100 Cost: 0.488846 Accuracy 76.81%\n","Epoch   80/100 Cost: 0.485612 Accuracy 76.28%\n","Epoch   90/100 Cost: 0.483146 Accuracy 76.55%\n","Epoch  100/100 Cost: 0.481234 Accuracy 76.81%\n"]}],"source":["# optimizer 설정\n","optimizer = optim.SGD(model.parameters(),lr=1)# #SGD optimizer를 사용하고 learning rate는 1로 적용하세요.\n","\n","nb_epochs = 100\n","for epoch in range(nb_epochs + 1):\n","\n","    # H(x) 계산\n","    hypothesis = model(x_train) #모델을 생성해서 예측값을 구해보세요.\n","\n","    # cost 계산\n","    cost = F.binary_cross_entropy(hypothesis,y_train) #PyTorch에서 제공하는 binary_cross_entropy 함수를 사용하세요.\n","\n","    # cost로 H(x) 개선\n","    optimizer.zero_grad()  # gradient를 0으로 초기화\n","    cost.backward()  # 비용 함수를 미분하여 gradient 계산\n","    optimizer.step()  # W와 b를 업데이트\n","    \n","    # 20번마다 로그 출력\n","    if epoch % 10 == 0:\n","        prediction = hypothesis >= torch.FloatTensor([0.5])\n","        correct_prediction = prediction.float() == y_train\n","        accuracy = correct_prediction.sum().item() / len(correct_prediction)\n","        print('Epoch {:4d}/{} Cost: {:.6f} Accuracy {:2.2f}%'.format(\n","            epoch, nb_epochs, cost.item(), accuracy * 100,\n","        ))\n"]},{"cell_type":"code","source":[""],"metadata":{"id":"Bn8u_QGRCZHg"},"execution_count":null,"outputs":[]}],"metadata":{"colab":{"collapsed_sections":[],"name":"Lab05_final.ipynb","provenance":[]},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.6"}},"nbformat":4,"nbformat_minor":0}