09 - Lesson - PyTorch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "91da3a70",
   "metadata": {},
   "source": [
    "# PyTorch\n",
    "\n",
    "### Goal of lesson\n",
    "- What is PyTorch\n",
    "- PyTorch vs Tensorflow\n",
    "- Get started with PyTorch\n",
    "- Work with image classification"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97a9bdde",
   "metadata": {},
   "source": [
    "### Resources to PyTorch\n",
    "- Free book [Deep Learning with PyTorch](https://pytorch.org/assets/deep-learning/Deep-Learning-with-PyTorch.pdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4023a000",
   "metadata": {},
   "source": [
    "### PyTorch and Tensorflow\n",
    "- PyTorch and Tensorflow are the most popular Deep Learning frameworks\n",
    "- PyTorch was released in 2016 by Facebook’s AI Research lab\n",
    "- Tensorflow was released in 2015 by Google Brain team\n",
    "- They are both sound choices for Deep Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "303dc52f",
   "metadata": {},
   "source": [
    "### Installing PyToch\n",
    "- PyTorch is not by default installed\n",
    "\n",
    "### Look at the datasets\n",
    "- PyTorch comes with a long list of prepared datasets ([see all here](https://pytorch.org/vision/0.8/datasets.html)).\n",
    "- We will cover\n",
    "    - [MNIST](https://en.wikipedia.org/wiki/MNIST_database) A datasets for handwritten digit-recognition.\n",
    "    - [CIFAR10](https://en.wikipedia.org/wiki/CIFAR-10) A dataset of 32x32 images of 10 classes\n",
    "- Datasets can be downloaded with **torchvision**\n",
    "\n",
    "``` Python\n",
    "from torchvision import datasets\n",
    "\n",
    "data_path = 'downloads/'\n",
    "mnist = datasets.MNIST(data_path, train=True, download=True)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c3cede0",
   "metadata": {},
   "source": [
    "### Transforming and Normalizing\n",
    "- Images are PIL\n",
    "- Need to be transformed to tensor\n",
    "    - **torchvision** has transformations **transform.ToTensor()**, which turns NumPy arrays and PIL to Tensor\n",
    "- Then we need to normalize images\n",
    "    - Need to determine the **mean** value and the **standard deviation**\n",
    "- Then we can apply nomalization\n",
    "    - **torchvision** has **transform.Normalize**, which takes **mean** and **standard deviation**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c4a0d9d",
   "metadata": {},
   "source": [
    "### Training the model\n",
    "<img src='img/mnist_nn.png' width=600 align='left'>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e964378d",
   "metadata": {},
   "source": [
    "### Optimize the model\n",
    "- If we have defined the model as above - we will use the [SGD](https://pytorch.org/docs/stable/optim.html) optimizer.\n",
    "\n",
    "```Python\n",
    "train_loader = torch.utils.data.DataLoader(mnist, batch_size=64,\n",
    "                                           shuffle=True)\n",
    "\n",
    "optimizer = optim.SGD(model.parameters(), lr=0.01)\n",
    "loss_fn = nn.NLLLoss()\n",
    "\n",
    "n_epochs = 10\n",
    "for epoch in range(n_epochs):\n",
    "    for imgs, labels in train_loader:\n",
    "        optimizer.zero_grad()\n",
    "        \n",
    "        batch_size = imgs.shape[0]\n",
    "        output = model(imgs.view(batch_size, -1))\n",
    "        \n",
    "        loss = loss_fn(output, labels)\n",
    "        \n",
    "        loss.backward()\n",
    "        \n",
    "        optimizer.step()\n",
    "    print(\"Epoch: %d, Loss: %f\" % (epoch, float(loss)))\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98919a5a",
   "metadata": {},
   "source": [
    "### Test the model\n",
    "- Finally we need to test the model\n",
    "\n",
    "```Python\n",
    "correct = 0\n",
    "total = 0\n",
    "with torch.no_grad():\n",
    "    for imgs, labels in val_loader:\n",
    "        batch_size = imgs.shape[0]\n",
    "        outputs = model(imgs.view(batch_size, -1))\n",
    "        _, predicted = torch.max(outputs, dim=1)\n",
    "        total += labels.shape[0]\n",
    "        correct += int((predicted == labels).sum())\n",
    "print(\"Accuracy: %f\", correct / total)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f26265f0",
   "metadata": {},
   "source": [
    "> #### Programming Notes:\n",
    "> - Libraries used\n",
    ">     - [**torch**](https://pytorch.org) open source machine learning framework\n",
    ">     - [**torchvision**](https://pypi.org/project/torchvision/) image and video datasets and models for torch deep learning\n",
    ">     - [**matplotlib**](http://matplotlib.org) - visualization with Python ([Lecture on **visualization**](https://youtu.be/htIh8YHh4xs))\n",
    "> - Functionality and concepts used\n",
    ">     - [**datasets**](https://pytorch.org/vision/stable/datasets.html) torchvision datasets.\n",
    ">     - [**ToTensor**](https://pytorch.org/vision/stable/transforms.html#torchvision.transforms.ToTensor) turns NumPy arrays and PIL to Tensor\n",
    ">     - [**Normalize**](https://pytorch.org/vision/stable/transforms.html#torchvision.transforms.Normalize) Normalize a tensor image with mean and standard deviation\n",
    ">     - [**Compose**](https://pytorch.org/vision/stable/transforms.html#torchvision.transforms.Compose) Composes several transforms together.\n",
    ">     - [**DataLoader**](https://pytorch.org/docs/stable/data.html) data loading utiltiy\n",
    ">     - [**Sequential**](https://www.tensorflow.org/api_docs/python/tf/keras/Sequential) provides training and inference features for a model\n",
    ">     - [**Linear**](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html) Applies a linear transformation to the incoming data\n",
    ">     - [**ReLU**](https://pytorch.org/docs/stable/generated/torch.nn.ReLU.html) Applies the rectified linear unit function element-wise (a simple function which allows positive values to pass through, whereas negative values are modified to zero)\n",
    ">     - [**LogSoftmax**](https://pytorch.org/docs/stable/generated/torch.nn.LogSoftmax.html) Applies the **log(Softmax(x))** function to an n-dimensional input Tensor\n",
    ">     - [**SGD**](https://pytorch.org/docs/stable/generated/torch.optim.SGD.html) Implements stochastic gradient descent\n",
    ">     - [**NLLLoss**](https://pytorch.org/docs/stable/generated/torch.nn.NLLLoss.html) The negative log likelihood loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aed835c4",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!pip install torchvision"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "087eadea",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchvision import datasets\n",
    "from torchvision import transforms\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch import optim\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ada05ad3",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz\n",
      "Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to downloads/MNIST/raw/train-images-idx3-ubyte.gz\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1f0ea608a2b344aa803851e92ad35b77",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/9912422 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting downloads/MNIST/raw/train-images-idx3-ubyte.gz to downloads/MNIST/raw\n",
      "\n",
      "Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz\n",
      "Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz to downloads/MNIST/raw/train-labels-idx1-ubyte.gz\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7be8a4db6bc04304bd4c133b6a8ee920",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/28881 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting downloads/MNIST/raw/train-labels-idx1-ubyte.gz to downloads/MNIST/raw\n",
      "\n",
      "Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz\n",
      "Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz to downloads/MNIST/raw/t10k-images-idx3-ubyte.gz\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f2862f77775043f4b863072ca387e775",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1648877 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting downloads/MNIST/raw/t10k-images-idx3-ubyte.gz to downloads/MNIST/raw\n",
      "\n",
      "Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz\n",
      "Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz to downloads/MNIST/raw/t10k-labels-idx1-ubyte.gz\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "21d7066b95a04e9383b2b66cc5368a26",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/4542 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting downloads/MNIST/raw/t10k-labels-idx1-ubyte.gz to downloads/MNIST/raw\n",
      "\n"
     ]
    }
   ],
   "source": [
    "data_path = 'downloads/'\n",
    "mnist = datasets.MNIST(data_path, train=True, download=True)\n",
    "mnist_val = datasets.MNIST(data_path, train=False, download=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "6869574d",
   "metadata": {},
   "outputs": [],
   "source": [
    "img, label = mnist[1000]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "dd6ef27b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0 - zero'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mnist.classes[label]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "5dda9f4d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7f82c6a13340>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAANz0lEQVR4nO3dYaxU9ZnH8d/Pu9QXwAuQoEjJ0iIaNxvXroRsAtm4Nm1YY4JN6AovKo24ty9KbMOqq66mJptG2GwrvjCNt1ELmy5qIlXSNGkNwbVrIvFCEFC2FQnbUq7cBUywRoPgsy/uobnCzJnrnJk5w32+n+RmZs4z55wnJ/w4Z+Y/M39HhABMfpfU3QCA3iDsQBKEHUiCsANJEHYgiT/r5c5s89Y/0GUR4UbLK53ZbS+z/RvbB23fV2VbALrL7Y6z2x6Q9FtJX5F0RNLrklZFxFsl63BmB7qsG2f2xZIORsShiDgt6RlJyytsD0AXVQn7XEm/H/f4SLHsU2wP2h62PVxhXwAqqvIGXaNLhQsu0yNiSNKQxGU8UKcqZ/YjkuaNe/x5SUertQOgW6qE/XVJC21/wfbnJK2UtK0zbQHotLYv4yPijO21kn4paUDSUxHxZsc6A9BRbQ+9tbUzXrMDXdeVD9UAuHgQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kETbUzYjh6uuuqq0ftddd5XW165d27RmN5xs9E/OnDlTWr/zzjtL61u2bGlaO336dOm6k1GlsNs+LOl9SWclnYmIRZ1oCkDndeLM/ncRcbwD2wHQRbxmB5KoGvaQ9Cvbu2wPNnqC7UHbw7aHK+4LQAVVL+OXRMRR27MlvWT7fyLilfFPiIghSUOSZDsq7g9Amyqd2SPiaHE7KulnkhZ3oikAndd22G1PtT393H1JX5W0v1ONAegsR7R3ZW37ixo7m0tjLwf+MyK+32IdLuN7bGBgoLR+++23l9Y3bNhQWp81a9Zn7umc0dHR0vrs2bPb3rYkLVy4sGntnXfeqbTtfhYRDT/A0PZr9og4JOmv2u4IQE8x9AYkQdiBJAg7kARhB5Ig7EASbQ+9tbUzht66YtWqVU1rN9xwQ+m669atq7TvF154obT++OOPN621Gv565plnSuuLF5d/huvll19uWrvppptK172YNRt648wOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kwzn4RKPs5Zkl67LHHmtZa/VzziRMnSuvLli0rre/evbu0XuXf17Rp00rrp06danvfS5YsKV33tddeK633M8bZgeQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJpmzuA63Gk1uNs5eNpX/wwQel695yyy2l9V27dpXWu6nVtMoHDhworV977bWdbOeix5kdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5JgnL0PTJ8+vbR+9dVXt73tjRs3ltZ37tzZ9ra7rdU4+759+0rrjLN/Wsszu+2nbI/a3j9u2UzbL9l+u7id0d02AVQ1kcv4n0g6/+dK7pO0PSIWStpePAbQx1qGPSJekXTyvMXLJW0q7m+SdGtn2wLQae2+Zr88IkYkKSJGbM9u9kTbg5IG29wPgA7p+ht0ETEkaUjiByeBOrU79HbM9hxJKm5HO9cSgG5oN+zbJK0u7q+W9GJn2gHQLS0v421vkXSjpFm2j0j6nqT1kp6zvUbS7yR9vZtNTnaXXXZZpfXLvrP+9NNPV9o2Jo+WYY+IVU1KX+5wLwC6iI/LAkkQdiAJwg4kQdiBJAg7kARfce0DK1asqLT+c88917R26NChStvG5MGZHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSYJy9B1p9hXXNmjWVtj88PFxp/X516aWXltaXLFnSo04mB87sQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AE4+w9cM0115TW586dW2n7J0+ePxXf5DAwMFBab3XcPvroo6a1Dz/8sK2eLmac2YEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcbZJ4Ft27bV3UJfOnjwYNPaG2+80cNO+kPLM7vtp2yP2t4/btnDtv9ge0/xd3N32wRQ1UQu438iaVmD5Y9GxPXF3y862xaATmsZ9oh4RdLk/DwmkEiVN+jW2t5bXObPaPYk24O2h21Pzh9KAy4S7Yb9R5IWSLpe0oikHzR7YkQMRcSiiFjU5r4AdEBbYY+IYxFxNiI+kfRjSYs72xaATmsr7LbnjHv4NUn7mz0XQH9oOc5ue4ukGyXNsn1E0vck3Wj7ekkh6bCkb3WvRWS1evXqSutv2LChQ51MDi3DHhGrGix+sgu9AOgiPi4LJEHYgSQIO5AEYQeSIOxAEo6I3u3M7t3O+siUKVNK62+99VZpfcGCBaX1qVOnNq31808mX3HFFaX13bt3V1r/yiuvbFp79913S9e9mEWEGy3nzA4kQdiBJAg7kARhB5Ig7EAShB1IgrADSfBT0j3w8ccfl9bPnj3bo076y9KlS0vrrcbRWx23Xn6G5GLAmR1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkmCcfRKYO3du01rZtMW9MHv27Ka1Bx98sHTdVuPoa9asKa0fO3astJ4NZ3YgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIJx9j7w7LPPltYfeuih0vqKFSua1tavX99WTxM1MDBQWr/33nub1q677rrSdUdGRkrrmzdvLq3j01qe2W3Ps73D9gHbb9r+TrF8pu2XbL9d3M7ofrsA2jWRy/gzkv4pIq6V9DeSvm37LyTdJ2l7RCyUtL14DKBPtQx7RIxExO7i/vuSDkiaK2m5pE3F0zZJurVLPQLogM/0mt32fElfkrRT0uURMSKN/Ydgu+GHoG0PShqs2CeAiiYcdtvTJD0v6bsRccpuOHfcBSJiSNJQsQ1+ARCoyYSG3mxP0VjQfxoRW4vFx2zPKepzJI12p0UAndDyzO6xU/iTkg5ExA/HlbZJWi1pfXH7Ylc6TGDv3r2V1h8cbP4q6Yknnihd97333qu075UrV5bW161b17R28uTJ0nWXL1/eVk9obCKX8UskfUPSPtt7imUPaCzkz9leI+l3kr7elQ4BdETLsEfEf0tq9gL9y51tB0C38HFZIAnCDiRB2IEkCDuQBGEHkuArrn1gx44dpfUTJ06U1ufPn9+0ds8995Su++ijj5bW77jjjtJ62VdYW9m4cWNpfXh4uO1t40Kc2YEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCUf07sdj+KWa9ixatKi0/uqrrzatTZkypXTd48ePl9ZnzpxZWr/kkvLzxdatW5vWbrvtttJ1W03ZjMYiouG3VDmzA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EASjLNPAnfffXfT2v3331+67owZ1SbffeSRR0rrZd+XbzXGj/Ywzg4kR9iBJAg7kARhB5Ig7EAShB1IgrADSbQcZ7c9T9JmSVdI+kTSUEQ8ZvthSf8o6f+Kpz4QEb9osS3G2YEuazbOPpGwz5E0JyJ2254uaZekWyX9g6Q/RsS/T7QJwg50X7OwT2R+9hFJI8X9920fkDS3s+0B6LbP9Jrd9nxJX5K0s1i01vZe20/Zbvi5S9uDtodtM5cPUKMJfzbe9jRJ/yXp+xGx1fblko5LCkn/qrFL/dKJwbiMB7qv7dfskmR7iqSfS/plRPywQX2+pJ9HxF+22A5hB7qs7S/C2LakJyUdGB/04o27c74maX/VJgF0z0TejV8q6deS9mls6E2SHpC0StL1GruMPyzpW8WbeWXb4swOdFmly/hOIexA9/F9diA5wg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBItf3Cyw45L+t9xj2cVy/pRv/bWr31J9NauTvb2580KPf0++wU7t4cjYlFtDZTo1976tS+J3trVq964jAeSIOxAEnWHfajm/Zfp1976tS+J3trVk95qfc0OoHfqPrMD6BHCDiRRS9htL7P9G9sHbd9XRw/N2D5se5/tPXXPT1fMoTdqe/+4ZTNtv2T77eK24Rx7NfX2sO0/FMduj+2ba+ptnu0dtg/YftP2d4rltR67kr56ctx6/prd9oCk30r6iqQjkl6XtCoi3uppI03YPixpUUTU/gEM238r6Y+SNp+bWsv2v0k6GRHri/8oZ0TEP/dJbw/rM07j3aXemk0z/k3VeOw6Of15O+o4sy+WdDAiDkXEaUnPSFpeQx99LyJekXTyvMXLJW0q7m/S2D+WnmvSW1+IiJGI2F3cf1/SuWnGaz12JX31RB1hnyvp9+MeH1F/zfcekn5le5ftwbqbaeDyc9NsFbeza+7nfC2n8e6l86YZ75tj187051XVEfZGU9P00/jfkoj4a0l/L+nbxeUqJuZHkhZobA7AEUk/qLOZYprx5yV9NyJO1dnLeA366slxqyPsRyTNG/f485KO1tBHQxFxtLgdlfQzjb3s6CfHzs2gW9yO1tzPn0TEsYg4GxGfSPqxajx2xTTjz0v6aURsLRbXfuwa9dWr41ZH2F+XtND2F2x/TtJKSdtq6OMCtqcWb5zI9lRJX1X/TUW9TdLq4v5qSS/W2Mun9Ms03s2mGVfNx6726c8joud/km7W2Dvy70j6lzp6aNLXFyW9Ufy9WXdvkrZo7LLuY41dEa2RdJmk7ZLeLm5n9lFv/6Gxqb33aixYc2rqbanGXhrulbSn+Lu57mNX0ldPjhsflwWS4BN0QBKEHUiCsANJEHYgCcIOJEHYgSQIO5DE/wO8iEGaqC78NAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.imshow(img, cmap='gray')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "55a35037",
   "metadata": {},
   "outputs": [],
   "source": [
    "mnist = datasets.MNIST(data_path, train=True, download=False, transform=transforms.ToTensor())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "9839aa01",
   "metadata": {},
   "outputs": [],
   "source": [
    "imgs = torch.stack([img_t for img_t, _ in mnist], dim=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "59aa2ee7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0.1307])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "imgs.view(1, -1).mean(dim=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "469847d8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0.3081])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "imgs.view(1, -1).std(dim=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "d0028073",
   "metadata": {},
   "outputs": [],
   "source": [
    "mnist = datasets.MNIST(data_path, train=True, download=False, \n",
    "                       transform=transforms.Compose([\n",
    "                           transforms.ToTensor(),\n",
    "                           transforms.Normalize((0.1307),\n",
    "                                               (0.3081))]))\n",
    "\n",
    "mnist_val = datasets.MNIST(data_path, train=False, download=False, \n",
    "                       transform=transforms.Compose([\n",
    "                           transforms.ToTensor(),\n",
    "                           transforms.Normalize((0.1307),\n",
    "                                               (0.3081))]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "ed0709a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "input_size = 784 # ?? 28*28\n",
    "hidden_sizes = [128, 64]\n",
    "output_size = 10\n",
    "\n",
    "model = nn.Sequential(nn.Linear(input_size, hidden_sizes[0]),\n",
    "                     nn.ReLU(),\n",
    "                     nn.Linear(hidden_sizes[0], hidden_sizes[1]),\n",
    "                     nn.ReLU(),\n",
    "                     nn.Linear(hidden_sizes[1], output_size),\n",
    "                     nn.LogSoftmax(dim=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "a43b093c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0, Loss: 0.398855\n",
      "Epoch: 1, Loss: 0.338213\n",
      "Epoch: 2, Loss: 0.269782\n",
      "Epoch: 3, Loss: 0.042949\n",
      "Epoch: 4, Loss: 0.189661\n",
      "Epoch: 5, Loss: 0.259607\n",
      "Epoch: 6, Loss: 0.086317\n",
      "Epoch: 7, Loss: 0.046707\n",
      "Epoch: 8, Loss: 0.037421\n",
      "Epoch: 9, Loss: 0.120779\n"
     ]
    }
   ],
   "source": [
    "train_loader = torch.utils.data.DataLoader(mnist, batch_size=64,\n",
    "                                           shuffle=True)\n",
    "\n",
    "optimizer = optim.SGD(model.parameters(), lr=0.01)\n",
    "loss_fn = nn.NLLLoss()\n",
    "\n",
    "n_epochs = 10\n",
    "for epoch in range(n_epochs):\n",
    "    for imgs, labels in train_loader:\n",
    "        optimizer.zero_grad()\n",
    "\n",
    "        batch_size = imgs.shape[0]\n",
    "        output = model(imgs.view(batch_size, -1))\n",
    "\n",
    "        loss = loss_fn(output, labels)\n",
    "\n",
    "        loss.backward()\n",
    "\n",
    "        optimizer.step()\n",
    "    print(\"Epoch: %d, Loss: %f\" % (epoch, float(loss)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "e05112ce",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: %f 0.9644\n"
     ]
    }
   ],
   "source": [
    "val_loader = torch.utils.data.DataLoader(mnist_val, batch_size=64,\n",
    "                                           shuffle=True)\n",
    "\n",
    "\n",
    "correct = 0\n",
    "total = 0\n",
    "with torch.no_grad():\n",
    "    for imgs, labels in val_loader:\n",
    "        batch_size = imgs.shape[0]\n",
    "        outputs = model(imgs.view(batch_size, -1))\n",
    "        _, predicted = torch.max(outputs, dim=1)\n",
    "        total += labels.shape[0]\n",
    "        correct += int((predicted == labels).sum())\n",
    "print(\"Accuracy: %f\", correct / total)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "786c58a1",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}