Video 3 - Autograd.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Autograd\n",
    "\n",
    "PyTorch's *Autograd* feature is part of what make PyTorch flexible and fast for building machine learning projects. It allows for the rapid and easy computation of multiple partial derivatives (also referred to as *gradients)* over a complex computation. This operation is central to backpropagation-based neural network learning.\n",
    "\n",
    "The power of autograd comes from the fact that it traces your computation dynamically *at runtime,* meaning that if your model has decision branches, or loops whose lengths are not known until runtime, the computation will still be traced correctly, and you'll get correct gradients to drive learning. This, combined with the fact that your models are built in Python, offers far more flexibility than frameworks that rely on static analysis of a more rigidly-structured model for computing gradients.\n",
    "\n",
    "## What Do We Need Autograd For?\n",
    "\n",
    "A machine learning model is a *function*, with inputs and outputs. For this discussion, we'll treat the inputs a as an *i*-dimensional vector $\\vec x$, with elements $x_{i}$. We can then express the model, *M*, as a vector-valued function of the input: *$\\vec y$ = $\\vec M$($\\vec x$)*. (We treat the value of M's output as a vector because in general, a model may have any number of outputs.)\n",
    "\n",
    "Since we'll mostly be discussing autograd in the context of training, our output of interest will be the model's loss. The *loss function* *L($\\vec y$) = L($\\vec M$($\\vec x$))* is a single-valued scalar function of the model's output. This function expresses how far off our model's prediction was from a particular input's *ideal* output. *Note: After this point, we will often omit the vector sign where it should be contextually clear - e.g., $y$ instead of $\\vec y$.*\n",
    "\n",
    "In training a model, we want to minimize the loss. In the idealized case of a perfect model, that means adjusting its learning weights - that is, the adjustable parameters of the function - such that loss is zero for all inputs. In the real world, it means an iterative process of nudging the learning weights until we see that we get a tolerable loss for a wide variety of inputs.\n",
    "\n",
    "How do we decide how far and in which direction to nudge the weights? We want to *minimize* the loss, which means making its first derivative with respect to the input equal to 0: $\\frac{\\partial L}{\\partial x} = 0$.\n",
    "\n",
    "Recall, though, that the loss is not *directly* derived from the input, but a function of the model's output (which is a function of the input directly), $\\frac{\\partial L}{\\partial x}$ = $\\frac{\\partial {L({\\vec y})}}{\\partial x}$. By the chain rule of differential calculus, we have $\\frac{\\partial {L({\\vec y})}}{\\partial x}$ = $\\frac{\\partial L}{\\partial y}\\frac{\\partial y}{\\partial x}$ = $\\frac{\\partial L}{\\partial y}\\frac{\\partial M(x)}{\\partial x}$.\n",
    "\n",
    "$\\frac{\\partial M(x)}{\\partial x}$ is where things get complex. The partial derivatives of the model's outputs with respect to its inputs, if we were to expand the expression using the chain rule again, would involve many local partial derivatives over every multiplied learning weight, every activation function, and every other mathematical transformation in the model. The full expression for each such partial derivative is the sum of the products of the local gradient of *every possible path* through the computation graph that ends with the variable whose gradient we are trying to measure.\n",
    "\n",
    "In particular, the gradients over the learning weights are of interest to us - they tell us *what direction to change each weight* to get the loss function closer to zero.\n",
    "\n",
    "Since the number of such local derivatives (each corresponding to a separate path through the model's computation graph) will tend to go up exponentially with the depth of a neural network, so does the complexity in computing them. This is where autograd comes in: It tracks the history of every computation. Every computed tensor in your PyTorch model carries a history of its input tensors and the function used to create it. Combined with the fact that PyTorch functions meant to act on tensors each have a built-in implementation for computing their own derivatives, this greatly speeds the computation of the local derivatives needed for learning.\n",
    "\n",
    "## A Simple Example\n",
    "\n",
    "That was a lot of theory - but what does it look like to use autograd in practice?\n",
    "\n",
    "Let's start with a straightforward example. First, we'll do some imports to let us graph our results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.ticker as ticker\n",
    "import math"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll create an input tensor full of evenly spaced values on the interval $[0, 2{\\pi}]$, and specify `requires_grad=True`. (Like most functions that create tensors, `torch.linspace()` accepts an optional `requires_grad` option.) Setting this flag means that in every computation that follows, autograd will be accumulating the history of the computation in the output tensors of that computation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([0.0000, 0.2618, 0.5236, 0.7854, 1.0472, 1.3090, 1.5708, 1.8326, 2.0944,\n",
      "        2.3562, 2.6180, 2.8798, 3.1416, 3.4034, 3.6652, 3.9270, 4.1888, 4.4506,\n",
      "        4.7124, 4.9742, 5.2360, 5.4978, 5.7596, 6.0214, 6.2832],\n",
      "       requires_grad=True)\n"
     ]
    }
   ],
   "source": [
    "a = torch.linspace(0., 2. * math.pi, steps=25, requires_grad=True)\n",
    "print(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll perform a computation, and plot its output in terms of its inputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7fa0004671c0>]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAD4CAYAAADhNOGaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3dd3RUZf7H8fc3PYQSUoCQQhISSkDqSBFERQRUpLiuva6KDXR119XVn7pr2SOrawFFRV0V1wUUC+gqSFcEgdBDSSGUhJJCIARCCEme3x8ZPDEGSJhJ7pTv65w5mblzb+YTjs5nnnvv3EeMMSillPJePlYHUEopZS0tAqWU8nJaBEop5eW0CJRSystpESillJfzszrAuYiIiDDx8fFWx1BKKbeydu3aQmNMZO3lblkE8fHxpKamWh1DKaXciojsrmu57hpSSikvp0WglFJeTotAKaW8nBaBUkp5OS0CpZTyck4pAhH5t4jki0jaaZ4XEZksIlkisklE+tR47jYRybTfbnNGHqWUUvXnrBHBh8DIMzx/OZBsv40H3gIQkTDgGaA/0A94RkRaOymTUkqpenDK9wiMMT+ISPwZVhkDTDfV17z+WURCRSQKuBhYYIwpAhCRBVQXygxn5FLn7kBxGT9lFXL0RAUDEsPp1LY5ImJ1LKVUI2iqL5RFAzk1Hufal51u+W+IyHiqRxPExcU1TkovdqTsJKuyi/gpq5DlWYVk5R/91fORLQIZ1DGcQUkRDE6OIKpVsEVJlVLO1lRFUNdHSXOG5b9daMw0YBqAzWbT2XQcVF5Rxfo9h35549+YW0xllSHI34f+CeFcZ4vlgqRwWgX7s2LHwV/W+2rDPgASI0MYnBTBoKQIBiRWr6eUck9NVQS5QGyNxzHAPvvyi2stX9pEmbxO+oESfswsYHlWIauyizh+shIfgZ6xodx/cUcGJUXQOy6UQD/fX213ra0Z19piMcaQnlfC8sxCfsoqZPbaXKav3I2PQI+Y0F+K4fz41vj56glpSrkLcdZUlfZjBN8YY7rX8dyVwATgCqoPDE82xvSzHyxeC5w6i2gd0PfUMYPTsdlsRq81VH9lJyv5+9dbmLG6ei9cxxqf5vs78Gm+vKKKDTmHWZ5VXQwbcg5TWWXoGRvKWzf1oX2o7j5SypWIyFpjjO03y51RBCIyg+pP9hFAHtVnAvkDGGPeluqjjG9QfSC4FLjDGJNq3/YPwBP2X/WCMeaDs72eFkH95R4q5b7/rGPz3mLuuSiR2y+Ib7T9+yVlJ5mXdoC/f72VAD8fptzQm0FJEY3yWkqphmvUImhqWgT1syyjgIdmrqey0vCva3syvFu7JnndHQVHuffjtewoOMqfhnfmvos64uOjZxwpZbXTFYHuyPVAVVWGyYsyuf2D1bRrGcTciYObrAQAOkY256sHBnFlj/a8ND+d8R+vpfj4ySZ7faVUw2gReJji0pPcNT2VVxZkMLZXNF/cfwEJESFNniMk0I/J1/fimatSWJqez5g3lrNt/5Emz6GUOjstAg+yZV8xV72xnB8zC3h2TDdeubYnzQKsm3tIRLhjUAIzxw+gtLyScVN/4sv1uZblUUrVTYvAQ3yWmsPVU1dQXlHFrHsGcuvAeJf5JrAtPoxvHhxMz5hQHp61kafnpFFeUWV1LKWUnRaBmztRUclfv9jMo7M30SeuNd88OJg+ca53uaY2LYL45K7+jB+SyPSVu7lu2kr2Fx+3OpZSCi0Ct7b38HGufXslM1bv4d6LOvLxnf2IaB5odazT8vP14YkrujL1pj5kHChh1OTlrNhRaHUspbyeFoGbWp5ZyKjJP5JdcIx3bunL45d3cZtv815xXhRzJgymdUgAN7+3ireW7sAdT2NWylO4xzuH+pXUXUXc8eFq2rQIYs6EQYxowlNDnSWpTXPmPDCIy8+LYtK87UxdusPqSEp5LetOKVHnZH/xce79zzqiQ4P59J6BtGrmvhd7Cwn0440beuMrwsvfp9M1qgVDu7S1OpZSXkdHBG6k7GQl93y8luPlFbx7q82tS+AUEWHS73qQEtWSh2ZsYEfB0bNvpJRyKi0CN2GM4YkvN7Mpt5hXr+tFctsWVkdymuAAX965pS/+fj6Mn55KSZl+C1mppqRF4Cb+/dMuvli3l4eHdWrSy0U0lZjWzZh6Ux92HSzl4VkbqKrSg8dKNRUtAjfwU1Yh//h2G8NT2jJxaJLVcRrNgMRwnh6VwsJt+by2MMPqOEp5DS0CF5dTVMoD/11Hx8gQXrmul8dfxfPWgR241hbD5MVZzEvbb3UcpbyCFoELKy2v4O7pqVRVGabdYqN5oOef5CUiPDe2O73jQnnk041sP6AXqlOqsWkRuChjDI9+tomMvBKm3NiHeAuuIGqVQD9f3r65L80D/Rg/fS2HS8utjqSUR3NKEYjISBFJF5EsEXm8judfFZEN9luGiByu8VxljefmOiOPJ5i6dAf/27yfx0Z24aJOkVbHaXJtWwbx9i19OVBcxsQZ66mo1IvUKdVYHC4CEfEF3gQuB1KAG0QkpeY6xpiHjTG9jDG9gCnAFzWePn7qOWPMaEfzeILF2/N4+ft0Rvdsz/ghiVbHsUyfuNY8P7Y7P2YWMmnedqvjKOWxnDEi6AdkGWOyjTHlwExgzBnWvwGY4YTX9Ug7Co7y0IwNpES1ZNLverjMpaStcu35sdw2sAPv/riTr9bvtTqOUh7JGUUQDeTUeJxrX/YbItIBSAAW11gcJCKpIvKziIw93YuIyHj7eqkFBQVOiO16jpSd5O7pqfj7+fDOLX0JDvC1OpJL+L9RKfRPCOOxzzexObfY6jhKeRxnFEFdH1lP922g64HZxpjKGsvi7JMp3wi8JiId69rQGDPNGGMzxtgiIz1vn3lVleHhmRvYc7CUqTf1IaZ1M6sjuQx/Xx+m3tSHiOaB3PNxKoVHT1gdSSmP4owiyAViazyOAfadZt3rqbVbyBizz/4zG1gK9HZCJrfz6sIMFm3P5+mrUhiQGG51HJcT3jyQd27pS1FpOfd/so6TevBYKadxRhGsAZJFJEFEAqh+s//N2T8i0hloDayssay1iATa70cAg4CtTsjkVual7WfK4iyus8Vyy4AOVsdxWd2jWzHpdz1YvbOI577xuv9MlGo0DheBMaYCmADMB7YBnxpjtojIsyJS8yygG4CZ5tczkHQFUkVkI7AEeNEY41X/hx86Vs4TX6bRI6YVz47t5vUHh89mTK9o7hqcwPSVu3V2M6WcRNxxZiibzWZSU1OtjuEUj83exOx1uXwzcTBdo1paHcctlJ2sZMRrP+Arwnd/vJBAPz2orlR9iMha+zHZX9FvFltoza4iZqXmcNfgBC2BBgjy9+XZMd3JLjzG20uzrY6jlNvTIrBIeUUVT365mejQYB4almx1HLdzUadIRvWI4s2lWewsPGZ1HKXcmhaBRd5fvpOMvKP8fXQ3mgV4/sXkGsPTo1II9PXhqa/ScMddnEq5Ci0CC+QUlfL6ogyGp7RlWIrO0Xuu2rQM4tGRnVmeVcjcjac7Y1kpdTZaBE3MGMPTc9LwFeFvo7tZHcft3dS/Az1jWvHcN9soPq5TXCp1LrQImti8tAMsSS/g4cs60T402Oo4bs/XR3hh3HkUHTvBS/P1wnRKnQstgiZUUnaSv329hZSoltx+QbzVcTxG9+hW3H5BAp+s2sP6PYesjqOU29EiaEKvLMggv+QEL4zrjp+v/tM70yPDO9G2RRBPfJmmcxco1UD6btRE0vYW89GKXdzUP47eca2tjuNxmgf68bfRKWzbf4QPV+yyOo5SbkWLoAlUVhme+HIz4c0DeXREF6vjeKwR3dpxaZc2vLIgg72Hj1sdRym3oUXQBP7z82425Rbz1KgUWgX7Wx3HY4kIfx/TDWPg73O3WB1HKbehRdDI8o6U8dL8dC5MjuCqHlFWx/F4Ma2b8dCwZL7fmseCrXlWx1HKLWgRNLJnv9lKeWUVz43prlcWbSJ3Dk6gc9sWPDMnjWMnKqyOo5TL0yJoREvT8/nfpv1MuCSJ+IgQq+N4DX9fH/5xdXf2FZfx+qJMq+Mo5fK0CBpJ2clKnp6zhcTIEO65KNHqOF6nb4cwbugXy/vLd7J13xGr4yjl0rQIGsmUxZnsKSrlhbHn6fXyLfLYyC6EBvvz5FebqarSi9IpdTpOKQIRGSki6SKSJSKP1/H87SJSICIb7Le7ajx3m4hk2m+3OSOP1TLzSpj2QzZX94lmYEedf9gqoc0CePLKrqzfc5gZa/ZYHUcpl+VwEYiIL/AmcDmQAtwgIil1rDrLGNPLfnvPvm0Y8AzQH+gHPCMibv1tK2MMT36VRrMAP568oqvVcbzeuN7RDEwMZ9J32ykoOWF1HKVckjNGBP2ALGNMtjGmHJgJjKnntiOABcaYImPMIWABMNIJmSwzd+M+Vu8s4q+XdyG8eaDVcbyeiPD8uO6Unaxi0jy9KJ1SdXFGEUQDOTUe59qX1fY7EdkkIrNFJLaB2yIi40UkVURSCwoKnBDb+corqnj5+3S6tW/JtbbYs2+gmkTHyObcPiieL9blkpFXYnUcpVyOM4qgrpPjax+Z+xqIN8b0ABYCHzVg2+qFxkwzxtiMMbbIyMhzDtuYZqzeQ07Rcf4ysgs+PvqdAVdy30UdCQnw46X56VZHUcrlOKMIcoGaH39jgF9NF2WMOWiMObWD9l2gb323dRfHTlQwZXEmAxLDGJIcYXUcVUvrkADuuSiRBVvzWLtbL1WtVE3OKII1QLKIJIhIAHA9MLfmCiJS89oKo4Ft9vvzgeEi0tp+kHi4fZnb+ffynRQeLecvI7voN4hd1B2DEohoHsikedt1jmOlanC4CIwxFcAEqt/AtwGfGmO2iMizIjLavtqDIrJFRDYCDwK327ctAp6jukzWAM/al7mVomPlTPshmxHd2tJHLzHtskIC/Xjw0iRW7yxiWYZrHmdSygrijp+MbDabSU1NtTrGL57/Ziv//mkn8/84hOS2LayOo86gvKKKYa8sIyTQj/9NHKzHcpRXEZG1xhhb7eX6zWIH7T18nOk/7+Z3fWK0BNxAgJ8PfxreiW37j/D1Jrc8HKWU02kROOj1hRlg4I+XdbI6iqqnq3q0p0u7Fvzr+wzKK3RaS6W0CByQlV/C7LW53DKwA9GhwVbHUfXk4yM8NrILe4pKmZWac/YNlPJwWgQOeHl+Bs0C/HjgkiSro6gGurhzJP3iw5i8KJPScp2zQHk3LYJztH7PIeZtOcD4IYmEhQRYHUc1kIjw2OWdKSg5wQc/7bI6jlKW0iI4B8YYJs3bTnhIAHcOTrA6jjpHfTuEMaxrW95euoNDx8qtjqOUZbQIzsGPmYX8nF3ExKFJhAT6WR1HOeDREZ05Wl7B28t2WB1FKctoETRQVZXhn/O3E9M6mBv6x1kdRzmoc7sWjOsdzYcrdrG/+LjVcZSyhBZBA/1v837S9h7hT8M76cxjHuLhYZ2oMobXF+r8xso7aRE0wMnKKv71fTpd2rVgdM86r5at3FBsWDNu6t+BT1NzyMo/anUcpZqcFkEDfJqaw66DpTw6ojO+emkCjzJhaBLB/r68skAvU628jxZBPR0vr+T1hZnYOrRmaJc2VsdRThbRPJC7Lkzk280H2Jhz2Oo4SjUpLYJ6+nDFLvJLTvDY5XqZaU9114UJhIUE6OQ1yutoEdRDcelJ3lqaxaVd2nB+fJjVcVQjaRHkzwOXJLE8q5DlmYVWx1GqyWgR1MNby3ZQcqKCP4/obHUU1chu6h9HdGiwTl6jvIpTikBERopIuohkicjjdTz/iIhstU9ev0hEOtR4rlJENthvc2tva7UDxWV88NNOxvaKpmtUS6vjqEYW5O/LH4cls3lvMd+lHbA6jlJNwuEiEBFf4E3gciAFuEFEUmqtth6w2Sevnw38s8Zzx40xvey30biYyYszqTKGR/Qy017j6j4xJLdpzsvz06mo1MtUK8/njBFBPyDLGJNtjCkHZgJjaq5gjFlijCm1P/yZ6knqXd6eg6XMWpPDjf3iiA1rZnUc1UR8fYQ/j+hMduExvli31+o4SjU6ZxRBNFDzou659mWncyfwXY3HQSKSKiI/i8jY020kIuPt66UWFDTNfLNvLsnC10e4Xy8z7XWGp7Sle3RL3liSxUkdFSgP54wiqOtcyjqPsonIzYANeKnG4jj7HJo3Aq+JSMe6tjXGTDPG2IwxtsjISEczn1VOUSmfr8vlxn5xtG0Z1Oivp1yLiPDg0GT2FJUyZ4NOaak8mzOKIBeIrfE4BvjN/zkiMgx4EhhtjDlxarkxZp/9ZzawFOjthEwOm7o0Cx8R7r2ozl5SXuCylLakRLXkjcWZeqxAeTRnFMEaIFlEEkQkALge+NXZPyLSG3iH6hLIr7G8tYgE2u9HAIOArU7I5JDcQ6V8lprL9f1iaddKRwPeSkR48NJkdh0sZe5GHRUoz+VwERhjKoAJwHxgG/CpMWaLiDwrIqfOAnoJaA58Vus00a5AqohsBJYALxpjLC+CqUt34CPCfRfraMDbDU9pS5d2LXhjcRaVVfq9AuWZnDKrijHmW+DbWsuernF/2Gm2WwGc54wMzrL38HE+S83huvNjiWqlE9J7Ox8f4aFLk7nvk3V8vXEfY3vrVWeV59FvFtfy1tIsAO67WM8UUtVGdGtH57YtmLw4U0cFyiNpEdSwv/g4n67J5fe2WKJDdTSgqvn4VB8ryC44xjeb9FiB8jxaBDW8tXQHVcZwn54ppGq5vHs7OrVtzhQ9VqA8kBaB3YHiMmauzuGavjH6LWL1Gz4+wsShyWTlH+XbzfutjqOUU2kR2L29rHo08IB+i1idxhXnRZHUpjlTFmdSpaMC5UG0CIC8I2X8d/Ueru4TraMBdVq+PsLEoUlk5B3VK5Mqj6JFALyzLJvKKsOES5KtjqJc3Kge7ekYGcLkRToqUJ7D64sgv6SMT1btZlzvaOLCdTSgzszXfqwgPa+E+Vt0VKA8g9cXwbRl2VRUGSbosQFVT1f1bE9iRAiv66hAeQivLoKCkhP8Z9VuxvRqT3xEiNVxlJvw9REeuCSJ7QdKWLAtz+o4SjnMq4vg3R+zKa+oYuJQPTagGmZMr/bEhzdj8qJMndtYuT2vLYLCoyf4eOVuxvSKJkFHA6qB/Hx9eOCSJLbsO8LCbfln30ApF+a1RfDuj9mUVVTq9wbUORvXO5q4sGa8vihDRwXKrXllERQdK+fjlbu5qkd7kto0tzqOclN+vj5MuCSJtL1HWLxdRwXKfXllEbz7YzbHT1by4KU6GlCOGdcnmtiwYF7XYwXKjXldERw6Vs70Fbu48rwoktq0sDqOcnP+vj48cHESm3KLWZpeYHUcpc6JU4pAREaKSLqIZInI43U8Hygis+zPrxKR+BrP/dW+PF1ERjgjz5m8tzyb0pOVPHipnimknOPqPjFEhwbzmo4KlJtyuAhExBd4E7gcSAFuEJGUWqvdCRwyxiQBrwKT7NumUD3HcTdgJDDV/vsaxeHScj5asZsrukfRqa2OBpRzBPhVn0G0MecwyzJ0VKDcjzNGBP2ALGNMtjGmHJgJjKm1zhjgI/v92cClIiL25TONMSeMMTuBLPvvaxT/Xr6ToycqmKjHBpSTXdO3elSgxwpUY8nKL+GOD1az52Cp03+3M4ogGsip8TjXvqzOdeyT3RcD4fXcFgARGS8iqSKSWlBwbp+6Dh4r58oeUXRp1/KctlfqdAL8fLjv4o6s33OYHzMLrY6jPNCUxVn8nF1ESKDzd5o4owikjmW1PxKdbp36bFu90JhpxhibMcYWGRnZwIjVXhh3HpOv731O2yp1Nr+3xRDVKkhHBcrpdhQc5euN+7h1YAfCmwc6/fc7owhygdgaj2OA2hO7/rKOiPgBrYCiem7rVL4+dXWPUo4L9PPl/os7snb3IVbsOGh1HOVB3licRaCfL3cPSWyU3++MIlgDJItIgogEUH3wd26tdeYCt9nvXwMsNtUfmeYC19vPKkoAkoHVTsiklCWuPT+Wdi2DeH2hjgqUc2QXHGXOhr3cPCCOiEYYDYATisC+z38CMB/YBnxqjNkiIs+KyGj7au8D4SKSBTwCPG7fdgvwKbAVmAc8YIypdDSTUlYJ9PPlvos7snpXESuzdVSgHPfmkh0E+PkwfkjHRnsNccdPLTabzaSmplodQ6k6lZ2sZMg/l5AQEcKsewZaHUe5sd0HjzH0X8u4/YJ4nhpV+6z8hhORtcYYW+3lXvfNYqUaW5C/L/de1JFVO4v4WUcFygFvLM7Cz0e456LGOTZwihaBUo3gxv5xRLYI5PWFmVZHUW5qz8FSvli/lxv7x9GmRVCjvpYWgVKNIMjfl3uGJLIy+yCrdxZZHUe5oTeXZOHrI9x7UeMdGzhFi0CpRnJT/w5ENA/k9UUZVkdRbianqJTP1+Vyw/mxtG3ZuKMB0CJQqtEEB1SPCn7KOkjqLh0VqPqbujQLHxHuvbjxRwOgRaBUo7ppQBzhIQG8vkiPFaj6yT1UymepuVx3fixRrYKb5DW1CJRqRM0C/Bg/JJEfMwtZu/uQ1XGUG5i6dAcicF8TjQZAi0CpRnfzgA6EhQQwWUcF6iz2Hj7OZ6k5XGuLpX1o04wGQItAqUYXEujHXRcmsCyjgA05h62Oo1zY20t3AHD/JU17qXwtAqWawK0D4wlt5s/rC/UMIlW3/cXHmbUmh2v6xhLdhKMB0CJQqkk0D/Tj7gsTWZJewEYdFag6vL10B1XGcH8THhs4RYtAqSZy68AOtAr212MF6jfyjpQxY00O1/SNITasWZO/vhaBUk2kRZA/dw1OYNH2fDbnFlsdR7mQt5buoKrK8EATHxs4RYtAqSZ026B4Wgb56fcK1C/yj5QxY/Ueru4TbcloALQIlGpSLYP8uXNwIgu35ZG2V0cFCt5elk2FhaMB0CJQqsndPiieFkF+eqxAkV9SxierdjO2VzQdwkMsy+FQEYhImIgsEJFM+8/WdazTS0RWisgWEdkkItfVeO5DEdkpIhvst16O5FHKHbQK9ueOQQl8vzWPrfuOWB1HWWjasmxOVlYxYah1owFwfETwOLDIGJMMLLI/rq0UuNUY0w0YCbwmIqE1nn/UGNPLftvgYB6l3MKdgxJoEejHlMU6KvBWhUdP8B/7aCAhwrrRADheBGOAj+z3PwLG1l7BGJNhjMm0398H5AORDr6uUm6tVTN/bh8Uz3dpB9h+QEcF3ujdH7Ipr7B+NACOF0FbY8x+APvPNmdaWUT6AQHAjhqLX7DvMnpVRALPsO14EUkVkdSCggIHYytlvTsHJ9A80I/XFuiowNsUlJxg+srdjO7ZnsTI5lbHOXsRiMhCEUmr4zamIS8kIlHAx8Adxpgq++K/Al2A84Ew4LHTbW+MmWaMsRljbJGROqBQ7i+0WQB3Dk5g3pYD+m1jL/PG4kzKK6t4aFgnq6MA9SgCY8wwY0z3Om5zgDz7G/ypN/r8un6HiLQE/gf8nzHm5xq/e7+pdgL4AOjnjD9KKXdx14UJhIUE8M/5262OoprInoOl/Hf1Hq47P9byYwOnOLpraC5wm/3+bcCc2iuISADwJTDdGPNZredOlYhQfXwhzcE8SrmVFkH+PHBJEj9lHWR5ZqHVcVQTeHVhBj4iPHRpstVRfuFoEbwIXCYimcBl9seIiE1E3rOvcy0wBLi9jtNEPxGRzcBmIAJ43sE8SrmdmwfEER0azKR52zHGWB1HNaJt+4/w1Ya93DEooUnmIq4vP0c2NsYcBC6tY3kqcJf9/n+A/5xm+6GOvL5SniDQz5eHL+vEnz/byHdpB7jivCirI6lG8vL8dFoE+nHfRU1/hdEz0W8WK+UCxvWOJrlNc16en05FZdXZN1BuZ82uIhZtz+e+i5No1czf6ji/okWglAvw9REeHdGZ7MJjfLY21+o4ysmMMUz6bjttWgRy+wXxVsf5DS0CpVzEZSlt6RMXymsLMyg7WWl1HOVEi7fnk7r7EA8NSyY4wNfqOL+hRaCUixARHhvZhbwjJ/hoxS6r4ygnqawy/HNeOvHhzbjWFmt1nDppESjlQvonhnNx50imLt1B8fGTVsdRTjB3417S80r40/DO+Pu65luua6ZSyos9OqIzxcdPMu2HHWdfWbm08ooq/vV9Bt2jW3KlC58NpkWglIvp1r4Vo3u25/3lO8k/UmZ1HOWA/67aTe6h4/xlRBd8fMTqOKelRaCUC/rT8E5UVBom62Wq3dbRExVMWZzFwMRwLkyOsDrOGWkRKOWCOoSHcEO/OGauzmFX4TGr46hz8O/lOzl4rJy/jOxM9VV0XJcWgVIuauLQJPx9fXhlQYbVUVQDFR0rZ9oP2Yzo1pbecb+ZuNHlaBEo5aLatAziD4PjmbtxH1v26UT37mTqkixKyyt4dERnq6PUixaBUi5s/JCOtAr256X56VZHUfW09/Bxpv+8m2v6xpDUpoXVcepFi0ApF9Yq2J8HLunI0vQCfs4+aHUcVQ+v2XflucqkM/WhRaCUi7t1YDztWgbpZardQGZeCZ+vy+XWAR2IDg22Ok69aREo5eKC/H3547Bk1u85zIKteVbHUWfw8vfpNAvw4/5LrJ+QviEcKgIRCRORBSKSaf9Z5+FxEamsMSnN3BrLE0RklX37WfbZzJRStVzTN4bEiBBemp9OZZWOClzR+j2HmL8lj/FDEgkLca+3MkdHBI8Di4wxycAi++O6HDfG9LLfRtdYPgl41b79IeBOB/Mo5ZH8fH3484jOZOYf5cv1e62Oo2oxxjBp3nYimgdw5+AEq+M0mKNFMAb4yH7/I6rnHa4X+zzFQ4HZ57K9Ut7m8u7t6BHTilcXZHCiQi9T7Up+zCzk5+wiJg5NJiTQoYkfLeFoEbQ1xuwHsP9sc5r1gkQkVUR+FpFTb/bhwGFjTIX9cS4QfboXEpHx9t+RWlBQ4GBspdzPqctU7z18nPd+3Gl1HGV3srKKF/63jZjWwdzQL87qOOfkrNUlIguBdnU89WQDXifOGLNPRBKBxfYJ64/Usd5pd34aY6YB0wBsNpvuJFVeaVBSBCO7tWPK4kxG92xPbFgzqyN5vfeX7yQ9r4R3b7UR4NZOGzcAAA+WSURBVOee59+cNbUxZpgxpnsdtzlAnohEAdh/5p/md+yz/8wGlgK9gUIgVEROlVEMsM/hv0gpD/fM6BR8RXhqTpqeTmqxnKJSXluYwfCUtlyW0tbqOOfM0fqaC9xmv38bMKf2CiLSWkQC7fcjgEHAVlP9X/AS4Jozba+U+rWoVsE8MrwzS9ML+C7tgNVxvJYxhmfmbsFHhL+N7mZ1HIc4WgQvApeJSCZwmf0xImITkffs63QFUkVkI9Vv/C8aY7ban3sMeEREsqg+ZvC+g3mU8gq3DexAt/Yt+fvXWygp05nMrDB/ywEWb8/nkcs60d6NvjxWF3HHoaXNZjOpqalWx1DKUhtyDjNu6k/cNjDe7T+RupujJyoY9q9ltA4J4OsJg/Bz0SkoaxORtcYYW+3l7pFeKfUbvWJDuWVAB6av3MXmXL06aVN65fsM8krK+Me47m5TAmfi/n+BUl7szyM6E948kCe+3KzfOG4iaXuL+XDFTm7qH+cWcw3UhxaBUm6sZZA/T49KYfPeYj5eucvqOB6vssrw5JebCQsJ5NERXayO4zRaBEq5uVE9ohjSKZKXv8/gQLFOdt+YPlm1m425xTw1qiutgv2tjuM0WgRKuTkR4bkx3ThZWcWz32yxOo7HyjtSxkvz0hmcFMHonu2tjuNUWgRKeYAO4SFMHJrEt5sPsGR7nd/rVA567putnKis4vmx3V1+MvqG0iJQykOMH9KRpDbNeWpOGsfL9aJ0zrQso4BvNu1nwiVJxEeEWB3H6bQIlPIQAX4+PD+2O7mHjjNlcabVcTxG2clKnvoqjcTIEO65KNHqOI1Ci0ApDzIgMZxr+sYw7YdsMvJKrI7jEd5YnMWeolKeH9udQD9fq+M0Ci0CpTzME1d0pXmQH09+uZkq/W6BQ7LyS3jnhx1c3TuaCzpGWB2n0WgRKOVhwkICeOLyrqzZdYjZa3OtjuO2jDE88WUazQL8eOLKrlbHaVRaBEp5oGv6xnB+fGv+8d02Dh49YXUctzR7bS6rdxbx+OVdiGgeaHWcRqVFoJQH8vERXhh3HkfLKvjHt9utjuN2io6V849vt2Hr0JrrbLFWx2l0WgRKeahObVswfkgin6/LZeWOg1bHcSsvfreNkrIKnh/XHR8fz/rOQF20CJTyYBOHJhMbFsyTX22mtLzi7BsoVmQV8mlqLndemECXdi2tjtMktAiU8mDBAb68eHUPdhUe49HPNunUlmeRe6iUCTPW0zEyhIcuTbY6TpNxqAhEJExEFohIpv3nb67JKiKXiMiGGrcyERlrf+5DEdlZ47lejuRRSv3WoKQIHhvZhf9t3s/UpTusjuOyjpdXcs/HazlZWcW7t9poFuB39o08hKMjgseBRcaYZGCR/fGvGGOWGGN6GWN6AUOBUuD7Gqs8eup5Y8wGB/Mopeowfkgio3u25+Xv0/VaRHUwxvDY55vYuv8Ik6/vTWJkc6sjNSlHi2AM8JH9/kfA2LOsfw3wnTGm1MHXVUo1gIgw6Xc9SIlqyYMz15NdcNTqSC5l2g/ZzN24jz8P78wlXdpYHafJOVoEbY0x+wHsP8/2L3g9MKPWshdEZJOIvCoipz1ZV0TGi0iqiKQWFBQ4llopLxQc4Ms7t/TF39eHu6en6qT3dssyCpg0bztX9oji/os7Wh3HEmctAhFZKCJpddzGNOSFRCQKOA+YX2PxX4EuwPlAGPDY6bY3xkwzxtiMMbbIyMiGvLRSyi6mdTPevLEPuw6W8vCsDV5/CYpdhceY+N91dGrbgpeu6eFxl5eur7MWgTFmmDGmex23OUCe/Q3+1Bv9mXY+Xgt8aYz55WOIMWa/qXYC+ADo59ifo5Q6m4Edw3l6VAoLt+Xz2sIMq+NY5uiJCu6enoqvj3jdweHaHN01NBe4zX7/NmDOGda9gVq7hWqUiFB9fCHNwTxKqXq4dWAHrrXFMHlxFvPS9lsdp8lVVRkembWB7MJjvHljH2LDmlkdyVKOFsGLwGUikglcZn+MiNhE5L1TK4lIPBALLKu1/ScishnYDEQAzzuYRylVDyLCc2O70ys2lEc+3Uj6Ae+6ZPWUxVl8vzWPJ6/oygVJnntV0foSd/yCic1mM6mpqVbHUMrt5R0p46opywny92XuhEGENguwOlKj+37LAcZ/vJbf9Ynh5d9713EBEVlrjLHVXq7fLFbKi7VtGcTbt/TlQHEZE2esp6KyyupIjSozr4SHZ22gZ0wrXhjneXMPnystAqW8XJ+41jw3ths/Zhbyz/npVsdpNMXHT3L39FSCA/x4+5a+BPl75mxj58J7D5MrpX5x3flxbNl3hGk/ZJMS1ZKxvaOtjuRUlVWGB2esZ+/h48y4ewBRrYKtjuRSdESglALgqVEp9EsI47HPN7E5t9jqOE710vx0lmUU8PfR3bHFh1kdx+VoESilAPD39WHqTX0IDwngno9TKfSQmc2+3riPt5ft4Kb+cdzYP87qOC5Ji0Ap9YuI5oFMu9VGUWk5N7+3it0Hj1kdySFfrs/lz59t5Pz41jxzVTer47gsLQKl1K90j27Fu7fa2F9cxqgpy1m0Lc/qSA1WXlHFU1+l8fCsjfSKDeWdW2wE+Onb3enov4xS6jcuTI7km4mD6RDejDs/SuXl+elUusl1ifYXH+fad1by8c+7uWdIIp/c1Z+wEM//foQjtAiUUnWKDWvG7Hsv4DpbLG8syeL2D1ZTdKzc6lhntCKrkFGTl5OVf5S3burDX6/oip+vvs2djf4LKaVOK8jfl0nX9ODFq89j1c4irpqynI05h62O9RvGGKYuzeLm91cRFhLAnAmDuPy8KKtjuQ0tAqXUWV3fL47P770AgN+/vZL/rtrjMvMfHyk7yfiP1/LPeelccV4UXz0wiI5eNsOYo7QIlFL1cl5MK76ZOJgBHcN54svNPDp7E2UnKy3NtP3AEUZPWc6S7fk8PSqFKTf0JiRQvyfbUFoESql6ax0SwAe3n8+DlyYze20uV09dwZ6D1sw8+9X6vYx98ydKyyuZMX4AfxicoNcOOkdaBEqpBvH1ER65rBMf3H4+uYdKGTXlRxZvb7pTTMsrqnh6Thp/nLWBHjGhfPPgYM7Xbws7RItAKXVOLunShm8mXkhM62b84cNUXvk+vdF3FeUUlXLdtJVMX7mbuy9M4JO7+tOmRVCjvqY30PkIlFIOKTtZyf99lcbstbkE+vlgi2/NoKQIBidF0K19K3x9zn13TUnZSVZlF7E8q5CfsgrJzD9KSIAv/7ymJ1f20LOCGup08xE4VAQi8nvgb0BXoJ8xps53ZxEZCbwO+ALvGWNOzWSWAMykeuL6dcAtxpiznqisRaCUazHGsGLHQRZvz+enrEK222c8axXszwUdw38phg7hzc64H7+8oooNOYd/eePfkHOYyipDkL8P/RLCGdQxnCvOi/L6qSXPVWMVQVegCngH+HNdRSAivkAG1VNZ5gJrgBuMMVtF5FPgC2PMTBF5G9hojHnrbK+rRaCUa8svKWPljoMsz6x+Q99XXAZAdGgwg5MiGJQcwQUdwwkPCSA9r+SX9VbtLKK0vBIfgR4xodXrJkXQp0MogX46f4CjTlcEDp1nZYzZZv/lZ1qtH5BljMm2rzsTGCMi24ChwI329T6ienRx1iJQSrm2Ni2CGNMrmjG9ojHGsOtgafWn/MxCvkvbz6zUHABaBvlxpKwCgMTIEK7pG8OgpAgGJIbTKtjfyj/BqzTFCbfRQE6Nx7lAfyAcOGyMqaix/LSzYYjIeGA8QFycXkpWKXchIiREhJAQEcItAzpQWWVI21vM8qxC9hws/eWYQvtQnSzGKmctAhFZCLSr46knjTFz6vEadQ0XzBmW18kYMw2YBtW7hurxukopF+TrI/SMDaVnbKjVUZTdWYvAGDPMwdfIBWJrPI4B9gGFQKiI+NlHBaeWK6WUakJN8T2CNUCyiCSISABwPTDXVB+lXgJcY1/vNqA+IwyllFJO5FARiMg4EckFBgL/E5H59uXtReRbAPun/QnAfGAb8KkxZov9VzwGPCIiWVQfM3jfkTxKKaUaTr9QppRSXuJ0p4/qJSaUUsrLaREopZSX0yJQSikvp0WglFJezi0PFotIAbD7HDePoPo7DO7K3fOD+/8N7p4f3P9vcPf8YM3f0MEYE1l7oVsWgSNEJLWuo+buwt3zg/v/De6eH9z/b3D3/OBaf4PuGlJKKS+nRaCUUl7OG4tgmtUBHOTu+cH9/wZ3zw/u/ze4e35wob/B644RKKWU+jVvHBEopZSqQYtAKaW8nFcVgYiMFJF0EckSkcetztMQIvJvEckXkTSrs5wLEYkVkSUisk1EtojIQ1ZnaigRCRKR1SKy0f43/N3qTOdCRHxFZL2IfGN1lnMhIrtEZLOIbBARt7v6pIiEishsEdlu//9hoOWZvOUYgYj4AhnAZVRPlrMGuMEYs9XSYPUkIkOAo8B0Y0x3q/M0lIhEAVHGmHUi0gJYC4x1l39/AKmenDvEGHNURPyB5cBDxpifLY7WICLyCGADWhpjRlmdp6FEZBdgM8a45RfKROQj4EdjzHv2OVqaGWMOW5nJm0YE/YAsY0y2MaYcmAmMsThTvRljfgCKrM5xrowx+40x6+z3S6iem+K0c1S7IlPtqP2hv/3mVp+kRCQGuBJ4z+os3khEWgJDsM+9Yowpt7oEwLuKIBrIqfE4Fzd7I/IUIhIP9AZWWZuk4ey7VTYA+cACY4y7/Q2vAX8BqqwO4gADfC8ia0VkvNVhGigRKAA+sO+ee09EQqwO5U1FIHUsc6tPc55ARJoDnwN/NMYcsTpPQxljKo0xvaieY7ufiLjNbjoRGQXkG2PWWp3FQYOMMX2Ay4EH7LtN3YUf0Ad4yxjTGzgGWH680puKIBeIrfE4BthnURavZN+v/jnwiTHmC6vzOMI+nF8KjLQ4SkMMAkbb97HPBIaKyH+sjdRwxph99p/5wJdU7/Z1F7lAbo2R5Gyqi8FS3lQEa4BkEUmwH6C5HphrcSavYT/Q+j6wzRjzitV5zoWIRIpIqP1+MDAM2G5tqvozxvzVGBNjjImn+r//xcaYmy2O1SAiEmI/2QD7LpXhgNucSWeMOQDkiEhn+6JLActPmPCzOkBTMcZUiMgEYD7gC/zbGLPF4lj1JiIzgIuBCBHJBZ4xxrxvbaoGGQTcAmy272MHeMIY862FmRoqCvjIfgaaD/CpMcYtT8F0Y22BL6s/V+AH/NcYM8/aSA02EfjE/oE0G7jD4jzec/qoUkqpunnTriGllFJ10CJQSikvp0WglFJeTotAKaW8nBaBUkp5OS0CpZTycloESinl5f4fTW8RYgcaEuUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "b = torch.sin(a)\n",
    "plt.plot(a.detach(), b.detach())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a closer look at the tensor `b`. When we print it, we see an indicator that it is tracking its computation history:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([ 0.0000e+00,  2.5882e-01,  5.0000e-01,  7.0711e-01,  8.6603e-01,\n",
      "         9.6593e-01,  1.0000e+00,  9.6593e-01,  8.6603e-01,  7.0711e-01,\n",
      "         5.0000e-01,  2.5882e-01, -8.7423e-08, -2.5882e-01, -5.0000e-01,\n",
      "        -7.0711e-01, -8.6603e-01, -9.6593e-01, -1.0000e+00, -9.6593e-01,\n",
      "        -8.6603e-01, -7.0711e-01, -5.0000e-01, -2.5882e-01,  1.7485e-07],\n",
      "       grad_fn=<SinBackward>)\n"
     ]
    }
   ],
   "source": [
    "print(b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This `grad_fn` gives us a hint that when we execute the backpropagation step and compute gradients, we'll need to compute the derivative of $sin(x)$ for all this tensor's inputs.\n",
    "\n",
    "Let's perform some more computations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([ 0.0000e+00,  5.1764e-01,  1.0000e+00,  1.4142e+00,  1.7321e+00,\n",
      "         1.9319e+00,  2.0000e+00,  1.9319e+00,  1.7321e+00,  1.4142e+00,\n",
      "         1.0000e+00,  5.1764e-01, -1.7485e-07, -5.1764e-01, -1.0000e+00,\n",
      "        -1.4142e+00, -1.7321e+00, -1.9319e+00, -2.0000e+00, -1.9319e+00,\n",
      "        -1.7321e+00, -1.4142e+00, -1.0000e+00, -5.1764e-01,  3.4969e-07],\n",
      "       grad_fn=<MulBackward0>)\n",
      "tensor([ 1.0000e+00,  1.5176e+00,  2.0000e+00,  2.4142e+00,  2.7321e+00,\n",
      "         2.9319e+00,  3.0000e+00,  2.9319e+00,  2.7321e+00,  2.4142e+00,\n",
      "         2.0000e+00,  1.5176e+00,  1.0000e+00,  4.8236e-01, -3.5763e-07,\n",
      "        -4.1421e-01, -7.3205e-01, -9.3185e-01, -1.0000e+00, -9.3185e-01,\n",
      "        -7.3205e-01, -4.1421e-01,  4.7684e-07,  4.8236e-01,  1.0000e+00],\n",
      "       grad_fn=<AddBackward0>)\n"
     ]
    }
   ],
   "source": [
    "c = 2 * b\n",
    "print(c)\n",
    "\n",
    "d = c + 1\n",
    "print(d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's compute a single-element output. When you call `.backward()` on a tensor with no arguments, it expects the calling tensor to contain only a single element, as is the case when computing a loss function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(25.0000, grad_fn=<SumBackward0>)\n"
     ]
    }
   ],
   "source": [
    "out = d.sum()\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each `grad_fn` stored with our tensors allows you to walk the computation all the way back to its inputs with its `next_functions` property. We can see below that drilling down on this property on `d` shows us the gradient functions for all the prior tensors. Note that `a.grad_fn` is reported as `None`, indicating that this was an input to the function with no history of its own."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "d:\n",
      "<AddBackward0 object at 0x7fa00048dfd0>\n",
      "((<MulBackward0 object at 0x7fa00048d3a0>, 0), (None, 0))\n",
      "((<SinBackward object at 0x7fa00048dfd0>, 0), (None, 0))\n",
      "((<AccumulateGrad object at 0x7fa00048d280>, 0),)\n",
      "()\n",
      "\n",
      "c:\n",
      "<MulBackward0 object at 0x7fa0004a8070>\n",
      "\n",
      "b:\n",
      "<SinBackward object at 0x7fa0004a8070>\n",
      "\n",
      "a:\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "print('d:')\n",
    "print(d.grad_fn)\n",
    "print(d.grad_fn.next_functions)\n",
    "print(d.grad_fn.next_functions[0][0].next_functions)\n",
    "print(d.grad_fn.next_functions[0][0].next_functions[0][0].next_functions)\n",
    "print(d.grad_fn.next_functions[0][0].next_functions[0][0].next_functions[0][0].next_functions)\n",
    "print('\\nc:')\n",
    "print(c.grad_fn)\n",
    "print('\\nb:')\n",
    "print(b.grad_fn)\n",
    "print('\\na:')\n",
    "print(a.grad_fn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With all this machinery in place, how do we get derivatives out? You call the `backward()` method on the output, and check the input's `grad` property to inspect the gradients:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([ 2.0000e+00,  1.9319e+00,  1.7321e+00,  1.4142e+00,  1.0000e+00,\n",
      "         5.1764e-01, -8.7423e-08, -5.1764e-01, -1.0000e+00, -1.4142e+00,\n",
      "        -1.7321e+00, -1.9319e+00, -2.0000e+00, -1.9319e+00, -1.7321e+00,\n",
      "        -1.4142e+00, -1.0000e+00, -5.1764e-01,  2.3850e-08,  5.1764e-01,\n",
      "         1.0000e+00,  1.4142e+00,  1.7321e+00,  1.9319e+00,  2.0000e+00])\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7fa0004bbfd0>]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD4CAYAAADvsV2wAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3dd3yV5f3/8dcng4SVAFkEws5gzxRU0CIgIKJUq1brqG0t5asiVmu1am31W7usHU5EcIF1fFVELYoLRYZoQJQNIawww0rIzkk+vz9y9EdpAoRzkuuMz/PxyCMn59zkfsfx5sp17vu6RFUxxhgT+iJcBzDGGNM0rPCNMSZMWOEbY0yYsMI3xpgwYYVvjDFhIsp1gBNJTEzUrl27uo5hjDFBY8WKFQdUNamu1wK68Lt27UpOTo7rGMYYEzREZHt9r9mUjjHGhAkrfGOMCRNW+MYYEyas8I0xJkxY4RtjTJjwufBFpJOILBSR9SKyVkSm1XGMiMjDIpIrIl+LyGBfz2uMMaZh/HFZpge4TVVXikhrYIWIvK+q64455nwgw/sxDHjC+9kYY0wT8XmEr6p7VHWl9/FRYD3Q8bjDJgHPa63PgDYikurruevzyIebefOr3RwormisUxhjjN/tLypn7pf5TP9kS6N8f7/eeCUiXYFBwPLjXuoI7Dzm63zvc3vq+B6TgckAnTt3bnCG8qpqnl6ylcOlVQD0So1jRHoCw9MTGdqtHS2aBfS9ZsaYMFJc4WF53kEW5x5gSe4BNu0rBqB9XCw/O7s7kRHi1/P5rf1EpBXwGnCLqhYd/3Idf6TOnVdUdQYwAyA7O7vBu7PERkeSc895rNlV+O0/xOeWbuepT7cSHSkM6tyWEemJDE9PZEBaPFGR9r61MaZpVFXXsGrnERZvru2mVTuP4KlRYqIiGNqtHd8fnMbw9ER6p8YR4eeyBxB/7HglItHA28ACVf1bHa8/CXysqi96v94IjFTV/xrhHys7O1v9sbRCWWU1OdsPffsXwNrdRahC65gohnVPYER6AmdnJtEjqZXP5zLGmGNt3neURd6CX553kJLKaiIE+qW1+Xb2YXDntsRGR/rlfCKyQlWz63rN5xG+iAgwC1hfV9l7vQncJCIvUftmbeHJyt6fmjeL5OyMJM7OqF1P6HBJJcuO+TXqg/X7AJh8Tnd+NS7LRv3GGJ9VeKr5/dvrmf1Z7dI23RNbcol3BH9m9wTiW0Q3eSZ/TOkMB64BVovIKu9zdwGdAVR1OjAfmADkAqXAj/1w3tPWtmUzJvRLZUK/2veNdx4q5clFW5ixKI+vdh7h0R8OJql1jMuIxpggtvtIGTe8sJJVO49w/Yhu/HhENzq2ae46ln+mdBqLv6Z0TtXrK/O5a+5q4ptH8/hVgxnSpV2TndsYExqW5B5g6otfUlFVzV8vG8D5/RrtgsQ6nWhKx+YujnHJ4DTm3jCc2OhIfvDkZzyzZCuB/BeiMSZwqCqPf5zLNbOW065lM+bdNKLJy/5krPCP0ys1jjdvGsHIrGTue2sd015aRUmFx3UsY0wAKyqvYvLsFfzl3Y1M6JfKvBuHk54ceBeB2EXpdYhvHs2Ma4bwxCdbeOi9jWzYW8T0q4fQ3a7iMcYcZ8PeIqbMXkH+4TLundibHw/vSu21LIHHRvj1iIgQbjw3ned/MowDxZVc9OgS3l2z13UsY0wAeePLXXzvsSWUVlbz4uQz+MmIbgFb9mCFf1IjMhJ5e+oIeiS3YsqcFfzxnfV4qmtcxzLGOFTpqeHeeWu45eVV9E9rw9s3j+A7XQP/Ig8r/FPQoU1zXvn5GVw1rDNPfpLHNbM+t3V6jAlTewrL+MGMZTy/bDvXj+jGC9cPI7l1rOtYp8QK/xTFREXywMX9eOiyAazccZiJDy9mxfbDrmMZY5rQ0i0HmPjwYjbtPcpjPxzMPRN7Ex1EN2oGT9IA8f0htZduNouK4OqZy1m/5/hlg4wxoejLHYe57ukvaNuyGfNuGs4F/QPrkstTYYV/Gnp3iOPVKWcS1zyKybNzOFxS6TqSMaYR7S8qZ8qcFaTEx/B/Pz+T9OTWriOdFiv805QcF8v0q4ewr7CCm15caW/kGhOiKjzVTJmzgqIyDzOuyaZty2auI502K3wfDOrclgcu7suS3IP88Z0NruMYY/xMVfntvLWs3HGEhy4fQK/UONeRfGI3XvnosuxOrN1dxKzFW+mdGsf3h6S5jmSM8ZM5n23npS92ctO56d8uthjMbITvB3df0Iszuyfw67mr+Tr/iOs4xhg/WJ53kPveWsfonsncel6m6zh+YYXvB9GRETx21WCSWsXw89krKDhq1+gbE8x2eZc37pzQgr9fMbBRdp9ywQrfT9q1bMaMa4dwuLSS/5mzgkqPvYlrTDAqq6zm57NzqPTU8NS12cTFNv1GJY3FCt+P+nSI58FLB5Cz/TD3vbXWdRxjTAOpKne+/jVrdxfxzysHhty2p34pfBF5WkT2i8iael4fKSKFIrLK+3GvP84biC4c0IEp3+3BC8t38K/lO1zHMcY0wMxPtzJv1W5+OTaLUT1TXMfxO3+N8J8Fxp/kmE9VdaD3434/nTcg3T4ui5FZSfz2zTXkbDvkOo4x5hQs2lTAH99Zz4R+7blhZA/XcRqFXwpfVRcB1mxekRHCP68YRFrbFkyZs5I9hWWuIxljTmD7wRKmvvglmSmtefDSAQG9xLEvmnIO/0wR+UpE3hGRPvUdJCKTRSRHRHIKCgqaMJ5/fbOJSlmlhymzV1BeVe06kjGmDsUVHn72fA4i8NS12bSMCd3bk5qq8FcCXVR1APAI8EZ9B6rqDFXNVtXspKSkJorXODJSWvP3Hwzkq/xC7p67xvbHNSbA1NQot72yii0FJTz2w8F0atfCdaRG1SSFr6pFqlrsfTwfiBaRxKY4t2tj+7TnF2MyeW1lPs8s2eY6jjHmGI8uzGXB2n3cNaEXw9NDv5KapPBFpL14J8VEZKj3vAeb4tyBYOqodMb1SeGB+etZuuWA6zjGGODD9fv42/ubuGRQR34yvKvrOE3CX5dlvggsA7JEJF9EfioiU0RkiveQS4E1IvIV8DBwhYbR/EZEhPDQ5QPpktCC2//va8oqbT7fGJcKy6q447XV9OkQxx8u6Reyb9Iezy/vTqjqlSd5/VHgUX+cK1i1ioniT5f05/Inl/HPDzdz5/k9XUcyJmw9uGADh0oqePbH3yE2OtJ1nCZjd9o2oaHd2nF5dhozP81j496jruMYE5a+3HGYF5bv4LqzutG3Y7zrOE3KCr+J3Xl+L1rHRnH33NXU1ITNrJYxAcFTXcNdc9eQ0jqWW8eGxgqYDWGF38TatWzGXRN6kbP9MK/k7HQdx5iw8uzSbazfU8TvLupNqxC+3r4+VvgOXDokjaHd2vHHdzZwsNiWUjamKew+Usbf3t/EqJ7JjOvT3nUcJ6zwHRAR/nBxX0orPTwwf73rOMaEhd+9uZYaVe67qE/YXJVzPCt8R9KTW/Pzc3rw+spddm2+MY3s/XX7eG/dPqaNzgz5u2lPxArfoZtGpdO5XQvueWMNFR67Nt+YxlBa6eF3b64lM6UV15/dzXUcp6zwHYqNjuT+SX3IKyjhyU/yXMcxJiT944PN7DpSxh8u7kd0ZHhXXnj/9AFgZFYyF/RP5dGFuWw7UOI6jjEhZf2eImYt3soV3+lEdtd2ruM4Z4UfAH47sTcxkRH8Zp6tqGmMv9TUKHfNXU1882juGG93toMVfkBIjovl9vFZfLr5AG99vcd1HGNCwktf7OTLHUe4e0Iv2rZs5jpOQLDCDxBXDetC/7R47n9rHYVlVa7jGBPUCo5W8Kd31nNG93ZcMrij6zgBwwo/QERGCH+4uB+HSip4cMEG13GMCWoP/HsdZVXV/P574bMS5qmwwg8gfTvG86OzuvLC8h18ueOw6zjGBKUluQd4Y9Vu/ue7PUhPbuU6TkCxwg8wt43NIqV1LHfPXYOnusZ1HGOCSnlVNfe8sYYuCS244dx013ECjhV+gGkVE8XvLurNuj1FPLt0m+s4xgSV6Z9sYeuBEv53Ut+wWuf+VPlrx6unRWS/iKyp53URkYdFJFdEvhaRwf44b6ga16c9o3om87f3N7H7SJnrOMYEhbyCYh5fuIULB3TgnMwk13ECkr9G+M8C40/w+vlAhvdjMvCEn84bkkSE+y7qQ40qv3tzres4xgQ8VeWeN9YQEx3Bbyb2ch0nYPml8FV1EXDoBIdMAp7XWp8BbUQk1R/nDlWd2rVg2uhM3lu3j4837ncdx5iANn/1XpZuOcivxmWR3DrWdZyA1VRz+B2BY3f7yPc+919EZLKI5IhITkFBQZOEC1Q/HdGNzu1a8Od3N9ruWMbUo6q6hr++t5GslNb8cFgX13ECWlMVfl0XwtbZYKo6Q1WzVTU7KSm85+GaRUVw29hM1u8p4q2vd7uOY0xAeiVnJ1sPlHD7uCwiI+ya+xNpqsLPBzod83UaYA12Ci7s34Ge7Vvz0HubqPTYZZrGHKusspp/frCZIV3aMrpXsus4Aa+pCv9N4Frv1TpnAIWqaovGnIKICOGO8T3ZcaiUl20PXGP+w7NLt7H/aAV3jO9pd9SeAn9dlvkisAzIEpF8EfmpiEwRkSneQ+YDeUAu8BRwgz/OGy5GZiUxtGs7Hv5wM6WVHtdxjAkIhaVVPPFxLqN6JjO0my19fCr8sm27ql55ktcVuNEf5wpHIsId52fx/SeW8cySbdxodxAaw/RFWzha4eH2cVmuowQNu9M2SAzp0o4xvVKY/vEWDpdUuo5jjFP7isp5ZslWJg3oQK/UONdxgoYVfhC5fVwWxZUepn+yxXUUY5z654eb8VQrt55no/uGsMIPIlntW3PxoI48u3QbewptyQUTnrYeKOHlL3byw2Gd6ZzQwnWcoGKFH2R+MSaTGlUe/nCz6yjGOPHQexuJiYpg6qgM11GCjhV+kOnUrgVXDevCKzn5bCkodh3HmCa1Zlchb3+9h5+O6EZS6xjXcYKOFX4QumlUOrFRETz03kbXUYxpUn9ZsJE2LaL52TndXUcJSlb4QSixVQzXn92d+av38tXOI67jGNMklm45wKJNBdw4Mp242GjXcYKSFX6Quv7sbrRr2YwHF9go34Q+VeXP724kNT6Wa860BdJOlxV+kGodG82N56azOPcAizcfcB3HmEa1YO0+vtp5hF+MybSdrHxghR/ErhrWmY5tmvPndzdQezOzMaHH413+uEdSSy4ZXOeq6uYUWeEHsdjoSG4Zk8HqXYW8s2av6zjGNIrXv9xF7v5ibh+XRVSkVZYv7J9ekLtkcBoZya3464KNeKpt+WQTWsqrqvnH+5sY0KkN4/q0dx0n6FnhB7nICOH2cVnkHSjh1RX5ruMY41dzPtvO7sJy7hifZcsf+4EVfgg4r3cKgzq34R8fbKa8qtp1HGP8oqi8iscW5nJ2RiJn9Uh0HSckWOGHAJHaTVL2FpXz3NJtruMY4xczF+VxuLSKX43r6TpKyPDXBijjRWSjiOSKyJ11vH6diBSIyCrvx/X+OK/5/87onsB3M5N4/OMtFJZVuY5jjE8KjlYwc/FWLuifSr+0eNdxQobPhS8ikcBjwPlAb+BKEeldx6Evq+pA78dMX89r/tvt47IoLKtixiJbPtkEt8cW5lLhqeG28zJdRwkp/hjhDwVyVTVPVSuBl4BJfvi+poH6doznwgEdeHrxNg4UV7iOY8xp2XWkjBeWb+fy7E50T2rlOk5I8UfhdwSO3V073/vc8b4vIl+LyKsi0qm+byYik0UkR0RyCgoK/BAvvEwbnUG5p5qnPs1zHcWY0/L4wlygdpFA41/+KPy6rpU6/rbPt4Cuqtof+AB4rr5vpqozVDVbVbOTkpL8EC+8pCe34sL+HZi9bDuHbCtEE2R2HynjlZydXJbdiY5tmruOE3L8Ufj5wLEj9jRg97EHqOpBVf1mjuEpYIgfzmvqcfPodMqqbJRvgs8323feMLKH4yShyR+F/wWQISLdRKQZcAXw5rEHiEjqMV9eBKz3w3lNPdKTW3NBv1SeX7rNNjw3QWNvYTkvfb6TS4ekkdbWti5sDD4Xvqp6gJuABdQW+SuqulZE7heRi7yH3Swia0XkK+Bm4Dpfz2tO7ObRGZRWVTNzsY3yTXCY/skWalS5YaTN3TeWKH98E1WdD8w/7rl7j3n8a+DX/jiXOTWZKa2Z0DeV55Zu52dnd6dNi2auIxlTr31F5fzr8x18f3AandrZ6L6x2J22IWzq6HSKKzzMWrzVdRRjTmj6J1uorlFuPNdG943JCj+E9Wwfx/l92/Pskm0UltrdtyYw7S8q51/Ld3DJoI50TrDRfWOywg9xU0dlcLTCw6wlNso3genJRXl4atSuu28CVvghrneHOMb2TuGZJVttjR0TcAqOVvDC8u1MGtiBLgktXccJeVb4YeDm0RkcLffw7JJtrqMY8x9mLNpCpaeGqaMyXEcJC1b4YaBvx3jG9Eph1uI8isptlG8Cw4HiCmZ/tp1JAzvSLdFG903BCj9MTBudQVG5h+dslG8CxFOL8qj01NjcfROywg8T/dLiGd0zmZmLt3LURvnGsYPFFTy/bDsXDuhAD1sRs8lY4YeRaWMyKCyr4vll211HMWFu5uKtlHuqmWqj+yZlhR9G+qe14dysJJ76NI/iCo/rOCZMHS6p5Pml25jYvwPpya1dxwkrVvhhZtqYTI6UVvH8sm2uo5gwNXNxHqVV1dxso/smZ4UfZgZ2asN3M5N4alEeJTbKN03sSGklzy3dzoR+qWSk2Oi+qVnhh6FpYzI4XFrFnM9sLt80rVmLt1Jc4eFmu+7eCSv8MDS4c1vOzkhkxqI8SittlG+aRmFpFc8u2caEfu3Jam+jexes8MPUtNEZHCyp5IXPdriOYsLErCVbOVrhsbtqHbLCD1PZXdsxPD2BJxdtoayy2nUcE+IKy6p4ZslWxvVJoVdqnOs4YcsvhS8i40Vko4jkisiddbweIyIve19fLiJd/XFe45tpozM5UFzJC8ttLt80rmeWbOVouYebR9vo3iWfC19EIoHHgPOB3sCVItL7uMN+ChxW1XTg78CffT2v8d3Qbu04s3sCTy7Ko7zKRvmmcRSVV/H04q2c1zuFPh3iXccJa/4Y4Q8FclU1T1UrgZeASccdMwl4zvv4VWC0iIgfzm18NG1MBgVHK/jXcpvLN43j2SXbKCr3MM1G9875o/A7AjuP+Trf+1ydx3g3PS8EEur6ZiIyWURyRCSnoKDAD/HMiZzRPYFh3dox/ZMtNso3fne0vIpZi7cyplcyfTva6N41fxR+XSN1PY1jap9UnaGq2aqanZSU5HM4c3LTxmSw/2gFL31uo3zjX88t3UZhWRXTRme6jmLwT+HnA52O+ToN2F3fMSISBcQDh/xwbuMHZ3ZPYGjXdjxho3zjR8UVHmYu3sqonsn0S7PRfSDwR+F/AWSISDcRaQZcAbx53DFvAj/yPr4U+EhV6xzhm6YnIkwbk8G+ogpeydl58j9gzCl4ftk2jpRW2dx9APG58L1z8jcBC4D1wCuqulZE7heRi7yHzQISRCQXuBX4r0s3jVtn9Uggu0tbnvh4CxUeG+Ub35RUeHhqUR4js5IY0KmN6zjGyy/X4avqfFXNVNUeqvqA97l7VfVN7+NyVb1MVdNVdaiq5vnjvMZ/vhnl7yks5/9y8l3HMUFu9mfbOVxaZdfdBxi709Z8a0R6IoM6t+GJj2s3ljbmdJRW1o7uz85IZHDntq7jmGNY4ZtviQjTRmew60gZr66wUb45PXM+287BkkpuGWOj+0BjhW/+w3cza+dcH1uYa6N802BlldXMWJTHiPREhnRp5zqOOY4VvvkPIsIt3lH+6yttlG8a5oXl2zlQXMk0G90HJCt8819GZiXRPy2eRxfmUlVto3xzasoqq5n+SR5n9UjgO11tdB+IrPDNf/lmLj//cBlzV+5yHccEiX99voMDxRV23X0As8I3dRrVM5l+HW2Ub05NeVU10z/Zwhnd2zGse53LZJkAYIVv6iQi3Dw6gx2HSnnjSxvlmxN76fMdFBytsDVzApwVvqnXmF7J9OkQx6MLc/HYKN/Uo7yqmic+2VK7v0IPG90HMit8U69vRvnbD5Yyb9Xx6+EZU+uVnJ3sK6rgFpu7D3hW+OaExvau3YPURvmmLhWeap74eAvf6drWRvdBwArfnFDtFTvpbD1Qwltf2yjf/KdXcvLZU1jOtNGZ2CZ2gc8K35zU2N7t6dm+NY98lEt1ja1qbWpVeKp5YmEuQ7q0ZXi6je6DgRW+OamIiNq5/LyCEt62Ub7xenVFPrsLy5k2OsNG90HCCt+ckvF92pOZ0oqHP9xso3xDpaeGxxduYWCnNpydkeg6jjlFVvjmlERECFNHZbCloIR/r97jOo5x7LWV+ew6Usa0MTa6DyY+Fb6ItBOR90Vks/dznYtfi0i1iKzyfhy//aEJEhP6pZKe3IpHPtxMjY3yw1ZVdQ2PLcxlQFo8IzOTXMcxDeDrCP9O4ENVzQA+pP6tC8tUdaD346J6jjEBLjJCmDoqnc37i3lnzV7XcYwjc1fuIv+wje6Dka+FPwl4zvv4OeB7Pn4/E+Am9u9Aj6SWPGyj/LBUVV3Dowtz6Z8Wz7lZya7jmAbytfBTVHUPgPdzff8FxIpIjoh8JiIn/EtBRCZ7j80pKCjwMZ7xt0jvXP7GfUeZv8bm8sPN3JW72HGolJtH2eg+GJ208EXkAxFZU8fHpAacp7OqZgM/BP4hIj3qO1BVZ6hqtqpmJyXZ/GAgunBABzKSW/HQe5tsJc0wUl5Vzd8/2MSATm0Y3ctG98HopIWvqmNUtW8dH/OAfSKSCuD9vL+e77Hb+zkP+BgY5LefwDS5yAjh9nFZbD1QYnvfhpE5n21nT2E5d4zPstF9kPJ1SudN4Efexz8C5h1/gIi0FZEY7+NEYDiwzsfzGsfO653C4M5t+McHmyivqnYdxzSyovIqHluYy9kZiZzVw667D1a+Fv6fgPNEZDNwnvdrRCRbRGZ6j+kF5IjIV8BC4E+qaoUf5ESEO8b3ZF9RBc8t3eY6jmlkMxflcbi0ijvG93Qdxfggypc/rKoHgdF1PJ8DXO99vBTo58t5TGAa1j2BkVlJPP7xFq4Y2pn45tGuI5lGUHC0gpmLtzKxfyp9O8a7jmN8YHfaGp/cPi6LwrIqnvxki+soppE8+tFmKjw13DY2y3UU4yMrfOOTPh3iuWhAB55espX9ReWu4xg/23GwlH99voMffKcT3RJbuo5jfGSFb3x263mZeKqVhz/a7DqK8bO/f7CJyAhhmu1mFRKs8I3Puia25MqhnXnp851sO1DiOo7xk/V7inhj1S5+PLwbKXGxruMYP7DCN34xdVQ60ZER/O39Ta6jGD95cMFGWsdEMeWceu+TNEHGCt/4RXJcLD8Z0ZU3v9rN2t2FruMYH32+9RAfbdjP/4xMJ76FXX0VKqzwjd9MPqcH8c2j+cu7G11HMT5QVf7y7gZS4mK47qyuruMYP7LCN34T3zyaG8/twSebCli25aDrOOY0fbRhPznbDzNtdCbNm0W6jmP8yArf+NW1Z3alfVwsf1mwAVVbPjnYVNcof3l3I90SW3JZdprrOMbPrPCNX8VGR3LLmAy+3HGE99ftcx3HNNC8VbvYuO8ot43NJDrS6iHU2L9R43eXDkmje2JLHlyw0TY8DyIVnmr+9v4m+naMY0LfVNdxTCOwwjd+FxUZwS/HZbF5fzFzv9zlOo45RS8u30H+4TJ+Na4nERG2/HEossI3jeL8vu3pnxbP39+35ZODQXGFh0c+yuWsHgmcnWHLH4cqK3zTKL5ZPnnXkTJeWL7DdRxzErM+3crBkkp+Nb6nbW4SwqzwTaMZnp7IiPREHluYy9HyKtdxTD0OFlfw1Kd5jO/TnoGd2riOYxqRT4UvIpeJyFoRqRGR7BMcN15ENopIrojc6cs5TXC5fVwWh0oqmfnpVtdRTD0e/3gLpZUefjku03UU08h8HeGvAS4BFtV3gIhEAo8B5wO9gStFpLeP5zVBYkCnNkzo156Zn+ZxoLjCdRxznF1Hypi9bDuXDkkjPbm16zimkflU+Kq6XlVPdh/9UCBXVfNUtRJ4CZjky3lNcLltbBblnhoeW5jrOoo5zj/e3wQCt4yx0X04aIo5/I7AzmO+zvc+VycRmSwiOSKSU1BQ0OjhTOPrkdSKy7PTmPPZdnL3F7uOY7xW5xfy2sp8rj2jCx3aNHcdxzSBkxa+iHwgImvq+DjVUXpdb/nXezeOqs5Q1WxVzU5KSjrFU5hAd9vYLFo0i+LuuattyYUAUF2j3P3GahJaxXDzGNvcJFycdBNzVR3j4znygU7HfJ0G7Pbxe5ogk9gqhjvP78mvX1/Nayt3cekQW6fFpdnLtvF1fiGPXDmIuFhb/jhcNMWUzhdAhoh0E5FmwBXAm01wXhNgfpDdiSFd2vKH+es5XFLpOk7Y2ldUzl/f28TZGYlM7G9LKIQTXy/LvFhE8oEzgX+LyALv8x1EZD6AqnqAm4AFwHrgFVVd61tsE4wiIoQHLu5LYVkVf3png+s4Yev+t9ZRWV3D77/X126yCjO+XqUzV1XTVDVGVVNUdZz3+d2qOuGY4+araqaq9lDVB3wNbYJXz/ZxXD+iGy/n7OSLbYdcxwk7Czfu59+r9zD13HS6JLR0Hcc0MbvT1jS5aWMy6NimOXfPXU2lp8Z1nLBRVlnNvfPW0COpJZO/2911HOOAFb5pci2aRXH/pD5s2lfMzMV5ruOEjUc+2szOQ2U8cHE/YqJsJ6twZIVvnBjdK4VxfVJ4+MPN7DxU6jpOyNu07ygzFuXx/cFpnNE9wXUc44gVvnHmdxf1IVKEe+etsWvzG1FNjXLP3DW0io3irgk9XccxDlnhG2dS45vzi/MyWbixgHfW7HUdJ2S9uiKfz7cd4tfn9yShVYzrOMYhK3zj1HVndaV3ahz3vbXWllBuBIdKKvnDO+v5Tte2XDak08n/gAlpVvjGqajICP5wST/2H63gofc2uY4Tcv4wfz3F5R4euLifbVtorPCNewM7teHqYV14fuR30l8AAAyaSURBVNk2VucXuo4TMj7LO8irK/L52TndyUyxpY+NFb4JELePzyKhVQx3v7Ga6hp7A9dXlZ4a7nljDWltm3PzKFsczdSywjcBIS42mt9M7M3X+YXMXrbNdZygN2PRFnL3F/O/k/rSvJldc29qWeGbgHFh/1TOzkjkr+9tYl9Rues4QWv7wRIe+SiXCf3ac27PZNdxTACxwjcBQ0T430l9qayu4f631rmOE5RUld/MW0t0ZAT3TuzjOo4JMFb4JqB0TWzJ1HPT+ffqPSzcuN91nKDz9td7WLSpgNvGZtI+PtZ1HBNgrPBNwJn83e70SGrJvfPWUFZZ7TpO0Cgqr+L+t9fRr2M8157Z1XUcE4Cs8E3AiYmK5Pff68fOQ2X8/QO7Nv9U/XH+Bg4WV/DAxX2JtGvuTR2s8E1AOrNHAlcN68yMRXnMX73HdZyA90rOTl78fAc/O7s7/dPauI5jApSvO15dJiJrRaRGRLJPcNw2EVktIqtEJMeXc5rwce+FvRncuQ23vfIV6/cUuY4TsFbuOMw9c9cwIj2R28dluY5jApivI/w1wCXAolM49lxVHaiq9f7FYMyxYqIimX71EOKaRzF5do7tg1uH/UXlTJm9gpT4GB65chBRkfZLu6mfr1scrlfVjf4KY8zxkuNimX71EPYVVnDTiyvxVNsOWd+o8FTz8zkrOFruYcY12bRt2cx1JBPgmmo4oMB7IrJCRCaf6EARmSwiOSKSU1BQ0ETxTCAb1Lktv7+4L0tyD/JH2/wcqL3e/t431vLljiM8dPkAeqXGuY5kgkDUyQ4QkQ+A9nW8dLeqzjvF8wxX1d0ikgy8LyIbVLXOaSBVnQHMAMjOzrZFVQwAl2d3Yt3uImYt3kqfDnFcMjjNdSSn5ny2nZdzdnLTuelM6JfqOo4JEictfFUd4+tJVHW39/N+EZkLDOXU5v2N+dbdF/Ri496j3Pn6atKTW4Xt1SjL8w5y31vrGNUzmVvPy3QdxwSRRp/SEZGWItL6m8fAWGrf7DWmQaIjI3j0h4NIahXDz2evoOBohetITW7XkTJueGElnRNa8I8rBtoa96ZBfL0s82IRyQfOBP4tIgu8z3cQkfnew1KAxSLyFfA58G9VfdeX85rwldAqhhnXDuFwaSU3vLCCSk/4vIlbVlnN5OdzqPTU8NS12cTFRruOZIKMr1fpzFXVNFWNUdUUVR3nfX63qk7wPs5T1QHejz6q+oA/gpvw1adDPA9eOoAvth3mvrfWuo7TJFSVO1//mnV7ivjnlQPpkdTKdSQThE46h29MILpwQAfW7i5i+idb6NMhnh8O6+w6UqN66tM85q3azS/HZjKqZ4rrOCZI2V0aJmjdPi6L72Ym8ds315Cz7ZDrOI1m0aYC/vTOBib0a8+N56a7jmOCmBW+CVqREcLDVwyiY5vmTJmzkj2FZa4j+d32gyVMffFLMlNa8+ClAxCxN2nN6bPCN0EtvkU0T12bTVmlhymzV1BeFTrLKRdXePjZ8zmIwIxrsmkZYzOwxjdW+CboZaS05u8/GMhX+YXcPXcNqsF/v15NjXLbK6vI3V/Mo1cOpnNCC9eRTAiwwjchYWyf9twyJoPXVuZzzxtrqPAE70i/rLKaX776FQvW7uOuCb0YkZHoOpIJEfY7ogkZN4/KoKyqmic/yWPt7iIev2owHdo0dx2rQbYfLGHKnJVs2FvELWMy+OmIbq4jmRBiI3wTMiIihF+f34vpVw8md38xEx9ZzJLcA65jnbIP1+9j4iOL2X2kjKev+w63jMm0N2mNX1nhm5Azvm8q824aTkLLZlwzazmPLcylpiZw5/Wra5S/LtjIT5/LoUtCC96eOoJzs5JdxzIhyArfhKQeSa1448bhXNC/Aw8u2Mjk2SsoLKtyHeu/HCqp5LpnPufRhblcnp3Gq1POolM7e4PWNA4rfBOyWsZE8fAVA/nthb35eON+Jj26mA17A2erxK92HuHCRxazfOsh/nRJP/5y6QBioyNdxzIhzArfhDQR4cfDu/HS5DMorazme48t4Y0vdznNpKr8a/kOLpu+DIBXp5zJFUNDe2kIExis8E1YyO7ajrdvHkH/tDbc8vIq7p23xslKm+VV1dz+6tfcNXc1Z/RI4O2pI8J2XX/T9OyyTBM2klvH8sL1w/jLuxt46tOtrN5VyONXDSY1vmku3dxxsJQpc1awbk8RN4/OYNroDCJtPXvThGyEb8JKdGQEd1/Qm8evGsymvUeZ+PBiPtlU0Kh359bUKO+t3cvERz4l/3ApT1+Xza3nZVrZmyZnI3wTlib0SyUzpTVT5qzgR09/TmKrGEakJ3BWeiIj0hN9vmFr56FSluQeYHHuAZZuOcihkkp6p8Yx/eohtkyCccanwheRB4ELgUpgC/BjVT1Sx3HjgX8CkcBMVf2TL+c1xh/Sk1sx78bh/Hv1nm/L+Y1VuwHontiS4emJDE9P5MweCcQ3P/HuUodLKlmWd5DFuQdYknuA7QdLAUiJi2FkVhIj0hOZ0C/VrsIxTokvv8qKyFjgI1X1iMifAVT1juOOiQQ2AecB+cAXwJWquu5k3z87O1tzcnJOO58xDaGqbNx3lMWba0t7+dZDlFZWEyHQL60NI9ITGJ6eyJAubVGFnG2Hvy34NbsLUYVWMVGc0T2BEekJjMhIpEdSK7tb1jQpEVmhqtl1vuavuUsRuRi4VFWvOu75M4HffbP9oYj8GkBV/3iy72mFb1yq9NSwaueRb0t91c4jVNcosdER1Gjt69GRwqDObRnh/W1gQFo8UZH21phx50SF7885/J8AL9fxfEdg5zFf5wPD6vsmIjIZmAzQubNdm2zcaRYVwdBu7RjarR23npfJ0fIqPt96iCW5B4kQGJ6RyNCu7WydehM0Tvpfqoh8ALSv46W7VXWe95i7AQ/wQl3foo7n6v21QlVnADOgdoR/snzGNJXWsdGM7pXC6F62p6wJTictfFUdc6LXReRHwERgtNY9P5QPdDrm6zRgd0NCGmOM8Z1Pk43eq2/uAC5S1dJ6DvsCyBCRbiLSDLgCeNOX8xpjjGk4X99dehRoDbwvIqtEZDqAiHQQkfkAquoBbgIWAOuBV1R1rY/nNcYY00A+vdukqun1PL8bmHDM1/OB+b6cyxhjjG/s+jFjjAkTVvjGGBMmrPCNMSZMWOEbY0yY8NvSCo1BRAqA7af5xxOBA36M09SCPT8E/88Q7Pkh+H8Gy99wXVQ1qa4XArrwfSEiOfWtJxEMgj0/BP/PEOz5Ifh/BsvvXzalY4wxYcIK3xhjwkQoF/4M1wF8FOz5Ifh/hmDPD8H/M1h+PwrZOXxjjDH/KZRH+MYYY45hhW+MMWEi5ApfRMaLyEYRyRWRO13naSgReVpE9ovIGtdZToeIdBKRhSKyXkTWisg015kaSkRiReRzEfnK+zPc5zrT6RCRSBH5UkTedp3ldIjINhFZ7V2JN+j2OhWRNiLyqohs8P7/cKbzTKE0h+/LhumBQkTOAYqB51W1r+s8DSUiqUCqqq4UkdbACuB7QfbvQICWqlosItHAYmCaqn7mOFqDiMitQDYQp6oTXedpKBHZBmSralDeeCUizwGfqupM714gLVT1iMtMoTbCHwrkqmqeqlYCLwGTHGdqEFVdBBxyneN0qeoeVV3pfXyU2j0QOrpN1TBaq9j7ZbT3I6hGRiKSBlwAzHSdJRyJSBxwDjALQFUrXZc9hF7h17VhelCVTSgRka7AIGC52yQN550OWQXsB95X1WD7Gf4B/AqocR3EBwq8JyIrRGSy6zAN1B0oAJ7xTqvNFJGWrkOFWuE3aMN003hEpBXwGnCLqha5ztNQqlqtqgOp3YN5qIgEzfSaiEwE9qvqCtdZfDRcVQcD5wM3eqc7g0UUMBh4QlUHASWA8/cUQ63wbcP0AOCd934NeEFVX3edxxfeX8M/BsY7jtIQw4GLvHPgLwGjRGSO20gN5905D1XdD8yldso2WOQD+cf8ZvgqtX8BOBVqhW8bpjvmfcNzFrBeVf/mOs/pEJEkEWnjfdwcGANscJvq1Knqr1U1TVW7Uvv/wEeqerXjWA0iIi29b/rjnQoZCwTNlWuquhfYKSJZ3qdGA84vXPBpT9tAo6oeEflmw/RI4Olg2zBdRF4ERgKJIpIP/FZVZ7lN1SDDgWuA1d45cIC7vPsaB4tU4DnvVV8RwCuqGpSXNgaxFGBu7fiBKOBfqvqu20gNNhV4wTv4zAN+7DhPaF2WaYwxpn6hNqVjjDGmHlb4xhgTJqzwjTEmTFjhG2NMmLDCN8aYMGGFb4wxYcIK3xhjwsT/A4h8Ue4ZlLYFAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "out.backward()\n",
    "print(a.grad)\n",
    "plt.plot(a.detach(), a.grad.detach())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall the computation steps we took to get here:\n",
    "\n",
    "```\n",
    "a = torch.linspace(0., 2. * math.pi, steps=25, requires_grad=True)\n",
    "b = torch.sin(a)\n",
    "c = 2 * b\n",
    "d = c + 1\n",
    "out = d.sum()\n",
    "```\n",
    "\n",
    "Adding a constant, as we did to compute `d`, does not change the derivative. That leaves $c = 2 * b = 2 * sin(a)$, the derivative of which should be $2 * cos(a)$. Looking at the graph above, that's just what we see.\n",
    "\n",
    "Be aware than only *leaf nodes* of the computation have their gradients computed. If you tried, for example, `print(c.grad)` you'd get back `None`. In this simple example, only the input is a leaf node, so only it has gradients computed.\n",
    "\n",
    "## Autograd in Training\n",
    "\n",
    "We've had a brief look at how autograd works, but how does it look when it's used for its intended purpose? Let's define a small model and examine how it changes after a single training batch. First, define a few constants, our model, and some stand-ins for inputs and outputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "BATCH_SIZE = 16\n",
    "DIM_IN = 1000\n",
    "HIDDEN_SIZE = 100\n",
    "DIM_OUT = 10\n",
    "\n",
    "class TinyModel(torch.nn.Module):\n",
    "\n",
    "    def __init__(self):\n",
    "        super(TinyModel, self).__init__()\n",
    "        \n",
    "        self.layer1 = torch.nn.Linear(1000, 100)\n",
    "        self.relu = torch.nn.ReLU()\n",
    "        self.layer2 = torch.nn.Linear(100, 10)\n",
    "    \n",
    "    def forward(self, x):\n",
    "        x = self.layer1(x)\n",
    "        x = self.relu(x)\n",
    "        x = self.layer2(x)\n",
    "        return x\n",
    "    \n",
    "some_input = torch.randn(BATCH_SIZE, DIM_IN, requires_grad=False)\n",
    "ideal_output = torch.randn(BATCH_SIZE, DIM_OUT, requires_grad=False)\n",
    "\n",
    "model = TinyModel()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One thing you might notice is that we never specify `requires_grad=True` for the model's layers. Within a subclass of `torch.nn.module`, it's assumed that we want to track gradients on the layers' weights for learning.\n",
    "\n",
    "If we look at the layers of the model, we can examine the values of the weights, and verify that no gradients have been computed yet:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([ 0.0244,  0.0843,  0.0378, -0.0212, -0.0114,  0.0351, -0.0460,  0.0561,\n",
      "         0.0040,  0.0179], grad_fn=<SliceBackward>)\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "print(model.layer2.weight[0][0:10]) # just a small slice\n",
    "print(model.layer2.weight.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how this changes when we run through one training batch. For a loss function, we'll just use the square of the Euclidean distance between our `prediction` and the `ideal_output`, and we'll use a basic stochastic gradient descent optimizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(157.9841, grad_fn=<SumBackward0>)\n"
     ]
    }
   ],
   "source": [
    "optimizer = torch.optim.SGD(model.parameters(), lr=0.001)\n",
    "\n",
    "prediction = model(some_input)\n",
    "\n",
    "loss = (ideal_output - prediction).pow(2).sum()\n",
    "print(loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's call `loss.backward()` and see what happens:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([ 0.0244,  0.0843,  0.0378, -0.0212, -0.0114,  0.0351, -0.0460,  0.0561,\n",
      "         0.0040,  0.0179], grad_fn=<SliceBackward>)\n",
      "tensor([ 2.8054,  1.4428,  4.5022,  4.4852, 10.5636,  8.2099,  7.4645,  1.9516,\n",
      "         1.9949,  3.5033])\n"
     ]
    }
   ],
   "source": [
    "loss.backward()\n",
    "print(model.layer2.weight[0][0:10])\n",
    "print(model.layer2.weight.grad[0][0:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the gradients have been computed for each learning weight, but the weights remain unchanged, because we haven't run the optimizer yet. The optimizer is responsible for updating model weights based on the computed gradients."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([ 0.0216,  0.0829,  0.0332, -0.0257, -0.0220,  0.0269, -0.0535,  0.0541,\n",
      "         0.0020,  0.0144], grad_fn=<SliceBackward>)\n",
      "tensor([ 2.8054,  1.4428,  4.5022,  4.4852, 10.5636,  8.2099,  7.4645,  1.9516,\n",
      "         1.9949,  3.5033])\n"
     ]
    }
   ],
   "source": [
    "optimizer.step()\n",
    "print(model.layer2.weight[0][0:10])\n",
    "print(model.layer2.weight.grad[0][0:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should see that `layer2`'s weights have changed.\n",
    "\n",
    "One important thing about the process: After calling `optimizer.step()`, you need to call `optimizer.zero_grad()`, or else every time you run `loss.backward()`, the gradients on the learning weights will accumulate:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([ 2.8054,  1.4428,  4.5022,  4.4852, 10.5636,  8.2099,  7.4645,  1.9516,\n",
      "         1.9949,  3.5033])\n",
      "tensor([ 4.6037, -1.8794,  6.1310, 20.7784, 36.1072, 11.4850, 43.7955,  1.5995,\n",
      "         3.7064, -0.5014])\n",
      "tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])\n"
     ]
    }
   ],
   "source": [
    "print(model.layer2.weight.grad[0][0:10])\n",
    "\n",
    "for i in range(0, 5):\n",
    "    prediction = model(some_input)\n",
    "    loss = (ideal_output - prediction).pow(2).sum()\n",
    "    loss.backward()\n",
    "    \n",
    "print(model.layer2.weight.grad[0][0:10])\n",
    "\n",
    "optimizer.zero_grad()\n",
    "\n",
    "print(model.layer2.weight.grad[0][0:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After running the cell above, you should see that after running `loss.backward()` multiple times, the magnitudes of most of the gradients will be much larger. Failing to zero the gradients before running your next training batch will cause the gradients to blow up in this manner, causing incorrect and unpredictable learning results.\n",
    "\n",
    "## Turning Autograd Off and On\n",
    "\n",
    "There are situations where you will need fine-grained control over whether autograd is enabled. There are multiple ways to do this, depending on the situation.\n",
    "\n",
    "The simplest is to change the `requires_grad` flag on a tensor directly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1., 1., 1.],\n",
      "        [1., 1., 1.]], requires_grad=True)\n",
      "tensor([[2., 2., 2.],\n",
      "        [2., 2., 2.]], grad_fn=<MulBackward0>)\n",
      "tensor([[2., 2., 2.],\n",
      "        [2., 2., 2.]])\n"
     ]
    }
   ],
   "source": [
    "a = torch.ones(2, 3, requires_grad=True)\n",
    "print(a)\n",
    "\n",
    "b1 = 2 * a\n",
    "print(b1)\n",
    "\n",
    "a.requires_grad = False\n",
    "b2 = 2 * a\n",
    "print(b2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the cell above, we see that `b1` has a `grad_fn` (i.e., a traced computation history), which is what we expect, since it was derived from a tensor, `a`, that had autograd turned on. When we turn off autograd explicitly with `a.requires_grad = False`, computation history is no longer tracked, as we see when we compute `b2`.\n",
    "\n",
    "If you only need autograd turned off temporarily, a better way is to use the `torch.no_grad()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[5., 5., 5.],\n",
      "        [5., 5., 5.]], grad_fn=<AddBackward0>)\n",
      "tensor([[5., 5., 5.],\n",
      "        [5., 5., 5.]])\n",
      "tensor([[6., 6., 6.],\n",
      "        [6., 6., 6.]], grad_fn=<MulBackward0>)\n"
     ]
    }
   ],
   "source": [
    "a = torch.ones(2, 3, requires_grad=True) * 2\n",
    "b = torch.ones(2, 3, requires_grad=True) * 3\n",
    "\n",
    "c1 = a + b\n",
    "print(c1)\n",
    "\n",
    "with torch.no_grad():\n",
    "    c2 = a + b\n",
    "\n",
    "print(c2)\n",
    "\n",
    "c3 = a * b\n",
    "print(c3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`torch.no_grad()` can also be used as a function or method dectorator:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[5., 5., 5.],\n",
      "        [5., 5., 5.]], grad_fn=<AddBackward0>)\n",
      "tensor([[5., 5., 5.],\n",
      "        [5., 5., 5.]])\n"
     ]
    }
   ],
   "source": [
    "def add_tensors1(x, y):\n",
    "    return x + y\n",
    "\n",
    "@torch.no_grad()\n",
    "def add_tensors2(x, y):\n",
    "    return x + y\n",
    "\n",
    "\n",
    "a = torch.ones(2, 3, requires_grad=True) * 2\n",
    "b = torch.ones(2, 3, requires_grad=True) * 3\n",
    "\n",
    "c1 = add_tensors1(a, b)\n",
    "print(c1)\n",
    "\n",
    "c2 = add_tensors2(a, b)\n",
    "print(c2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There's a corresponding context manager, `torch.enable_grad()`, for turning autograd on when it isn't already. It may also be used as a decorator.\n",
    "\n",
    "Finally, you may have a tensor that requires gradient tracking, but you want a copy that does not. For this we have the `Tensor` object's `detach()` method - it creates a copy of the tensor that is *detached* from the computation history:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([0.5443, 0.0012, 0.8133, 0.8857, 0.7629], requires_grad=True)\n",
      "tensor([0.5443, 0.0012, 0.8133, 0.8857, 0.7629])\n"
     ]
    }
   ],
   "source": [
    "x = torch.rand(5, requires_grad=True)\n",
    "y = x.detach()\n",
    "\n",
    "print(x)\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We did this above when we wanted to graph some of our tensors. This is because `matplotlib` expects a NumPy array as input, and the implicit conversion from a PyTorch tensor to a NumPy array is not enabled for tensors with requires_grad=True. Making a detached copy lets us move forward.\n",
    "\n",
    "### Autograd and In-place Operations\n",
    "\n",
    "In every example in this notebook so far, we've used variables to capture the intermediate values of a computation. Autograd needs these intermediate values to perform gradient computations. *For this reason, you must be careful about using in-place operations when using autograd.* Doing so can destroy information you need to compute derivatives in the `backward()` call. PyTorch will even stop you if you attempt an in-place operation on leaf variable that requires autograd, as shown below.\n",
    "\n",
    "*Note: The following code cell throws a runtime error. This is expected.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "ename": "RuntimeError",
     "evalue": "a leaf Variable that requires grad is being used in an in-place operation.",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-20-cffbbcd7d649>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0ma\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlinspace\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m0.\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m2.\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0mmath\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpi\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msteps\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m25\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrequires_grad\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msin_\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mRuntimeError\u001b[0m: a leaf Variable that requires grad is being used in an in-place operation."
     ]
    }
   ],
   "source": [
    "a = torch.linspace(0., 2. * math.pi, steps=25, requires_grad=True)\n",
    "torch.sin_(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Autograd Profiler\n",
    "\n",
    "Autograd tracks every step of your computation in detail. Such a computation history, combined with timing information, would make a handy profiler - and autograd has that feature baked in. Here's a quick example usage:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  \n",
      "Name     Self CPU total %  Self CPU total   CPU total %      CPU total        CPU time avg     Number of Calls  \n",
      "-------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  \n",
      "mul      51.18%           7.985ms          51.18%           7.985ms          7.985us          1000             \n",
      "div      48.82%           7.616ms          48.82%           7.616ms          7.616us          1000             \n",
      "-------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  \n",
      "Self CPU time total: 15.601ms\n",
      "\n"
     ]
    }
   ],
   "source": [
    "device = torch.device('cpu')\n",
    "run_on_gpu = False\n",
    "if torch.cuda.is_available():\n",
    "    device = torch.device('cuda')\n",
    "    run_on_gpu = True\n",
    "    \n",
    "x = torch.randn(2, 3, requires_grad=True)\n",
    "y = torch.rand(2, 3, requires_grad=True)\n",
    "z = torch.ones(2, 3, requires_grad=True)\n",
    "\n",
    "with torch.autograd.profiler.profile(use_cuda=run_on_gpu) as prf:\n",
    "    for _ in range(1000):\n",
    "        z = (z / x) * y\n",
    "        \n",
    "print(prf.key_averages().table(sort_by='self_cpu_time_total'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The profiler can also label individual sub-blocks of code, break out the data by input tensor shape, and export data as a Chrome tracing tools file. For full details of the API, see the [documentation](https://pytorch.org/docs/stable/autograd.html#profiler)\n",
    "\n",
    "## Advanced Topic: More Autograd Detail and the High-Level API\n",
    "\n",
    "If you have a function with an n-dimensional input and m-dimensional output, $\\vec{y}=f(\\vec{x})$, the complete gradient is a matrix of the derivative of every output with respect to every input, called the *Jacobian:*\n",
    "\n",
    "\\begin{align}J=\\left(\\begin{array}{ccc}\n",
    "   \\frac{\\partial y_{1}}{\\partial x_{1}} & \\cdots & \\frac{\\partial y_{1}}{\\partial x_{n}}\\\\\n",
    "   \\vdots & \\ddots & \\vdots\\\\\n",
    "   \\frac{\\partial y_{m}}{\\partial x_{1}} & \\cdots & \\frac{\\partial y_{m}}{\\partial x_{n}}\n",
    "   \\end{array}\\right)\\end{align}\n",
    "\n",
    "If you have a second function, $l=g\\left(\\vec{y}\\right)$ that takes m-dimensional input (that is, the same dimensionality as the output above), and returns a scalar output, you can express its gradients with respect to $\\vec{y}$ as a column vector, $v=\\left(\\begin{array}{ccc}\\frac{\\partial l}{\\partial y_{1}} & \\cdots & \\frac{\\partial l}{\\partial y_{m}}\\end{array}\\right)^{T}$ - which is really just a one-column Jacobian.\n",
    "\n",
    "More concretely, imagine the first function as your PyTorch model (with potentially many inputs and many outputs) and the second function as a loss function (with the model's output as input, and the loss value as the scalar output).\n",
    "\n",
    "If we multiply the first function's Jacobian by the gradient of the second function, and apply the chain rule, we get:\n",
    "\n",
    "\\begin{align}J^{T}\\cdot v=\\left(\\begin{array}{ccc}\n",
    "   \\frac{\\partial y_{1}}{\\partial x_{1}} & \\cdots & \\frac{\\partial y_{m}}{\\partial x_{1}}\\\\\n",
    "   \\vdots & \\ddots & \\vdots\\\\\n",
    "   \\frac{\\partial y_{1}}{\\partial x_{n}} & \\cdots & \\frac{\\partial y_{m}}{\\partial x_{n}}\n",
    "   \\end{array}\\right)\\left(\\begin{array}{c}\n",
    "   \\frac{\\partial l}{\\partial y_{1}}\\\\\n",
    "   \\vdots\\\\\n",
    "   \\frac{\\partial l}{\\partial y_{m}}\n",
    "   \\end{array}\\right)=\\left(\\begin{array}{c}\n",
    "   \\frac{\\partial l}{\\partial x_{1}}\\\\\n",
    "   \\vdots\\\\\n",
    "   \\frac{\\partial l}{\\partial x_{n}}\n",
    "   \\end{array}\\right)\\end{align}\n",
    "\n",
    "*Note: You could also use the equivalent operation $v^{T}\\cdot J$, and get back a row vector.*\n",
    "\n",
    "The resulting column vector is the *gradient of the second function with respect to the inputs of the first* - or in the case of our model and loss function, the gradient of the loss with respect to the model inputs.\n",
    "\n",
    "**`torch.autograd` is an engine for computing these products.** This is how we accumulate the gradients over the learning weights during the backward pass.\n",
    "\n",
    "For this reason, the `backward()` call can *also* take an optional vector input. This vector represents a set of gradients over the tensor, which are multiplied by the Jacobian of the autograd-traced tensor that precedes it. Let's try a specific example with a small vector:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([-663.2701, 1493.5994,  789.5186], grad_fn=<MulBackward0>)\n"
     ]
    }
   ],
   "source": [
    "x = torch.randn(3, requires_grad=True)\n",
    "\n",
    "y = x * 2\n",
    "while y.data.norm() < 1000:\n",
    "    y = y * 2\n",
    "\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we tried to call `y.backward()` now, we'd get a runtime error and a message that gradients can only be *implicitly* computed for scalar outputs. For a multi-dimensional output, autograd expects us to provide gradients for those three outputs that it can multiply into the Jacobian:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([1.0240e+02, 1.0240e+03, 1.0240e-01])\n"
     ]
    }
   ],
   "source": [
    "v = torch.tensor([0.1, 1.0, 0.0001], dtype=torch.float) # stand-in for gradients\n",
    "y.backward(v)\n",
    "\n",
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(Note that the output gradients are all related to powers of two - which we'd expect from a repeated doubling operation.)\n",
    "\n",
    "### The High-Level API\n",
    "\n",
    "There is an API on autograd that gives you direct access to important differential matrix and vector operations. In particular, it allows you to calculate the Jacobian and the *Hessian* matrices of a particular function for particular inputs. (The Hessian is like the Jacobian, but expresses all partial *second* derivatives.) It also provides methods for taking vector products with these matrices.\n",
    "\n",
    "Let's take the Jacobian of a simple function, evaluated for a 2 single-element inputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(tensor([0.7327]), tensor([0.8072]))\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(tensor([[4.1616]]), tensor([[3.]]))"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def exp_adder(x, y):\n",
    "    return 2 * x.exp() + 3 * y\n",
    "\n",
    "inputs = (torch.rand(1), torch.rand(1)) # arguments for the function\n",
    "print(inputs)\n",
    "torch.autograd.functional.jacobian(exp_adder, inputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you look closely, the first output should equal $2e^x$ (since the derivative of $e^x$ is $e^x$), and the second value should be 3.\n",
    "\n",
    "You can, of course, do this with higher-order tensors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(tensor([0.3723, 0.7282, 0.0756]), tensor([0.9602, 0.6986, 0.2260]))\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(tensor([[2.9022, 0.0000, 0.0000],\n",
       "         [0.0000, 4.1428, 0.0000],\n",
       "         [0.0000, 0.0000, 2.1570]]),\n",
       " tensor([[3., 0., 0.],\n",
       "         [0., 3., 0.],\n",
       "         [0., 0., 3.]]))"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inputs = (torch.rand(3), torch.rand(3)) # arguments for the function\n",
    "print(inputs)\n",
    "torch.autograd.functional.jacobian(exp_adder, inputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `torch.autograd.functional.hessian()` method works identically (assuming your function is twice differentiable), but returns a matrix of all second derivatives.\n",
    "\n",
    "There is also a function to directly compute the vector-Jacobian product, if you provide the vector:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([ -52.9634, -894.0098, 1167.3909]),\n",
       " tensor([5.1200e+01, 5.1200e+02, 5.1200e-02]))"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def do_some_doubling(x):\n",
    "    y = x * 2\n",
    "    while y.data.norm() < 1000:\n",
    "        y = y * 2\n",
    "    return y\n",
    "\n",
    "inputs = torch.randn(3)\n",
    "my_gradients = torch.tensor([0.1, 1.0, 0.0001])\n",
    "torch.autograd.functional.vjp(do_some_doubling, inputs, v=my_gradients)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `torch.autograd.functional.jvp()` method performs the same matrix multiplication as `vjp()` with the operands reversed. The `vhp()` and `hvp()` methods do the same for a vector-Hessian product.\n",
    "\n",
    "For more information, including preformance notes on the [docs for the functional API](https://pytorch.org/docs/stable/autograd.html#functional-higher-level-api)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Advanced Topic: Higher-Order Derivatives\n",
    "+ computing your own chain-rule derivatives with the high-level API"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}