Azri_04_ReverseAD_question_2021_22.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.7"
    },
    "colab": {
      "name": "Azri_04_ReverseAD_question_2021_22.ipynb",
      "provenance": []
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "checksum": "ef88843b66adc91569384659597bfa0d",
          "grade": false,
          "grade_id": "cell-3f061ea27f528fcf",
          "locked": true,
          "schema_version": 1,
          "solution": false
        },
        "id": "5eJRjRqu-c_A"
      },
      "source": [
        "#Reverse Mode Automatic Differentiation (AD)\n",
        "\n",
        "Dynamic Reverse mode AD can be implemented by declaring a class `Var` to represent a value and the child expressions that the value depends on. We've provided the implementation that was shown in the lecture slides. \n",
        "\n",
        "Tasks:\n",
        "\n",
        "1. Complete Addition (`__add__`) method below. \n",
        "2. Complete division (`__truediv__`), subtraction (`__sub__`) and power (`__pow__`)?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deletable": false,
        "nbgrader": {
          "checksum": "2644b153f82b6e872e63cb5ead2d529f",
          "grade": false,
          "grade_id": "cell-b23d219550cd2934",
          "locked": false,
          "schema_version": 1,
          "solution": true
        },
        "id": "Fh_7z_CJ-c_Q"
      },
      "source": [
        "import math\n",
        "\n",
        "class Var:\n",
        "    def __init__(self, value):\n",
        "        self.value = value\n",
        "        self.children = []\n",
        "        self.grad_value = None #Initialize to None, which means it's not yet evaluated\n",
        "\n",
        "    def grad(self):\n",
        "        #recurse only if the value is not yet cached\n",
        "        if self.grad_value is None:\n",
        "            #calculate derivative using chain rule\n",
        "            self.grad_value = sum(weight * var.grad()\n",
        "                                  for weight, var in self.children)\n",
        "        return self.grad_value\n",
        "    \n",
        "    def __str__(self):\n",
        "        return str(self.value)\n",
        "\n",
        "    def __mul__(self, other):\n",
        "\n",
        "      z = Var(self.value * other.value)\n",
        "    \n",
        "      #weight = dz/dself=other.value\n",
        "      self.children.append((other.value, z))\n",
        "    \n",
        "      #weight = dz/dother=self.value\n",
        "      other.children.append((self.value, z))\n",
        "      return z\n",
        "\n",
        "# For a=x*y a is a new Var that is a child of both x and y\n",
        "\n",
        "    def __add__(self, other):\n",
        "      #your code here\n",
        "      z = Var(self.value + other.value)\n",
        "      \n",
        "      #weight = dz/dself=1\n",
        "      self.children.append((1, z))\n",
        "    \n",
        "      #weight = dz/dother=1\n",
        "      other.children.append((1, z))\n",
        "      return z\n",
        "\n",
        "    #Add other functions as well\n",
        "\n",
        "    def __sub__(self, other):\n",
        "      #your code here\n",
        "      z = Var(self.value - other.value)\n",
        "      \n",
        "      #weight = dz/dself=other.value\n",
        "      self.children.append((1, z))\n",
        "    \n",
        "      #weight = dz/dother=self.value\n",
        "      other.children.append((-1, z))\n",
        "      return z\n",
        "\n",
        "    def __truediv__(self, other):\n",
        "      #your code here\n",
        "      z = Var(self.value / other.value)\n",
        "      \n",
        "      #weight = dz/dself=other.value\n",
        "      self.children.append((1/other.value, z))\n",
        "\n",
        "      #weight = dz/dother=self.value\n",
        "      other.children.append(((-self.value/other.value**2), z))\n",
        "      return z\n",
        "\n",
        "\n",
        "def sin(x):\n",
        "    z = Var(math.sin(x.value))\n",
        "    x.children.append((math.cos(x.value), z))\n",
        "    return z"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "checksum": "afce3f80a9f8278e0ea2815a23a09c95",
          "grade": true,
          "grade_id": "cell-d2124fb6ca76110f",
          "locked": true,
          "points": 2,
          "schema_version": 1,
          "solution": false
        },
        "id": "F6CNuLk0-c_S",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "859d20f7-8f85-4d5a-d403-653da1b8318e"
      },
      "source": [
        "# Tests\n",
        "\n",
        "print(Var(1) + Var(1))\n",
        "print(Var(1) + Var(1) / Var(1) - Var(1))\n",
        "# print(Var(1) + Var(1) / Var(1) - Var(1)**Var(1))\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "2\n",
            "1.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qwUhQNOFiXvJ"
      },
      "source": [
        "#Forward computation first\n",
        "1. Try running the following code to compute the value of the function  $z=a+b, a=x*y, b =sin(x)$  given  $x=0.5$  and  $y=4.2$.\n",
        "2. Print out the children of x,y,a,b with their derivatives $\\frac{\\partial a}{\\partial x},\\frac{\\partial b}{\\partial x},\\frac{\\partial a}{\\partial y},\\frac{\\partial z}{\\partial a},\\frac{\\partial z}{\\partial b}$\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "C18AKo51i3Ve",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "20cd9d8d-0c34-4473-8fcd-122bfe9d66eb"
      },
      "source": [
        "x=Var(0.5)\n",
        "y=Var(4.2)\n",
        "\n",
        "a=x*y\n",
        "b=sin(x)\n",
        "z=a+b\n",
        "z.grad_value=1 \n",
        "\n",
        "#Complete the code here\n",
        "print(f'Children of x is {x.children}...and its derivative is ')\n",
        "print(f'Children of y is {y.children}...and its derivative is ')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Children of x is [(4.2, <__main__.Var object at 0x7facae6af290>), (0.8775825618903728, <__main__.Var object at 0x7facae6af550>)]...and its derivative is \n",
            "Children of y is [(0.5, <__main__.Var object at 0x7facae6af290>)]...and its derivative is \n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NpnFmDzMk134"
      },
      "source": [
        "#Reverse mode computation\n",
        "\n",
        "So far we have done forward computing as we go. But we haven't computed $\\frac{\\partial z}{\\partial x}$ and $\\frac{\\partial z}{\\partial y}$ which is what we want essentially.\n",
        "\n",
        "1. Run the code below\n",
        "2. Print out the gradient of each variable and complete the code"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Md1CQGt6k5N3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e40e9b2c-b1fb-4f7e-ebd5-67e5ab8880f2"
      },
      "source": [
        "z.grad_value=1  \n",
        "z.grad_value = 1.0 #Note that we have to 'seed' the gradient of z to 1 (e.g. ∂z/∂z=1) before computing grads\n",
        "\n",
        "print('z:', z)\n",
        "print(\"dz/dx: \",x.grad())\n",
        "\n",
        "#Complete the code here\n",
        "print(\"dz/dy: \",y.grad())\n",
        "\n",
        "\n",
        "\n",
        "#Test your functions\n",
        "assert abs(z.value - 2.579425538604203) <= 1e-15\n",
        "assert abs(x.grad() - (y.value + math.cos(x.value))) <= 1e-15\n",
        "assert abs(y.grad() - x.value) <= 1e-15"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "z: 2.579425538604203\n",
            "dz/dx:  5.077582561890373\n",
            "dz/dy:  0.5\n"
          ]
        }
      ]
    }
  ]
}