book/reinforce/policy/policy-gradient.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Policy Gradient"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Where can policy gradients be applied?\n",
    "\n",
    "Policy gradient is specifically used to update policy networks that outputs probability for each actions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{warning}\n",
    "Incoming math!\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The policy gradient in simple terms.\n",
    "\n",
    "First, we have a policy network $ \\pi $, the current state $ s $, reward for each action $ r_i $, and probability for each actions $ \\pi(s)_i $.\n",
    "\n",
    "The expected future reward (value function) of this state is thus\n",
    "\n",
    "$$\n",
    "G = \\sum_i \\pi(s)_i r_i\n",
    "$$\n",
    "\n",
    "To maximize the value function of this state, we wish to find the gradient of the expected future reward $ \\nabla G $, which is \n",
    "\n",
    "$$\n",
    "\\nabla \\sum_i \\pi(s)_i r_i\n",
    "$$\n",
    "\n",
    "Because rewards are scalars and determined by the environment,\n",
    "\n",
    "$$\n",
    "\\nabla G = \\sum_i r_i \\nabla \\pi(s)_i\n",
    "$$\n",
    "\n",
    "Since $ \\pi(s)_i $ is the probability of each action (which usually is non-zero), we divide and multiply by it in the previous equation\n",
    "\n",
    "$$\n",
    "\\nabla G = \\sum_i r_i \\frac{ \\nabla \\pi(s)_i }{ \\pi(s)_i } \\pi(s)_i\n",
    "$$\n",
    "\n",
    "We notice that this looks like the formula for expectation! So the equation is reduced to:\n",
    "\n",
    "$$\n",
    "\\nabla G = E_{\\sim\\pi(s)}[ r_i \\frac{ \\nabla \\pi(s)_i }{ \\pi(s)_i } ]\n",
    "$$\n",
    "\n",
    "Which is equivalent to\n",
    "\n",
    "$$\n",
    "\\nabla G = E_{\\sim\\pi(s)}[ r_i \\nabla \\log \\pi(s)_i ]\n",
    "$$\n",
    "\n",
    "Voila! This is the reason you see log-probability quite often in policy gradients."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}