book/notice/lr/lr.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Learning Rate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{note}\n",
    "We will abbreviate learning rate as LR.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{note}\n",
    "Learning rate is also called step size.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What are LR?\n",
    "\n",
    "In vanilla gradient descent, with the LR $ \\eta $, parameter $ x $, loss function $ f $, the update formula is as follows:\n",
    "\n",
    "$$\n",
    "x' = x - \\eta * \\frac{df}{dx}\n",
    "$$\n",
    "\n",
    "Other optimizers are updated in a similar fashion. The takeaway is, LR is how much you step forward performing a gradient update. If LR is huge, you update the parameters more. If LR is tiny, you don't update the parameter as much."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How should I choose my LR?\n",
    "\n",
    "Initially, we want the LR to be as big as possible (because updates are faster) whenever possible. However, with big LRs, it's difficult to move into a precise location because it only takes big steps. With training progresses, we want to slightly reduce the LR such that we find a more finetuned solution. Some optimizers sort of do this internally (reducing step size by reducing the gradients' scale), but we can always use a learning rate scheduler if we want an explicit LR schedule."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Can different learning rates be used on different parameters at the same time?\n",
    "\n",
    "The learning rate of different parameters can be different. Algorithms like Adam and Adagrad explicitly re-scales the learning rate to achieve faster training."
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": 3
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}