Chapter 06 - Convergence of Random Variables.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Convergence of Random Variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.2 Types of convergence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**$X_n$ converges to $X$ in probability**, written $X_n \\xrightarrow{\\text{P}} X$, if, for every $\\epsilon > 0$,:\n",
    "\n",
    "$$ \\mathbb{P}( |X_n - X| > \\epsilon ) \\rightarrow 0 $$\n",
    "\n",
    "as $n \\rightarrow \\infty$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**$X_n$ converges to $X$ in distribution**, written $X_n \\leadsto X$, if\n",
    "\n",
    "$$ \\lim _{n \\rightarrow \\infty} F_n(t) = F(t) $$\n",
    "\n",
    "for all $t$ for which $F$ is continuous."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**$X_n$ converges to $X$ in quadratic mean**, written $X_n \\xrightarrow{\\text{qm}} X$, if,\n",
    "\n",
    "$$ \\mathbb{E}(X_n - X)^2 \\rightarrow 0 $$\n",
    "\n",
    "as $n \\rightarrow \\infty$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Theorem 6.4**.  The following relationships hold:\n",
    "\n",
    "1. $X_n \\xrightarrow{\\text{qm}} X$ implies that $X_n \\xrightarrow{P} X$.\n",
    "\n",
    "2. $X_n \\xrightarrow{\\text{P}} X$ implies that $X_n \\leadsto X$.\n",
    "\n",
    "3. if $X_n \\leadsto X$ and if $\\mathbb{P}(X = c) = 1$ for some real number $c$, then $X_n \\xrightarrow{\\text{P}} X$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Proof**\n",
    "\n",
    "1. Fix $\\epsilon > 0$. Using Chebyshev's inequality,\n",
    "\n",
    "$$ \\mathbb{P}(|X_n - X| > \\epsilon) = \\mathbb{P}(|X_n - X|^2 < \\epsilon^2) \\leq \\frac{\\mathbb{E}|X_n - X|^2}{\\epsilon^2} \\rightarrow 0 $$ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. Fix $\\epsilon > 0$ and let $x$ be a point of continuity of $F$.  Then\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "F_n(x) & = \\mathbb{P}(X_n \\leq x) = \\mathbb{P}(X_n \\leq x, X \\leq x + \\epsilon) + \\mathbb{P}(X_n \\leq x, X > x + \\epsilon) \\\\\n",
    "       & \\leq \\mathbb{P}(X \\leq x + \\epsilon) + \\mathbb{P}(|X_n - X| > \\epsilon) \\\\\n",
    "       & = F(x + \\epsilon) + \\mathbb{P}(|X_n - X| > \\epsilon)\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "Also,\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "F(x - \\epsilon) & = \\mathbb{P}(X \\leq x - \\epsilon) = \\mathbb{P}(X \\leq x - \\epsilon, X_n \\leq x) + \\mathbb{P}(X \\leq x + \\epsilon, X_n > x) \\\\\n",
    "                & \\leq F_n(x) + \\mathbb{P}(|X_n - X| > \\epsilon)\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "Hence,\n",
    "\n",
    "$$ F(x - \\epsilon) - \\mathbb{P}(|X_n - X| > \\epsilon) \\leq F_n(x) \\leq F_n(x + \\epsilon) + \\mathbb{P}(|X_n - X| > \\epsilon) $$\n",
    "\n",
    "Take the limit as $n \\rightarrow \\infty$ to conclude that\n",
    "\n",
    "$$ F(x - \\epsilon) \\leq \\liminf_{n \\rightarrow \\infty} F_n(x) \\leq \\limsup_{n \\rightarrow \\infty} F_n(x) \\leq F(x + \\epsilon) $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3. Fix $\\epsilon > 0$. Then,\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\mathbb{P}(|X_n - c| > \\epsilon) & = \\mathbb{P}(X_n < c - \\epsilon) + \\mathbb{P}(X_n > c + \\epsilon) \\\\\n",
    "                                 & \\leq \\mathbb{P}(X_n \\leq c - \\epsilon) + \\mathbb{P}(X_n > c + \\epsilon) \\\\\n",
    "                                 & = F_n(c - \\epsilon) + 1 - F_n(c + \\epsilon) \\\\\n",
    "                                 & \\rightarrow F(c - \\epsilon) + 1 - F(c + \\epsilon) \\\\\n",
    "                                 & = 0 + 1 - 1 = 0\n",
    "\\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, to show that the reverse implications do not hold:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Convergence in probability does not imply convergence in quadratic mean\n",
    "\n",
    "Let $U \\sim \\text{Unif}(0, 1)$, and let $X_n \\sim \\sqrt{n} I_{(0, 1 / n)}(U)$.  Then $\\mathbb{P}(|X_n| > \\epsilon) = \\mathbb{P}(\\sqrt{n} I_{(0, 1 / n)}(U) > \\epsilon) = \\mathbb{P}(0 \\leq U < 1/n) = 1/n \\rightarrow 0$.  Hence, then $X_n \\xrightarrow{\\text{P}} 0$.  But $\\mathbb{E}(X_n^2) = n \\int_0^{1/n} du = 1$ for all $n$ so $X_n$ does not converge in quadratic mean."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Convergence in distribution does not imply convergence in probability\n",
    "\n",
    "Let $X \\sim N(0, 1)$.  Let $X_n = -X$ for $n = 1, 2, 3, \\dots$; hence $X_n \\sim N(0, 1)$.  $X_n$ has the same distribution as $X$ for all $n$ so, trivially, $\\lim _n F_n(x) \\rightarrow F(x)$ for all $x$.  Therefore, $X_n \\leadsto X$.  But $\\mathbb{P}(|X_n - X| > \\epsilon) = \\mathbb{P}(|2X| > \\epsilon) \\neq 0$.  So $X_n$ does not tend to $X$ in probability."
   ]
  },
  {
   "attachments": {
    "image.png": {
     "image/png": "iVBORw0KGgoAAAANSUhEUgAAAk8AAACHCAYAAAAP472eAAAgAElEQVR4Ae2dP6gc1xWHVbgIwSZWCEljQoxSmOAihQoXrlJELlykCCiFCxcuIkiRIhAjAipcGJEiARMiosIEFypESBGJFC4cUOHCCAlcGGKEixQpLEiRIoWLDd/GP+Xs0cz+eTv79N7ud2E0M/eef/e7d+49b3b13pmZRQISkIAEJCABCUhgbQJn1pZUUAISkIAEJCABCUhgZvLkJJCABCQgAQlIQAIbEDB52gCWohKQgAQkIAEJSMDkyTkgAQlIQAISkIAENiBg8rQBLEUlIAEJSEACEpCAyZNzQAISkIAEJCABCWxAwORpA1iKSkACEpCABCQgAZMn54AEJCABCUhAAhLYgIDJ0wawFJWABCQgAQlIQAImT84BCUhAAhKQgAQksAEBk6cNYCkqAQlIQAISkIAETJ6cAxKQgAQkIAEJSGADAiZPG8BSVAISkIAEJCABCZg8OQckIAEJSEACEpDABgRMnjaApagEJCABCUhAAhIweXIOSEACEpCABCQggQ0ImDxtAEtRCUhAAhKQgAQkYPLkHJCABCQgAQlIQAIbEDB52gCWohKQgAQkIAEJSMDkyTkgAQlIQAISkIAENiBg8rQBLEUlIAEJSEACEpCAyZNzQAISkIAEJCABCWxAwORpA1iKSkACEpCABCQgAZMn54AEJCABCUhAAhLYgIDJ0wawFJWABCQgAQlIQAImT84BCUhAAhKQgAQksAEBk6cNYCkqAQlIQAISkIAETJ6cAxKQgAQkcKoI3L9/f/bw4cNTFfMugr1z586Mo7LgfhcFH3BP4RpfDx48SNWkZ+zuyvYUgZo8TUFRGxKQgAQkcCwE2FDPnDkzu3z58rH4O6lOSFwuXbo0Z5Hk6caNG/P727dvTx529cUYwJ9x2IUvgj937tzs/Pnzk/djKoMmT1OR1I4EJCABCRwLgf62ZR2nFy9eXEfsVMnAgQQmyRNn6tYpSYDWkUUG+frmibopkicSvqEEDF/4PKnF5OmkjoxxSUACEpDAJATYhE/yW4yjdrInT5vYIWnZ9u3dFMkTb7SGkqdN+vIkZE2engR1fUpAAhI4UAL5rgzd57q/zahYSA56O4lQr+eeg1Kv4+PChQvz5ClteVNTfeU69jnX67THRu7reVlb+o3N2qex+mq3X8cP5/rmqdqKTt5GIYtv7klW+FiMxCW2Is89dqIXVpHLPfJJnsKpttHedWp8yOZjxmvXrs1laafQFt15RfknNkrV/LLK1+suN9W9ydNUJLUjAQlIQAJLCbDx8Qbo7NmzMzZMNl3OJDdcp7CpsrFnE+Ujt7TThn7emmAzydHVq1fnOsijT6Gda5KFVZtq3dCTWKCPP/wSKzLVPj5IRmq89BFfKcSHHQpyeQs2Vh+9fiYGdGHBQRw1eerfQ0pc2Ek/qCM27KDPdWLjnHripJ1jyBc28Z0+wAX/MKJgt8aHjRpfxhYbjBvyyESXODhSiA1fyCQedChpQ35oDsTGlGeTpylpaksCEpCABJYSYHOrmyLCbNRsjNlQSVZqQSft1HOd5Il72tmEU9hUa1Ix5DOyQ+ds6GnDX42520+SEHliQ4eCLLr0jcKZtrH6udDAP/GZBAOR1MU2dcROgkTBDwlXCnGhk7bKMDKVZRIV2lb5QoY4qv91dKp8YuBcxwy7zIkkeYmHuvCocacd25VNtb/t9f9n27aW1JeABCQgAQmsIFA3xYiywbPR5Y1CTVSQySaczZOkoG783Wbks3H29vgdO/cNvfvr9qsd2urbJWKgPxzEkcRmrL7aqtfY5O1ZLUNx1NjhSoKBbt70Rb/3KfVjrFb5ij79xB9lHZ0ab2xwrnEQO3IZz8hRlzddVX7Md/SmOJs8TUFRGxKQgAQksBaBvsmhlE2WM5v6WPKUxKNv/N1m7GWzre3UkaSRWGAnR03G+obe/XX7JHW8fcIGtqs/+odP/JFUJJlZVj8EkhjGuKSf6PXYiS3x4HsoASXm2Ihsj6H3ecgXdeHJ9To6Nd7Ehm6Ng2vkEiPtFOoyblWetiHf/9Oa5l+Tp2k4akUCEpCABNYg0Dc5VLI5cs1m2JOEvJmK+Z7MdJt946zttCUJi71+rhs6bd1ft09Sgo+U+EPu3XffXWgjQcD+rVu3But7ghCbvGHBT23vcSBbYyehq/JJ8JCrfSLeyCX2+M15la/IVRbr6NR4kwhhq8bBeCGXGGkn4aMOH12e+yHfc8GJ/jF5mgikZiQgAQlIYDUBNkU+fmLzo7AhsuGSIOWe5CmbInVs9Pl4JvdjGy3tfePENj7xxXV9w4F8L3VDp60mGtx3+8gnftrxRR/o6+9+97tHvmlLf7GRmGo910MFPWzWxA/7+A5L9GrsnRvMwpG3YCRTlGUsE0vvc3wRQ0rlTF0SnMRHe40PGRgkpmVxEG/1xXXixw73NekeijdxTnE2eZqCojYkIAEJSGAtAmxybJhspGxw3HOuJUkOiULk0o48SQEbJ3q0c01dbLHRJuHJxs0GnSO2+hlZNnB0sZH44i+xdPvIoUe8+IgdrknU4he5tI/V95jqfeUSNun33bt3F2InBngQM7KRjz1sEXPaqQ9LeKIbdsRd+xwb4RHb9A27tdBGPWfshG1swyFxJKlNHMhyncI1duKv1q8zByI/xdnkaQqK2pCABCQggbUIsCnXNwRrKSkkgRNGwOTphA2I4UhAAhLYZwImT/s8uofTN5OnwxlreyoBCUjgiRLIxzH5WOiJBqNzCWxBwORpC3iqSkACEpCABCRweARMng5vzO2xBCQgAQlIQAJbEDB52gKeqhKQgAQkIAEJHB4Bk6fDG3N7LAEJSEACOyDw2Wefzf7zn/88ZvnmzZuzn/3sZ7N//OMfC23Ivv3227O//OUvC/XenHwCJk8nf4yMUAISkIAETgiBf//737OPP/74sWj4Mjy/APL73//+QtsXX3wxe/rpp+dtb7755kLb9evX5/VPPfXU7F//+tdC2zvvvDN76aWXZvfu3Vuo9+ZkEDB5OhnjYBQSkIAEJHAKCLzwwgvzhKf+8kbCTiL0ne9857FekDSRVPVEiPtvfOMbs1deeeUxnfj5+c9/vtBG8oY93mZZnhwBk6cnx17PEpCABCRwAgnwZolk5yc/+clCdLxF+u53vztPnn7/+98/1vbXv/519vnnny/UH/WG5Oj111+fffrppwsmeCPFGy4Oy5MjIP0nx17PEpCABCTwBAl88MEHs5/+9KdLExTe9NTyz3/+c/b+++/XqmO9XpbY0RcSvh7zsQZ4IM5Mng5koO2mBCQggUMl8OGHHw5+T+nFF1+cv8F54403FtDwRe5f/epXp+qjMd5Q5Y2UH+ktDOdObkyedoJVoxKQgAQkcNwE+peu8f/RRx/Nkwq+lN0/AvvNb34z/xjuSb5JmooRHymS8PH2qb954uPEX//61zNkLNMQMHmahqNWJCABCUjgCRIgEeLNS/8fbZ988sn8f7vxxeypvo/0BLt5JNckjrAhgbJMQ8DkaRqOWpGABCQggWMgwP8+43s9/fcp8eVqEoRXX331sSh4E9PfxjwmtMcVL7/88jyB5ONLyzQETJ6m4agVCUhAAhKYiACJDm9J+u9T4sva+V7Pn//85wVv/AJK/ida/0WUC0LeLBCA11tvveXHeQtU1rsxeVqPk1ISkIAEJDAxAb6j1L+HhItf/OIX8ySp/8JJ2vhez2uvvXawH8FNNQT8NvQkou+9995UZg/GjsnTwQy1HZWABCRwsgjkdyb1t0j8uZKvfOUr8z9pcrIi3p9o+PI4H3GSoA4lsPvT0930xORpN1y1KgEJSEACs9n8o7exXzj53HPPzd9++ObjZE0VklcTquVjYvK0nI+tEpCABCSwBoFVv3CSN0n9S95s0Pw3esvJIZC/0ffss88e9JfsV42IydMqQrZLQAIS2HMC/YvZy7o79gsn87fY+i+c5Mvf/i22ZURPVhvjy682YDwt4wRMnsbZ2CIBCUhgrwnwpeEf//jHszt37jzWz6H/2o8cXzIe+oWTb7/99t78wsnHYBxYBV/k728JDwzByu6aPK1EpIAEJCCB/SLAxsh/UeejtKFfnEgbSRIJUS28oSJxOuRfOFl5HNL1lStX5r8r6vr164fU7dG+mjyNorFBAhKQwP4R4H+25X+5Pf/884O/cJL/hUXyNPQLJ/ldSxyWwyLwyiuvzOcEvybCMpuZPDkLJCABCRwAAb6cnQ2QxOiZZ56Zb4Zc918VwMd5/LmTQ/1zJgcwHTbuIvOHN5Emzv9DZ/K08RRSQQISkMDpIcBHdPxiST6iI1Gqxze/+c3ZD3/4Q5Ok0zOcRnpCCJg8nZCBMAwJSEACuyLAl7/v3bs3u3nz5vztAR+9vPTSS/PvLpFM8bfi/LMmu6Kv3X0kYPK0j6NqnyQgAQmsSYD/WfXRRx/NP7ob+h92a5pRTAIHRcDk6aCG285KQAISkIAEJLAtAZOnbQmqLwEJSEACEpDAQREweTqo4bazEpCABCQgAQlsS8DkaVuC6ktAAhKQgAQkcFAETJ4OarjtrAQkIAEJSEAC2xIwedqWoPoSkIAEJCABCRwUAZOngxruxc5eu3Ztdu7cudn58+cXGya8u3z58uzs2bMTWtSUBA6TAH+Ul+eV38v08OHDjSE8ePBgduHChbn+7du3R/XrMzumw5px8eLFURs2SGDfCZg87fsIr+jf1atXJ0ueSMZ6YcGn/iiLfbflvQQOnQDP01GTp7BDf1nyNPTMdp0bN27MOGoZev5ru9fTEGAthTVr9/379x8ZZUyXjesjwQ0v8IevrOGMO/e78EVo9Ok0zCWTpw0n0r6JT5U88WDt8g3WvnG3PxI4CoHjSJ6G4urJU5fx+e9EdncP68yDmsDwVpI3i+sU3iiuW0iWGH98UjjjizeU25ahOC5dujT/tCLJ2rY+dqVv8rQrsqfE7lTJEw+SydMpGfQDDJPfnJ3F/zR3P5vmNhvLqkRoiM8qnUN//j/88MMZf0PwOMuqMVkWy7pJ1pgN9KdInraNYyy+46g3eToOymv64HUlWTc/TfDakjPfK8hPGvX7DmPfRYhedOtPJoTBffXB5E3Sgx8eCB7KvBbm+0rUExs/gaBPwoWNLODUY4f4aOOgjRhrzMGADfxwTrxp62dsEwP+IovNGg9tHLUgGx+0IU8hrtjBNnxrG7HTf860Iwuf9LX68Pp0ESDxeOGFF2Zvvvnm/O+87SJ65knmEHOLOd7nWZ5d5hWyzM9sIuhn3qLHdZ17SZ7Qi+0qg+zY/E5/md+JLc9inoGxZ7Zu1OjwTGbdIM7+/N+9e3fuAz36R1zYRocDG/tUPvnkk/nfCnzjjTdmH3zwwbF0rY7JJg6ZQxm7TfSqLOPNvNumTBHHNv631TV52pbgRPosLCxILEQpWYSzeGbhzD1y9QFiAeQePQo2uc/CyIJF4kF9Cg9Af5DQyYKHLeRZbKlPoZ26FOS6Hdp6zD0GFnrs1j7FZs48qNiODDHXflHPPb4oPeHJYp+2KgtvuNdCO/1LwXeYpm7q82effTZfdFl4PXbH4I9//OPs61//+ny+vPjii7O33nprJ38QlzlUn+U8B3n2co8MzydzlnnMXKuJBe08synRix3qq0yepzwLtA3Nb+pTsEW8sRkfed6Qo73G1Z/3fo8O+n1N47mKn32b83/6059m3/ve9+asnnvuucmTdLgxboxD1vqMCfeMAeOfgjztHOgxrpyZTxzIczBOHLRxH73Y4kx99hHsJ3nCduzGL7a6Tmwn3mVx1LiqzfQlc5s2Yo2vGneNNTamPv9/N5zasvY2IkBCwELDxEvpi1i/R64vasgwiVJqOwtzTXiQYaJSX0vVST0266Ttev0+ej3mHgP9rXajV895UFM35KvGzMLQfyqiHT/xF849PnxUW9x3/4ljyvOVK1fmfvHtsXsGX/va1xY4v/zyy7Pr16/PPv/880mGtc8hjPJ8Z14OzTs2FPQyNxMIdbRRhvR4NuPvKPMbu/WZGfIR+4mpP4P9PnIkSzw/FGKrP5Ts65z/6le/ujC3pkjSk2xkbmSuJBlhDrC2Zy1nDMMd9rRl7o2NVRIyZDMHqMNHH39sM2cSD3L4Jg7qug726hwjpqE40KUef9127kmW6j4WFuhR8MWzRiy7LCZPu6S7gW0mYyZ+1DKBM2n6PXJ9UiPDQ8IEi3weMGTzAMXH0ATuNpElhmT4TFYWwRrvkB30EkP6MBRDYhk7w6bGPeSrxsw18eG7HnmYiB97nOkT8omPGKot7rv/sTi3qd/XjQSWJ/H41re+tRBX3hR8/PHH2wzjI90+h2ioz3h/LmhnXqNX5yL11GX+D+l1mU3n9zqx9f70Z7DfY5OSeNlciStrEW37OuefeeaZhbn17LPPzvg4DxZHKaxbnT92el0dA66Zbynhz32VS3vO2GRN7KX7GloTqatJzSqdsTgyZ/IckHQxd2qhLnFGnj6mdN+pn/Js8jQlzS1ssTgy4LVkUmQS9Xtk6yRhYSLjRi4l7fnJoE5uZIYmcHRigzOJEglJYoleftLIPbL4zyLZY8ZOjwGbsVt95ro/qNVXZGrM/SccZOIDzsSQRCrxcZ+6agvd7j8+pzzfu3dv9u6773rsmMFvf/vb2be//e35c/P000/PXn/99dn7778/5VDObfU5ROVR3jwxb7GVzSPztT4vzNv4O8r8Jrb6zAz5iP2A6s9gvUc/z39s017XD+r3bc7/4Q9/mK8tsHrqqadmP/rRj2Y3b97c+ovkQz/gwW/ZmDA/WOeYc6xfSTTQq2PFfS3dZtp6/dCaiF3mUebmKp2xOOr8y3WdT8SEfw5KZOKXuu57LjjxP4u79cTGNbc+ARZAJnoWSTT7Q0MCxKTIJMmkycTqk7G28yAhh49a+hsk2oYmXq9j4mITn9glVh4cCvf4piSGxExb7QMyLPhJXOZK7Z/+oPZ+Il7jG+pnfAzZii4y3Rb3XWcu5D+njgBf6uUjlFdffXX23nvvbb2pLQPAnKrPcuZk5nl/LrDFM5LnNLaxQV2en+jFDnI8e5Hpc5VnZWh+19hYV1gX8pN7fMQnPmIjcfVncOz5T3zYz/MVG/t05n9z8tEvxzvvvDPZx7/hB/86HtQvG5PMD8aUsan869ghl3Efspkx6r76PEMOu3V/WaUzFkedf7nm+akF/xyUyFQ+3XfVnera5GkqkhPYYRLzVoaFjYPFhklQJwV1HEwmZGhnEuUhQD/tPDRcxyYhokcdEw592rCBLDZo477qoMdExw/6yKLLgo088XGgg1zq8BH7JGkp2KA+fUBurOSBJDHDZ/wSI201ZuKLrfQz8dZ64o4tzsTCcevWrUf9j63ufyxO6082gS+++GI+5lN9p2lVb/MMZY4z/zMHOee5yLMYezxHyKKXOcwcT2GNYE5yZG4jHxnqxuZ3Nknko8/8H4stdrMm5JmozyDXlKHnPzETGzziP/X7cmZu8Ybp008/3UmXwo+xrQWmtY4xZewpXGdsuM+8SFvkmIvVRrc5NzaQqDEXmBe1UMe8Tum28Fl1arw1Dq7RZU5RSMhqX1LHPkTp8tR133PBif8xeZoY6JTmhibFlPa1JQEJ7IbAcSzeu4l8N1ZJxCxHJ0DyQHKSwj1zjAQkhcQkb/+TmCQByQ/SyLKvkJCQlCGXxBvZoXmb+prAJFGKLm31Izv8IFMTHHzW5GosDpI54ohtkm5s5R6b1U6XH4o3jKY8mzxNSXNiWyZPEwPVnASOicDQJnRMrk+Mm7xpYnPLJnpigjuFgZA8wJKDaxIfuHJN8sI9B9fsHVU+Y5Fucx856mKv2hiqR44Se4mHcy8kMfFBPIkxurETGe6RSwycU7AVX1W/yw/dx8bUZ5OnqYlOZI9JwE9rLMJ5fT6Rac1IQAI7IsDmko+5+OmY5/gQCxzqR+1sfhYJ7BMBk6d9Gk37IgEJSEACEpDAzgmYPO0csQ4kIAEJSEACEtgnAiZP+zSa9kUCEpCABCQggZ0TMHnaOWIdSEACEpCABCSwTwRMnvZpNO2LBCQgAQlIQAI7J2DytHPEOpCABCQgAQlIYJ8ImDzt02jaFwlIQAISkIAEdk7A5GnniHUgAQlIQAISkMA+ETB52qfRtC8SkIAEJCABCeycgMnTzhHrQAISkIAEJCCBfSJg8rRPo2lfJCABCUhAAhLYOQGTp50j1oEEJCABCUhAAvtEwORpn0bTvkhAAhKQgAQksHMCJk87R6wDCUhAAhKQgAT2iYDJ0z6Npn2RgAQkIAEJSGDnBEyedo5YBxKQgAQkIAEJ7BMBk6d9Gk37IgEJSEACEpDAzgmYPO0csQ4kIAEJSEACEtgnAiZP+zSa9kUCEpCABCQggZ0TMHnaOWIdSEACEpCABCSwTwRMnk7ZaN65c2d29erV0eP27duzBw8ePNZOfS0PHz6c3bhx45Hc/fv3583UUYb8oEOp/q9duzYoi0xszZW+/Kf6rHZyjb1Vhf4hh07iXqZDX9aRqzaG+j/Wp6pXr9PXdfpU9fo1sQyx7HJHva99zRh3W5lTvX7d+3V8rGtLuekJML6sEes+U9NHoEUJnC4CJk+na7zm0bLBnTlzZnb58uWF6Nmkz58//6iOduT6hshGfO7cuXkCgjDtLJoXL15c0B/zgw6yNSkYkqUdP2yctbBIE1dP6Hr8VSfXxE4fWexJiLDf+xdZ2unX2bNnH/MVmWXnTfo0ZocxqGMyJpd6mPT+wJp+7rJkTLrv+GRs4LhpEhp9zqt8VNmh610mkEP+jlI3NH5HsXOcOjxLjC1n5tpp4HycfPQlgSECJk9DVE5B3VDyRNgsfikkDj15YmGkric06CDfN/oxP8iyUdQyJIu9nuDgG9mujy3iH9vAs8hHj/OFCxdqCAvXbPTojPlaEB65WbdPI+qDTMdkqe+slslO2ZYxGWM/ha9tfDCOfW5OEdPUNp7U+G3Tj23GZRu/6krgNBMweTqlo9c39bwRqMnHUPLET5g1wardH9qgup/Ir5s8kdzgs27KWayTBBF73mJht8rGH2fe4mBr0zJ18jTUJ2ImMeXIWBAn/embPv2n7+k/ctQxLsSKTjhQz3X4pO/4QB9/4RVZ5BlL2tGr8aS+t6GL77THP/6o4z519R7baaN+WYmPxE5siT16QxzpY5JwfKFX+8p9YkiMuaeNMmQ3PrEPDw5KtR0e2Kkco5szOn387t69+1hc1fatW7ceta/i2GPEbzjSRnydZWLLmf7hH/kU7i9duvRo3oVB2usZ+2HdfQ3Zxk/GKxwzPnUOpa6yiX3qsJ1xjAz32AiXxIksR+qRy7hFnrrYzzxBBzn04is2OccvcrUM2aztXu8vAZOnUzq2PanpGzTdYmFALgsFCwD3Q4tDMLB41NL9pA3bfSHpsvgl2UG2lsQRfeLpH0FW+VyTtPCTPfaymKVvkRk6E1d8DbUvq1unTyy6xMVCSuE6/oi1jk3lxmZbeYdL71P/6I/NLkyRxT66FPwSM7Zpoz7Ja2Tngl++5Uuc8Y1tCvbrW73YTWyc8YM81zDAT+KIj3qOj8w/9GAVnU04JsbaN2yFJ7bCfcwu/pEJA/qSeZj+DnGsfarX6R92U7iGU+KiHq7pc9rHOI7FiH7GChn6znmoxEZ8hk3m61Dc3Q6ysIYL+pkbq2zDk9gSa51X6A6xwT6ljgf8Mp7EgE3GBntcd3l0OZBNghP99CX9p54j85K+5fnCLn2gjnhjk/oaU7c5D8h/9pqAydMpHV4WnTzkLCJZGGp3WACQ46Gn5J4FZd2CfjaUqoOtbgdZFhcW4ywsXQYbWayz+BH7kI/qj2sWSRbwLHrxkf51+dwT11AcaV92XqdPjAP9ToENdRSu69gQfxZpYsrmj2y49P5UG0My2Ks+iBk2Kek/ujCs/KirvtPW/fR7dGI3fmCQjSx19Txkg75FZxOO2CVWYkgfsJNxgG04j9mtvrGHnToeYxxrn+r1UP9oJybmOoWxzXV0l3Eci/HKlSuP5hh2eH76vIn9On9SRwx55sbijixnGCZuuMMJvXVtj80rbMZuZZOYokcMGeu0JclKnJUjY595hV3irc8E/SF2Smdc+9R9wQzdVTYTk+f9JWDydErHloUii19+kuxdYRFALosqGwr32VQiz6KCbI6aaFQ/keeMbJWjrssiw6ZeF0DksiBFn/v0hVi7fPyyaNUkoduJXD8TV3z1tlX36/QJGTYA4uFIMkNfwiB+aGO86CPX6GZ80p/cR6fa4Lpu8Mh0vd7feg8/7uvmMWSj2+z36FS73MOYuh4/bZQhG1UH3XU5fmlyPh+YO/CkT2yY+CdhyTwas/uDH/xgLk9cOZCtenXe0FbvE0POQ/2jDR3GLGOOXC3dbmWSPiW++Pjb3/42t4ldmNXEoNrmGhsctdQ5FZtj40bcNUbk0Oe8qe3uK2ywRR9op2A/PqnjYO5GhrYeb+Sjn7UiPrEZW8yPJNqVRddFJnZoI16OVTaRtew3AZOnUzq+LBRJOOjC0OKZBSiLDOcstr3btGEzC0rau5/U45sFpJYuWxeYKpd6FiEKvrNhZXGq8rlmoa4LWbcTuX4mrvjqbavu1+lTl6k2xxbmugBnfNKf3MdOtcF1EoS0Ry8Me39zj10OkjcSN+ZC5lBsxPeqe3zHbuKgT9QljtTn3G1Sn7ps0IknOjlXBqnjTD/gkaSUPmEzbzOQIaYhu30+VbvRq/Om97fLpy9hWNsTY40r7d1u5TgWYxjjk751G7HNGRvdb+YR7cviru2VRexvanvIF2PG+oWtFOKjT0Msh2ygRx850GEdS7yRz3185NznVr0f47/KZmx73l8CJk+ndGxZWIY2hNqdoQWIRQpdHv5ehmz2n7yiU5OY1HX9LDBJyOqCj+zQYoZsfx0f+9kocx/7Y/KRG/OV9mXndfrEAps+xhZ9YxGvC3HYdw70Iwf+0EdUUKkAAARrSURBVOM+fKqN9Dk28Ec7m3NK72/u0UU2BfsZx9jNZrXqHhuxG3vMRzbCsdJtIkc80VmXIzqJEw7EkY2XcUiikjjG7PLRV+WGPDGGbe9fv4/9nGv/uM740Q4bfFX+0et2K0fkh2L85S9/ufAMo8MxVIZsVCY17iF9WBMjz18KjNDb1PaQL8aMeVifochlLPAbn2nLHEhMJE6RqW1cJ0GLLOf8wEkf8hxQX+8zbtUe4/r3v/99qc3qx+v9JGDydArHlQeZxawuNkPd4MFHrj74yLFo9A2GhZEFpi/A6LKwsKBQWJzwywLWC766Pjaz+GeDY/FBtm4u2CKGoXjjJ7FkEWexrAzwFR9VB5tZKFOPLvKdTdpzXqdPJG/0Mf3BV2KERxbm9DsbQpjDljZiISauqUtSWG0QF33OT9jxnfHARmVb75Ehzuo/cSa28Fh1TxyVTY8j/OoZ/+kf9fiCDb4osZH7yjG6xE7fa8FG5kHiTh+X2Y3/zG38x07lho1+X/3nGpn0r44f7cQDrxpX9JZxHIsR+3Wuw4T4h0q3keefekqY5X7IBjpwxgdycKIvm9oe8oVNGPT46zzHT+b8kA1ihgfPCmw4mNvpU40fWeSYU7nOM9rve//QQZeyzCbzoI7PXMF/9oqAydMpG04e3iwOOWcRSFdY1NKWc5dhUeDhT3sWxS4XmyxYyNYFKW1DMaUtfvBFXNVnfNdzNvPo93PsodNjxXZdgId8xR669GmsbNInbBAXsde4qn+uKcSHHL7hwbn2OWOX2IZsYCfxoYtvSnTDMzK553sy1GGbI6y63Kr7ubMvk6f0hTgTR9r7mfb0GR/ocl/LEMe0J67EXeurnbBOO+dldsM4eqs4Vl/VB9fRzfjV9p70pY3EYRXHHiMs4JCxXBZT/HQb1Idp5sgyO+nb0LitY7v7SlycsTlUokM7Y5j7xMs9hdiSiFPHkR860ItM9NLPxE091/1+rvhlooRMH9cwoS020cFOn6ex5Xk/CJg87cc42gsJHDsBNv2+mRx7ECfYIZt2NlU20zFWctx+EOGcN0LVGm+AalJT27yWwDYETJ62oaeuBA6UAImBm/7ywWfTZvPmLcjYWyc5Lme4bitvefiYrL7tIWGtH8eta0s5CaxDwORpHUrKSEACjwjUjyryZuVRoxcLBEiceOOUj45qoxwrje2vYRzeMK+J1PbWtSCBRQImT4s8vJOABCQgAQlIQAJLCZg8LcVjowQkIAEJSEACElgkYPK0yMM7CUhAAhKQgAQksJSAydNSPDZKQAISkIAEJCCBRQImT4s8vJOABCQgAQlIQAJLCZg8LcVjowQkIAEJSEACElgkYPK0yMM7CUhAAhKQgAQksJSAydNSPDZKQAISkIAEJCCBRQImT4s8vJOABCQgAQlIQAJLCZg8LcVjowQkIAEJSEACElgkYPK0yMM7CUhAAhKQgAQksJSAydNSPDZKQAISkIAEJCCBRQImT4s8vJOABCQgAQlIQAJLCfwXgLXWnyyFRf0AAAAASUVORK5CYII="
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![image.png](attachment:image.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Theorem 6.5** Let $X_n, X, Y_n, Y$ be random variables.  Let $g$ be a continuous function.  Then:\n",
    "\n",
    "1. If $X_n \\xrightarrow{\\text{P}} X$ and $Y_n \\xrightarrow{\\text{P}} Y$, then $X_n + Y_n \\xrightarrow{\\text{P}} X + Y$.\n",
    "2. If $X_n \\xrightarrow{\\text{qm}} X$ and $Y_n \\xrightarrow{\\text{qm}} Y$, then $X_n + Y_n \\xrightarrow{\\text{qm}} X + Y$.\n",
    "3. If $X_n \\leadsto X$ and $Y_n \\leadsto c$, then $X_n + Y_n \\leadsto X + c$.\n",
    "4. If $X_n \\xrightarrow{\\text{P}} X$ and $Y_n \\xrightarrow{\\text{P}} Y$, then $X_n Y_n \\xrightarrow{\\text{P}} XY$.\n",
    "5. If $X_n \\leadsto X$ and $Y_n \\leadsto c$, then $X_n Y_n \\leadsto cX$.\n",
    "6. If $X_n \\xrightarrow{\\text{P}} X$ then $g(X_n) \\xrightarrow{\\text{P}} g(X)$ .\n",
    "7. If $X_n \\leadsto X$ then $g(X_n) \\leadsto g(X)$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.3 The Law of Large Numbers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Theorem 6.6 (The Weak Law of Large Numbers (WLLN))**.  If $X_1, X_2, \\dots, X_n$ are IID, then $\\overline{X}_n \\xrightarrow{\\text{P}} \\mu$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Proof**: Assume that $\\sigma < \\infty$.  This is not necessary but it simplifies the proof.  Using Chebyshev's inequality,\n",
    "\n",
    "$$ \\mathbb{P}(|\\overline{X}_n - \\mu| > \\epsilon) \\leq \\frac{\\mathbb{E}(|\\overline{X}_n - \\mu|^2)}{\\epsilon^2} = \\frac{\\mathbb{V}(\\overline{X}_n)}{\\epsilon^2} = \\frac{\\sigma^2}{n \\epsilon^2} $$\n",
    "\n",
    "which tends to 0 as $n \\rightarrow \\infty$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.4 The Central Limit Theorem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Theorem 6.8 (The Central Limit Theorem (CLT))**. Let $X_1, X_2, \\dots, X_n$ be IID with mean $\\mu$ and variance $\\sigma^2$.  Let $\\overline{X}_n = n^{-1}\\sum_{i=1}^n X_i$.  Then\n",
    "\n",
    "$$ Z_n \\equiv \\frac{\\sqrt{n} \\left( \\overline{X}_n - \\mu \\right)}{\\sigma} \\leadsto Z $$\n",
    "\n",
    "where $Z \\sim N(0, 1)$.  In other words,\n",
    "\n",
    "$$ \\lim _{n \\rightarrow \\infty} \\mathbb{P}(Z_n \\leq z) = \\Phi(z) = \\int _{-\\infty} ^z \\frac{1}{\\sqrt{2 \\pi}} e^{-x^2/2}dx$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition to $Z_n \\leadsto N(0, 1)$, there are several forms of notation to denote the fact that the distribution of $Z_n$ is converging to a Normal.  They all mean the same thing.  Here they are:\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "Z_n                                           & \\approx N(0, 1) \\\\\n",
    "\\overline{X}_n                                & \\approx N\\left( \\mu, \\frac{\\sigma^2}{n} \\right)  \\\\\n",
    "\\overline{X}_n - \\mu                          & \\approx N\\left( 0,   \\frac{\\sigma^2}{n} \\right)  \\\\\n",
    "\\sqrt{n}(\\overline{X}_n - \\mu)                & \\approx N\\left( 0, \\sigma^2 \\right)              \\\\\n",
    "\\frac{\\sqrt{n}(\\overline{X}_n - \\mu)}{\\sigma} & \\approx N(0, 1)\n",
    "\\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The central limit theorem tells us that $Z_n = \\sqrt{n}(\\overline{X}_n - \\mu)/\\sigma$ is approximately $N(0, 1)$.  However, we rarely know $\\sigma$.  We can estimate $\\sigma^2$ from $X_1, X_2, \\dots, X_n$ by\n",
    "\n",
    "$$ S_n^2 = \\frac{1}{n - 1} \\sum_{i=1}^n ( X_i - \\overline{X}_n )^2 $$\n",
    "\n",
    "This raises the following question: if we replace $\\sigma$ with $S_n$ is the central limit theorem still true?  The answer is yes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Theorem 6.10**.  Assume the same conditions as the CLT. Then,\n",
    "\n",
    "$$ \\frac{\\sqrt{n} \\left(\\overline{X}_n - \\mu \\right)}{S_n} \\leadsto N(0, 1)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You might wonder how accurate the normal approximation is.  The answer is given by the Berry-Essèen theorem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Theorem 6.11 (Berry-Essèen)**.  Suppose that $\\mathbb{E}|X_1|^3 < \\infty$.  Then\n",
    "\n",
    "$$ \\sup _z |\\mathbb{P}(Z_n \\leq z) - \\Phi(z)| \\leq \\frac{33}{4} \\frac{\\mathbb{E}|X_1 - \\mu|^3}{\\sqrt{n}\\sigma^3} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is also a multivariate version of the central limit theorem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Theorem 6.12 (Multivariate central limit theorem)**.  Let $X_1, \\dots, X_n$ be IID random vectors where\n",
    "\n",
    "$$ X_i = \\begin{pmatrix} X_{1i} \\\\ X_{2i} \\\\ \\vdots \\\\ X_{ki} \\end{pmatrix}$$\n",
    "\n",
    "with mean\n",
    "\n",
    "$$ \\mu \n",
    "= \\begin{pmatrix} \\mu_1 \\\\ \\mu_2 \\\\ \\vdots \\\\ \\mu_k \\end{pmatrix} \n",
    "= \\begin{pmatrix} \\mathbb{E}(X_{1i}) \\\\ \\mathbb{E}(X_{2i}) \\\\ \\vdots \\\\ \\mathbb{E}(X_{ki}) \\end{pmatrix} $$\n",
    "\n",
    "and variance matrix $\\Sigma$. Let\n",
    "\n",
    "$$ \\overline{X} = \\begin{pmatrix} \\overline{X}_1 \\\\ \\overline{X}_2 \\\\ \\vdots \\\\ \\overline{X}_k \\end{pmatrix}$$\n",
    "\n",
    "where $\\overline{X}_r = n^{-1} \\sum_{i=1}^n X_{ri} $.  Then,\n",
    "\n",
    "$$ \\sqrt{n} (\\overline{X} - \\mu) \\leadsto N(0, \\Sigma) $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.5 The Delta Method"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Theorem 6.13 (The Delta Method)**.  Suppose that\n",
    "\n",
    "$$ \\frac{\\sqrt{n}(Y_n - \\mu)}{\\sigma} \\leadsto N(0, 1)$$\n",
    "\n",
    "and that $g$ is a differentiable function such that $g'(u) \\neq 0$.  Then\n",
    "\n",
    "$$ \\frac{\\sqrt{n}(g(Y_n) - g(u))}{|g'(u)| \\sigma} \\leadsto N(0, 1)$$\n",
    "\n",
    "In other words,\n",
    "\n",
    "$$ Y_n \\approx N \\left( \\mu, \\frac{\\sigma^2}{n} \\right) \\Rightarrow g(Y_n) \\approx N \\left( g(\\mu), (g'(\\mu))^2 \\frac{\\sigma^2}{n} \\right) $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Theorem 6.15 (The Multivariate Delta method)**.  Suppose that $Y_n = (Y_{n1}, \\dots, Y_{nk})$ is a sequence of random vectors such that\n",
    "\n",
    "$$ \\sqrt{n}(Y_n - \\mu) \\leadsto N(0, \\Sigma) $$\n",
    "\n",
    "Let $g : \\mathbb{R}^k \\rightarrow \\mathbb{R}$ and let\n",
    "\n",
    "$$ \\nabla g = \\begin{pmatrix} \\frac{\\partial g}{\\partial y_1} \\\\ \\vdots \\\\  \\frac{\\partial g}{\\partial y_k} \\end{pmatrix} $$\n",
    "\n",
    "Let $\\nabla_\\mu$ denote $\\nabla g(y)$ evaluated at $y = \\mu$ and assume that the elements of $\\nabla_\\mu$ are non-zero.  Then\n",
    "\n",
    "$$ \\sqrt{n}(g(Y_n) - g(\\mu)) \\leadsto N(0, \\nabla_\\mu^T \\Sigma \\nabla_\\mu) $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.6 Technical appendix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**$X_n$ converges to $X$ almost surely**, written $X_n \\xrightarrow{\\text{as}} X$, if\n",
    "\n",
    "$$ \\mathbb{P}(\\{s : X_n(s) \\rightarrow X(s)\\}) = 1 $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**$X_n$ converges to $X$ in $L_1$**, written $X_n \\xrightarrow{L_1} X$, if\n",
    "\n",
    "$$ \\mathbb{E} |X_n - X| \\rightarrow 0 $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Theorem 6.17**.  Let $X_n$ and $X$ be random variables.  Then:\n",
    "\n",
    "1. $X_n \\xrightarrow{\\text{as}} X$ implies that $X_n \\xrightarrow{\\text{P}} X$.\n",
    "2. $X_n \\xrightarrow{\\text{qm}} X$ implies that $X_n \\xrightarrow{L_1} X$.\n",
    "3. $X_n \\xrightarrow{L_1} X$ implies that $X_n \\xrightarrow{\\text{P}} X$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The weak law of large numbers says that $\\overline{X}_n$ converges to $\\mathbb{E} X$ in probability.  The strong law asserts that this is also true almost surely."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Theorem 6.18 (The strong law of large numbers)**. Let $X_1, X_2, \\dots, X_n$ be IID.  If $\\mu = \\mathbb{E}|X_1| < \\infty$ then $\\overline{X}_n \\xrightarrow{\\text{as}} \\mu$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A sequence is **asymptotically uniformly integrable** if\n",
    "\n",
    "$$ \\lim _{M \\rightarrow \\infty} \\limsup _{n \\rightarrow \\infty} \\mathbb{E} ( |X_n| I(|X_n| > M) ) = 0 $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If $X_n \\xrightarrow{\\text{P}} b$ and $X_n$ is asymptotically uniformly integrable, then $\\mathbb{E}(X_n) \\rightarrow b$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **moment generating function** of a random variable $X$ is \n",
    "\n",
    "$$\\psi_X(t) = \\mathbb{E}(e^{tX}) = \\int_u e^{tu} f_X(u) du$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Lemma 6.19**.  Let $Z_1, Z_2, \\dots, Z_n$ be a sequence of random variables.  Let $\\psi_n$ be the mgf of $Z_n$. Let $Z$ be another random variable and denote its mgf by $\\psi$.  If $\\psi_n(t) \\rightarrow \\psi(t)$ for all $t$ in some open interval around 0, then $Z_n \\leadsto Z$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Proof of the Central Limit Theorem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let $Y_i = (X_i - \\mu) / \\sigma$.  Then, $Z_n = n^{-1/2} \\sum_i Y_i$.  Let $\\psi(t)$ be the mgf of $Y_i$.  The mgf of $\\sum_i Y_i$ is $(\\psi(t))^n$ and the mgf of $Z_n$ is $[\\psi(t / \\sqrt{n})]^n \\equiv \\xi_n(t)$.\n",
    "\n",
    "Now $\\psi'(0) = \\mathbb{E}(Y_1) = 0$ and $\\psi''(0) = \\mathbb{E}(Y_1^2) = \\mathbb{V}(Y_1) = 1$. So,\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\psi(t) & = \\psi(0) + t \\psi'(0) + \\frac{t^2}{2!} \\psi''(0) + \\frac{t^3}{3!} \\psi'''(0) + \\dots \\\\\n",
    "        & = 1 + 0 + \\frac{t^2}{2} +  \\frac{t^3}{3!} \\psi'''(0) + \\ldots \\\\\n",
    "        & = 1 + \\frac{t^2}{2} +  \\frac{t^3}{3!} \\psi'''(0) + \\ldots\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "Now,\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\xi_n(t) & = \\left[ \\psi \\left( \\frac{t}{\\sqrt{n}} \\right) \\right] ^n \\\\\n",
    "         & = \\left[  1 + \\frac{t^2}{2n} +  \\frac{t^3}{3!n^{3/2}} \\psi'''(0) + \\ldots \\right] ^n \\\\\n",
    "         & = \\left[  1 + \\frac{\\frac{t^2}{2} +  \\frac{t^3}{3!n^{1/2}} \\psi'''(0) + \\ldots}{n} \\right] ^n \\\\\n",
    "         & \\rightarrow e^{t^2/2}\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "which is the mgf of $N(0, 1)$.  The resolt follows from the previous theorem.  In the last step we used the fact that, if $a_n \\rightarrow a$, then\n",
    "\n",
    "$$ \\left( 1 + \\frac{a_n}{n} \\right) ^n \\rightarrow e^a $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.8 Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 6.8.1.** Let $X_1, \\dots, X_n$ be iid with finite mean $\\mu = \\mathbb{E}(X_i)$ and finite variance $\\sigma^2 = \\mathbb{V}(X_i)$.  Let $\\overline{X}_n$ be the sample mean and let $S_n^2$ be the sample variance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**(a)** Show that $\\mathbb{E}(S_n^2) = \\sigma^2$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solution**: \n",
    "\n",
    "$S_n^2$ is the sample variance, that is, $ S_n^2 = n^{-1} \\sum_{i=1}^n ( X_i - \\overline{X}_n)^2 $.  Therefore:\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\mathbb{E}[S_n^2] & = \\mathbb{E}\\left[ \\frac 1n \\sum_{i=1}^n \\left(X_i - \\frac 1n \\sum_{j=1}^n X_j \\right)^2 \\right] \\\\\n",
    "& = \\frac 1n \\sum_{i=1}^n \\mathbb{E}\\left[ X_i^2 - \\frac 2n X_i \\sum_{j=1}^n X_j + \\frac{1}{n^2} \\sum_{j=1}^n X_j \\sum_{k=1}^n X_k \\right] \\\\\n",
    "& = \\frac 1n \\sum_{i=1}^n \\left[ \\frac{n-2}{n} \\mathbb{E}[X_i^2] - \\frac{2}{n} \\sum_{j \\neq i} \\mathbb{E}[X_i X_j] + \\frac{1}{n^2} \\sum_{j=1}^n \\sum_{k \\neq j}^n \\mathbb{E}[X_j X_k] +\\frac{1}{n^2} \\sum_{j=1}^n \\mathbb{E}[X_j^2] \\right] \\\\\n",
    "& = \\frac 1n \\sum_{i=1}^n \\left[ \\frac{n-2}{n} (\\sigma^2+\\mu^2) - \\frac 2n (n-1) \\mu^2 + \\frac{1}{n^2} n (n-1) \\mu^2 + \\frac 1n (\\sigma^2+\\mu^2) \\right] \\\\\n",
    "& = \\frac{n-1}{n} \\sigma^2\n",
    "\\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**(b)** Show that $S_n^2 \\xrightarrow{\\text{P}} \\sigma^2$.  \n",
    "\n",
    "Hint: show that $S_n^2 = c_n n^{-1} \\sum_{i=1}^n X_i^2 - d_n \\overline{X}_n^2$ where $c_n \\rightarrow 1$ and $d_n \\rightarrow 1$.  Apply the law of large numbers to $n^{-1}\\sum_{i=1}^n X_i^2$ and to $\\overline{X}_n$.  Then use part (e) of Theorem 6.5."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solution**: \n",
    "\n",
    "We have:\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\overline{X}_n^2 & = \\frac{1}{n^2} \\left( \\sum_{i=1}^n X_i \\right)^2 \\\\\n",
    "  &= \\frac{1}{n^2} \\sum_{i=1}^n X_i^2 +  \\frac{1}{n^2} \\sum_{i=1}^n \\sum_{j=1, i \\neq j}^n X_i X_j\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "Isolating the sum of products, we have:\n",
    "\n",
    "$$  \\sum_{j=1, i \\neq j}^n X_i X_j = n^2 \\overline{X}_n^2 - \\sum_{i=1}^n X_i^2 $$\n",
    "\n",
    "Now, from the sample variance :\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "S_n^2 & = \\frac{1}{n} \\sum_{i=1}^n ( X_i - \\overline{X}_n)^2 \\\\\n",
    "  & = \\frac{1}{n} \\sum_{i=1}^n \\left( X_i^2 - 2 X_i \\overline{X}_n + \\overline{X}_n^2  \\right)\\\\\n",
    "  & = \\frac{1}{n} \\sum_{i=1}^n X_i^2 - \\frac{2}{n^2} \\sum_{i=1}^n \\sum_{j=1}^n X_i X_j + \\overline{X}_n^2 \\\\\n",
    "  & = \\left( \\frac{1}{n} - \\frac{2}{n^2}\\right) \\sum_{i=1}^n X_i^2 - \\frac{2}{n^2} \\sum_{i=1}^n \\sum_{j=1, i \\neq j}^n X_i X_j + \\overline{X}_n^2 \\\\\n",
    "  & = \\left( \\frac{1}{n} - \\frac{2}{n^2}\\right) \\sum_{i=1}^n X_i^2 - \\frac{2}{n^2} \\left( n^2 \\overline{X}_n^2 - \\sum_{i=1}^n X_i^2 \\right) + \\overline{X}_n^2 \\\\\n",
    " & = \\frac{1}{n} \\sum_{i=1}^n X_i^2 - \\overline{X}_n^2\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "We can use $c_n = 1$ and $d_n = 1$, as suggested in the hint.  Applying the law of large numbers,\n",
    "\n",
    "$$ n^{-1} \\sum_{i=1}^n X_i^2 \\xrightarrow{\\text{P}} n^{-1} \\sum_{i=1}^n \\mathbb{E}(X_i^2) = \\sigma^2 + \\mu^2$$\n",
    "\n",
    "$$ \\overline{X}_n \\xrightarrow{\\text{P}} n^{-1} \\sum_{i=1}^n \\mathbb{E}(X_i) = \\mu \\Rightarrow \\overline{X}_n^2 \\xrightarrow{\\text{P}} \\mu^2 $$\n",
    "\n",
    "Therefore, from theorem 6.5.e, $ S_n^2 = c_n n^{-1} \\sum_{i=1}^n X_i^2 - d_n \\overline{X}_n^2 \\xrightarrow{\\text{P}} \\sigma^2 + \\mu^2 - \\mu^2 = \\sigma^2$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 6.8.2.**  Let $X_1, X_2, \\dots, X_n$ be a sequence of random variables.  Show that $X_n \\xrightarrow{\\text{qm}} b$ if and only if \n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "\\lim_{n \\rightarrow \\infty} \\mathbb{E}(X_n) = b\n",
    "\\quad\\mathrm{and}\\quad \n",
    "\\lim_{n \\rightarrow \\infty} \\mathbb{V}(X_n) = 0\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solution**:\n",
    "\n",
    "$X_n \\xrightarrow{\\text{qm}} b$ id equivalent to:\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\mathbb{E}[(X_n - b)^2]           & \\rightarrow 0 \\\\\n",
    "\\mathbb{E}[X_n^2 - 2b X_n + b^2]  & \\rightarrow 0 \\\\\n",
    "\\mathbb{E}[X_n^2] - 2b \\mathbb{E}[X_n] + b^2 & \\rightarrow 0 \\\\\n",
    "\\mathbb{E}[X_n^2] - 2b \\mathbb{E}[X_n] + b^2 & \\rightarrow 0 \\\\\n",
    "\\mathbb{V}[X_n] + (\\mathbb{E}[X_n])^2 - 2b \\mathbb{E}[X_n] + b^2  & \\rightarrow 0\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "If $\\lim_{n \\rightarrow \\infty} \\mathbb{V}[X_n] = 0$ and $\\lim_{n \\rightarrow \\infty} \\mathbb{E}[X_n] = b$, then\n",
    "\n",
    "$$ \n",
    "\\begin{align}\n",
    "& \\lim_{n \\rightarrow \\infty} \\mathbb{E}[(X_n - b)^2] = \\\\\n",
    "& = \\lim_{n \\rightarrow \\infty} \\mathbb{V}[X_n] + (\\mathbb{E}[X_n])^2 - 2b \\mathbb{E}[X_n] + b^2 \\\\\n",
    "& = \\lim_{n \\rightarrow \\infty} \\mathbb{V}[X_n] + (\\lim_{n \\rightarrow \\infty} \\mathbb{E}[X_n])^2 - 2b \\lim_{n \\rightarrow \\infty} \\mathbb{E}[X_n] + b^2 \\\\\n",
    "&= 0 + b^2 - 2b^2 + b^2 \\\\\n",
    "&= 0\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "On the other direction, if $X_n \\xrightarrow{\\text{qm}} b$, then\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\lim_{n \\rightarrow \\infty} \\mathbb{V}[X_n] + (\\lim_{n \\rightarrow \\infty} \\mathbb{E}[X_n])^2 - 2b \\lim_{n \\rightarrow \\infty} \\mathbb{E}[X_n] + b^2 &= 0 \\\\\n",
    "\\lim_{n \\rightarrow \\infty} \\mathbb{V}[X_n] + (\\lim_{n \\rightarrow \\infty} \\mathbb{E}[X_n] - b)^2 &= 0 \\\\\n",
    "\\lim_{n \\rightarrow \\infty} \\mathbb{V}[X_n - b] + \\lim_{n \\rightarrow \\infty}  (\\mathbb{E}[X_n - b])^2 &= 0\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "Since both terms inside the limits are non-negative, the limits themselves are non-negative.  Two non-negative values add up to 0, so they must both be zero, and so we have:\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "\\lim_{n \\rightarrow \\infty} \\mathbb{E}(Y_n) = 0\n",
    "\\quad\\mathrm{and}\\quad \n",
    "\\lim_{n \\rightarrow \\infty} \\mathbb{V}(Y_n) = 0\n",
    "\\end{equation}\n",
    "$$\n",
    "\n",
    "or, equivalently,\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "\\lim_{n \\rightarrow \\infty} \\mathbb{E}(X_n) = b\n",
    "\\quad\\mathrm{and}\\quad \n",
    "\\lim_{n \\rightarrow \\infty} \\mathbb{V}(X_n) = 0\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 6.8.3**.  Let $X_1, X_2, \\dots, X_n$ be iid and let $\\mu = \\mathbb{E}(X_i)$.  Suppose that variance is finite.  Show that $\\overline{X}_n \\xrightarrow{\\text{qm}} \\mu$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solution**.\n",
    "\n",
    "Let $Y_i = X_i - \\mu$.  It has variance $\\sigma_Y = \\sigma$ and mean $\\mu_Y = 0$. We have:\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "& \\mathbb{E}[(\\overline{X}_n - \\mu)^2] = \\\\\n",
    "& = \\mathbb{E}\\left[\\left(\\frac{1}{n} \\sum_{i=1}^n (X_i - \\mu) \\right)^2\\right] \\\\\n",
    "& = \\frac{1}{n^2} \\mathbb{E} \\left[ \\left(\\sum_{i=1}^n Y_i \\right)^2 \\right] \\\\\n",
    "& = \\frac{1}{n^2} \\left( \\sum_{i=1}^n \\mathbb{E}[Y_i^2] - \\sum_{i=1}^n \\sum_{j=1, j \\neq i}^n \\mathbb{E}[Y_i Y_j] \\right) \\\\\n",
    "& = \\frac{1}{n} \\left( (\\sigma_Y^2 + \\mu_Y^2) - (n-1) \\mu_Y^2 \\right) \\\\\n",
    "& = \\frac{\\sigma}{n}\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "Therefore, $\\lim _{n \\rightarrow \\infty} \\mathbb{E}[(\\overline{X}_n - \\mu)^2] = \\lim _{n \\rightarrow \\infty} \\sigma / n = 0$, and so $\\overline{X}_n \\xrightarrow{\\text{qm}} \\mu$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 6.8.4**.  Let $X_1, X_2, \\dots$ be a sequence of random variables such that\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "\\mathbb{P}\\left(X_n = \\frac{1}{n}\\right) = 1 - \\frac{1}{n^2}\n",
    "\\quad\\mathrm{and}\\quad \n",
    "\\mathbb{P}\\left(X_n = n\\right) = \\frac{1}{n^2}\n",
    "\\end{equation}\n",
    "$$\n",
    "\n",
    "Does $X_n$ converge in probability?  Doex $X_n$ converge in quadratic mean?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solution**.\n",
    "\n",
    "For any distribution $X$, we have:\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "& \\mathbb{P}( |X_n - X| > \\epsilon ) = \\\\\n",
    "&= \\mathbb{P}\\left( |X_n - X| > \\epsilon \\;\\middle|\\; X_n = \\frac{1}{n} \\right)\\mathbb{P}\\left(X_n = \\frac{1}{n}\\right)\n",
    "  + \\mathbb{P}\\left( |X_n - X| > \\epsilon \\;\\middle|\\; X_n = n \\right)\\mathbb{P}\\left(X_n = n\\right) \\\\\n",
    "&= \\mathbb{P}\\left( \\middle|\\frac{1}{n} - X\\middle|\\; > \\epsilon \\right)\\left(1 - \\frac{1}{n^2} \\right)\n",
    "  + \\mathbb{P}\\left( |n - X| > \\epsilon \\right)\\frac{1}{n^2}\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "Looking at the limit as $n \\rightarrow \\infty$,\n",
    "\n",
    "$$ \n",
    "\\begin{align}\n",
    "& \\lim _{n \\rightarrow \\infty} \\mathbb{P}( |X_n - X| > \\epsilon ) = \\\\\n",
    "& = \\lim _{n \\rightarrow \\infty} \\mathbb{P}\\left( \\middle|\\frac{1}{n} - X\\middle|\\; > \\epsilon \\right)\\left(1 - \\frac{1}{n^2} \\right)\n",
    "  + \\lim _{n \\rightarrow \\infty} \\mathbb{P}\\left( |n - X| > \\epsilon \\right)\\frac{1}{n^2} \\\\\n",
    "& = \\lim _{n \\rightarrow \\infty} \\mathbb{P}\\left( |X| > \\epsilon \\right)\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "If we set $X = 0$, the limit above will be zero for any positive $\\epsilon$ -- so we have $X_n \\xrightarrow{\\text{P}} 0$.\n",
    "\n",
    "Now, for any quadratic mean potential convergence, we have:\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "& \\mathbb{E}\\left[(X_n - X)^2\\right] = \\\\\n",
    "& = \\mathbb{E}\\left[(X_n - X)^2 \\bigg| X_n = \\frac{1}{n} \\right] \\mathbb{P}\\left(X_n = \\frac{1}{n}\\right)\n",
    "   + \\mathbb{E}\\left[(X_n - X)^2 \\big| X_n = n \\right] \\mathbb{P}\\left(X_n = n\\right) \\\\\n",
    "& = \\mathbb{E}\\left[\\left(X - \\frac{1}{n}\\right)^2  \\right] \\left(1 - \\frac{1}{n^2}\\right)\n",
    "   + \\mathbb{E}\\left[(X - n)^2  \\right] \\frac{1}{n^2} \\\\\n",
    "& = \\mathbb{E}\\left[X^2 - 2Xn^{-1} + n^{-2} \\right] \\left(1 - \\frac{1}{n^2}\\right)\n",
    "   + \\mathbb{E}\\left[X^2 - 2Xn + n^2 \\right] \\frac{1}{n^2} \\\\\n",
    "& = \\mathbb{E}\\left[X^2\\right] + \\mathbb{E}\\left[X\\right] \\left(\\frac{-2}{n} \\left(1 - \\frac{1}{n^2} \\right) -\\frac{2}{n}\\right) + \\frac{1}{n^2} \\left(1 - \\frac{1}{n^2} \\right) + 1 \\\\\n",
    "& = \\mathbb{E}\\left[X^2\\right] - \\mathbb{E}\\left[X\\right] \\frac{2}{n} \\left( 2 + \\frac{1}{n^2} \\right) + \\frac{1}{n^2} \\left(1 - \\frac{1}{n^2} \\right) + 1\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "Taking the limit as $n \\rightarrow \\infty$,\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "& \\lim _{n \\rightarrow \\infty} \\mathbb{E}\\left[(X_n - X)^2\\right] = \\\\\n",
    "& = 1 + \\lim _{n \\rightarrow \\infty} \\mathbb{E}\\left[X^2\\right] \\\\\n",
    "& = 1 + \\mathbb{E}\\left[X^2\\right] \\\\\n",
    "& \\geq 1\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "so there is no distribution $X$ for which this value is 0, and so there is no quadratic mean convergence."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 6.8.5**. Let $X_1, \\dots, X_n \\sim \\text{Bernoulli}(p)$.  Prove that\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "\\frac{1}{n} \\sum_{i=1}^n X_i^2 \\xrightarrow{\\text{P}} p\n",
    "\\quad\\mathrm{and}\\quad \n",
    "\\frac{1}{n} \\sum_{i=1}^n X_i^2 \\xrightarrow{\\text{qm}} p\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solution**.\n",
    "\n",
    "Given that quadratic mean convergence implies probability convergence, we only need to prove the second proposition.\n",
    "\n",
    "Let $Y_i = X_i^2 - p$.  Then:\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\mathbb{E}[Y_i] & = \\\\\n",
    "& = \\mathbb{E}[X_i^2] - p \\\\\n",
    "& = \\mathbb{V}[X_i] + \\mathbb{E}[X_i]^2 - p \\\\\n",
    "& = p(1-p) + p^2 - p \\\\\n",
    "& = 0 \\\\\n",
    "\\mathbb{E}[Y_i^2] & = \\\\\n",
    "& = \\mathbb{V}[Y_i] + \\mathbb{E}[Y_i]^2 \\\\\n",
    "& = \\mathbb{V}[X_i^2 - p] + 0^2 \\\\\n",
    "& = \\mathbb{V}[X_i^2] + 0^2 \\\\\n",
    "& = \\mathbb{V}[X_i]\\\\\n",
    "& = p(1-p) \\\\\n",
    "\\mathbb{E}[Y_i Y_j] & = \\text{(for independent variables)}\\\\\n",
    "& = \\mathbb{E}[Y_i] \\mathbb{E}[Y_j] \\\\\n",
    "& = 0\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "& \\mathbb{E}\\left[\\left(\\left(\\frac{1}{n} \\sum_{i=1}^n X_i^2\\right) - p\\right)^2\\right] = \\\\\n",
    "& = \\mathbb{E}\\left[\\left(\\frac{1}{n} \\sum_{i=1}^n \\left(X_i^2 - p\\right)\\right)^2\\right] \\\\\n",
    "& = \\frac{1}{n^2} \\mathbb{E}\\left[\\left(\\sum_{i=1}^n Y_i\\right)^2\\right] \\\\\n",
    "& = \\frac{1}{n^2} \\mathbb{E}\\left[\\sum_{i=1}^n Y_i^2 - \\sum_{i=1}^n \\sum_{j=1, j \\neq i}^n Y_i Y_j\\right] \\\\\n",
    "& = \\frac{1}{n^2} \\left( \\sum_{i=1}^n \\mathbb{E}\\left[Y_i^2\\right] - \\sum_{i=1}^n \\sum_{j=1, j \\neq i}^n \\mathbb{E}\\left[ Y_i Y_j \\right] \\right) \\\\\n",
    "& = \\frac{p(1-p)}{n}\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "So, as $n \\rightarrow \\infty$, this expectation goes to 0, and we have quadratic mean convergence."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 6.8.6**.  Suppose that the height of men has mean 68 inches and standard deviation 4 inches.  We draw 100 men at random.  Find (approximately) the probability that the average height of men in our sample will be at least 68 inches."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solution**.\n",
    "\n",
    "We assume all men's heights are measurements from iid variables $X_i$ with mean $\\mu = 68$ and variance $\\sigma^2 = 16$.\n",
    "\n",
    "We need to approximate $\\mathbb{P}(\\overline{X}_{100} > \\mu)$.  But by the central limit theorem, \n",
    "\n",
    "$$ \\overline{X}_n \\approx N\\left(\\mu, \\frac{\\sigma^2}{n}\\right) $$\n",
    "\n",
    "so this probability will be approximately $$\\mathbb{P}\\left(\\frac{\\sqrt{n}(\\overline{X}_n - \\mu)}{\\sigma} \\geq \\frac{\\sqrt{n}(\\mu- \\mu)}{\\sigma}\\right) = \\mathbb{P}\\left(\\frac{\\sqrt{n}(\\overline{X}_n - \\mu)}{\\sigma} \\geq 0 \\right) = P(Z \\geq 0) = \\frac{1}{2}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 6.8.7**.  Let $\\lambda_n = 1/n$ for $n = 1, 2, \\dots$.  Let $X_n \\sim \\text{Poisson}(\\lambda_n)$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**(a)** Show that $X_n \\xrightarrow{\\text{P}} 0$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solution**.\n",
    "\n",
    "$$\n",
    "\\mathbb{E}(X_n^2) = \\mathbb{V}(X_n) + \\mathbb{E}(X_n)^2\n",
    "= \\lambda_n^2 + \\lambda_n^2 = 2 \\lambda_n^2 = 2/n^2\n",
    "$$\n",
    "\n",
    "This quantity goes to zero as $n \\rightarrow \\infty$, so we have $X_n \\xrightarrow{\\text{qm}} 0$, which implies $X_n \\xrightarrow{\\text{P}} 0$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**(b)** Let $Y_n = n X_n$.  Show that $Y_n \\xrightarrow{\\text{P}} 0$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solution**.\n",
    "\n",
    "$$\n",
    "\\mathbb{E}(Y_n^2) = \\mathbb{V}(Y_n) + \\mathbb{E}(Y_n)^2\n",
    "= n^2 \\lambda_n^2 + n^2\\lambda_n^2 = 2 n^2 \\lambda_n^2 = 2\n",
    "$$\n",
    "\n",
    "so we *don't* have a straightforward quadratic mean convergence on $Y_n$.\n",
    "\n",
    "We *don't* have a straightforward $L_1$ convergence either:\n",
    "\n",
    "$$\\mathbb{E}(|Y_n|) = \\mathbb{E}(Y_n) = n \\lambda_n = 1$$\n",
    "\n",
    "However, we can show that $Y_n \\leadsto 0$:\n",
    "\n",
    "$$\\lim _{n \\rightarrow \\infty} F_{Y_n}(t) = \\lim _{n \\rightarrow \\infty} F_{Y_1}(t / n) = \\lim _{n \\rightarrow \\infty} F_{Y_1}(t / n) = 0 $$\n",
    "\n",
    "as, when $n \\rightarrow \\infty$, the portion of the CDF in the positive neighborhood of 0 shrinks to $F_{Y_1}(0) = 0$.\n",
    "\n",
    "We also have a point mass distribution on our target distribution $Y_\\infty = 0$: probability of 1 in point 0, and 0 everywhere else.\n",
    "\n",
    "Therefore, from theorem 6.4 item c, we have $Y_n \\xrightarrow{\\text{P}} 0$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 6.8.8**.  Suppose we have a computer program consisting of $n = 100$ pages of code.  Let $X_i$ be the number of errors in the $i$-th page of code.  Suppose that the $X_i$'s are Poisson with mean 1 and that they are independent.  Let $Y = \\sum_{i=1}^n X_i$ be the total number of errors.  Use the central limit theorem to approximate $\\mathbb{P}(Y < 90)$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solution**.  We have $Y = n \\overline{X}_n$, the total being $n$ times the sample mean.  We need to approximate:\n",
    "\n",
    "We need to approximate $\\mathbb{P}(\\overline{X}_{100} < 0.9)$.  But by the central limit theorem, \n",
    "\n",
    "$$ \\overline{X}_n \\approx N\\left(\\mu, \\frac{\\sigma^2}{n}\\right) $$\n",
    "\n",
    "so this probability will be approximately \n",
    "$$\n",
    "\\mathbb{P}\\left(\\frac{\\sqrt{n}(\\overline{X}_n - \\mu)}{\\sigma} < \\frac{\\sqrt{100}(0.9 - 1)}{0.1}\\right) \n",
    "= \\mathbb{P}\\left(\\frac{\\sqrt{n}(\\overline{X}_n - \\mu)}{\\sigma} < -10 \\right) \n",
    "= P(Z < -10)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 6.8.9**.   Suppose that $\\mathbb{P}(X = 1) = \\mathbb{P}(X = -1) = 1/2$.  Define\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "  X_n =\n",
    "    \\begin{cases}\n",
    "      X   & \\text{with probability } 1 - \\frac{1}{n}\\\\\n",
    "      e^n & \\text{with probability } \\frac{1}{n}\n",
    "    \\end{cases}       \n",
    "\\end{equation}\n",
    "$$\n",
    "\n",
    "Does $X_n$ converge to $X$ in probability? Does $X_n$ converge to $X$ in distribution?  Does $\\mathbb{E}(X - X_n)^2$ converge to 0?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solution**.\n",
    "\n",
    "For any potential quadratic mean convergence, we'd have:\n",
    "\n",
    "$$ \n",
    "\\begin{align}\n",
    "&\\mathbb{E}(X - X_n)^2 = \\\\\n",
    "& = \\left(\\mathbb{E}(X - X_n \\middle| X_n = X)\\mathbb{P}(X_n = X) + \\mathbb{E}(X - X_n \\middle| X_n = e^n)\\mathbb{P}(X_n = e^n) \\right)^2 \\\\\n",
    "& = \\left(\\mathbb{E}(0)\\left(1 - \\frac{1}{n} \\right) + \\mathbb{E}(X - e^n)\\frac{1}{n} \\right)^2 \\\\\n",
    "& = \\frac{1}{n^2} \\mathbb{E}(X - e^n)^2 \\\\\n",
    "& = \\frac{1}{n^2} \\left( \\mathbb{E}(X) - e^n  \\right)^2 \\\\\n",
    "& = \\frac{e^{2n}}{n^2}\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "which does not converge to 0, so we do not have quadratic mean convergence.\n",
    "\n",
    "For any potential distribution convergence, $X_n$ has a point mass distribution, and we can write its CDF $F_{X_n}$ explicitly as:\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "  F_{X_n}(t) =\n",
    "    \\begin{cases}\n",
    "      0   & \\text{if } t < -1 \\\\\n",
    "      \\frac{1}{2} \\left(1 - \\frac{1}{n}\\right) & \\text{if } -1 \\leq t < 1 \\\\\n",
    "      1 - \\frac{1}{n} & \\text{if } 1 \\leq t < e^n \\\\\n",
    "      1 & \\text{if } e^n \\leq t\n",
    "    \\end{cases}       \n",
    "\\end{equation}\n",
    "$$\n",
    "\n",
    "On the other hand, the CDF $F_X$ of the target distribution $X$ is:\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "  F_X(t) =\n",
    "    \\begin{cases}\n",
    "      0   & \\text{if } t < -1 \\\\\n",
    "      \\frac{1}{2} & \\text{if } -1 \\leq t < 1 \\\\\n",
    "      1 & \\text{if } 1 \\leq t\n",
    "    \\end{cases}       \n",
    "\\end{equation}\n",
    "$$\n",
    "\n",
    "We then have:\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "  F_X(t) - F_{X_n}(t) =\n",
    "    \\begin{cases}\n",
    "      0   & \\text{if } t < -1 \\\\\n",
    "      \\frac{1}{2n} & \\text{if } -1 \\leq t < 1 \\\\\n",
    "      \\frac{1}{n} & \\text{if } 1 \\leq t < e^n \\\\\n",
    "      0 & \\text{if } e^n \\leq t\n",
    "    \\end{cases}       \n",
    "\\end{equation}\n",
    "$$\n",
    "\n",
    "so $0 \\leq F_X(t) - F_{X_n}(t) \\leq 1/n$, which goes to 0 as $n \\rightarrow \\infty$.  Therefore $\\lim _{n \\rightarrow \\infty} F_{X_n}(t) = F_X(t)$, or $X_n \\leadsto X$.\n",
    "\n",
    "Distribution convergence implies probability convergence, so we also have probability convergence, $X_n \\xrightarrow{\\text{P}} X$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 6.8.10**.  Let $Z \\sim N(0, 1)$. Let $t > 0$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**(a)** Show that, for any $k > 0$,\n",
    "\n",
    "$$\\mathbb{P}(|Z| > t) \\leq \\frac{\\mathbb{E}|Z|^k}{t^k}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solution**.\n",
    "\n",
    "We have:\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "&\\mathbb{E}|Z|^k = \\\\\n",
    "&= \\int _{-\\infty}^{\\infty} |z|^{k+1}\\left( \\frac{1}{\\sqrt{2\\pi}} e^{-z^2/2} \\right) dz \\\\\n",
    "&= \\int _{-\\infty}^{0} (-z)^{k+1}\\left( \\frac{1}{\\sqrt{2\\pi}} e^{-z^2/2} \\right) dz\n",
    "  + \\int _{0}^{\\infty} z^{k+1}\\left( \\frac{1}{\\sqrt{2\\pi}} e^{-z^2/2} \\right) dz \\\\\n",
    "&=  \\left\\{ \\frac{2}{\\pi} \\right\\}^{1/2} \\int _{0}^{\\infty} z^k \\left( z e^{-z^2/2} \\right) dz\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "For t > 0,\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "&\\mathbb{P}(|Z| > t) = \\\\\n",
    "&= 2 \\int _t^{\\infty} z\\left( \\frac{1}{\\sqrt{2\\pi}} e^{-z^2/2} \\right) dz \\\\\n",
    "&= \\left\\{ \\frac{2}{\\pi} \\right\\}^{1/2} \\int _t^{\\infty} z e^{-z^2/2} dz\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "Now we need to prove:\n",
    "\n",
    "$$\\int _t^{\\infty} z e^{-z^2/2} dz \\leq \\frac{1}{t^k}\\int _{0}^{\\infty} z^k \\left( z e^{-z^2/2} \\right) dz $$\n",
    "\n",
    "As the integrands are always positive, we can prove the stronger statement that, for $k \\geq 0$:\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\int _t^{\\infty} z e^{-z^2/2} dz & \\leq \\frac{1}{t^k}\\int _{t}^{\\infty} z^k \\left( z e^{-z^2/2} \\right) dz  \\\\\n",
    "t^k \\int _t^{\\infty} z e^{-z^2/2} dz & \\leq \\int _{t}^{\\infty} z^k \\left( z e^{-z^2/2} \\right) dz  \\\\\n",
    "0 & \\leq \\int _{t}^{\\infty} (z^k - t^k) \\left( z e^{-z^2/2} \\right) dz  \\\\\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "But that's true, since $(z^k - t^k) (z e^{-z^2/2}) \\geq 0$ whenever $z \\geq t$.  So the given statement follows."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**(b) (Mill's inequality)**  Show that\n",
    "\n",
    "$$\\mathbb{P}(|Z| > t) \\leq \\left\\{ \\frac{2}{\\pi} \\right\\}^{1/2} \\frac{e^{-t^2/2}}{t}$$\n",
    "\n",
    "Hint.  Note that $\\mathbb{P}(|Z| > t) = 2\\mathbb{P}(Z > t)$.  Now write out what $\\mathbb{P}(Z > t)$ means and note that $x/t > 1$ whenever $x > t$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solution**.\n",
    "\n",
    "The stronger result we proved in (a) was, for $k \\geq 0$,\n",
    "\n",
    "$$\n",
    "\\mathbb{P}(|Z| > t) = \\left\\{ \\frac{2}{\\pi} \\right\\}^{1/2}  \\int _t^{\\infty} z e^{-z^2/2} dz \\leq \\left\\{ \\frac{2}{\\pi} \\right\\}^{1/2} \\frac{1}{t^k}\\int _{t}^{\\infty} z^k \\left( z e^{-z^2/2} \\right) dz\n",
    "$$\n",
    "\n",
    "If we use $k = 0$, we get:\n",
    "\n",
    "$$\n",
    "\\mathbb{P}(|Z| > t) \\leq \\left\\{ \\frac{2}{\\pi} \\right\\}^{1/2} \\frac{1}{t}\\int _{t}^{\\infty} z e^{-z^2/2} dz = \\left\\{ \\frac{2}{\\pi} \\right\\}^{1/2} \\frac{e^{-t^2/2}}{t}\n",
    "$$\n",
    "\n",
    "which is the desired result."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 6.8.11**. Suppose that $X_n \\sim N(0, 1/n)$ and let $X$ be a random variable with distribution $F(x) = 0$ if $x < 0$ and $F(x) = 1$ if $x \\geq 0$.  Does $X_n$ converge to $X$ in probability?  Does $X_n$ converge to $X$ in distribution?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solution**.\n",
    "\n",
    "We do not have convergence in distribution: $ F_{X_n}(0) = 1/2$ for any $n$ (as the normal distribution is symmetric around its mean), so $\\lim _{n \\rightarrow \\infty} F_{X_n}(0) = 1/2 \\neq F_X(0) = 1$.\n",
    "\n",
    "We do have convergence in probability: for every $\\epsilon > 0$,\n",
    "\n",
    "$$ \\mathbb{P}(|X - X_n| > \\epsilon) = \\mathbb{P}(|X_n| > \\epsilon) = 2 \\mathbb{P}(X_n > \\epsilon) = 2 (1 - F_{X_n}(\\epsilon))$$\n",
    "\n",
    "so\n",
    "\n",
    "$$ \\lim _{n \\rightarrow \\infty} \\mathbb{P}(|X - X_n| > \\epsilon)  = 2 (1 - \\lim _{n \\rightarrow \\infty} F_{X_n}(\\epsilon)) =  2 (1 - \\lim _{n \\rightarrow \\infty} F_{X_1}(n \\epsilon)) = 2 (1 - 1) = 0 $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 6.8.12**.  Let $X, X_1, X_2, X_3, \\cdots$ be random variables that are positive and integer valued.  Show that $X_n \\leadsto X$ if and only if\n",
    "\n",
    "$$ \\lim _{n \\rightarrow \\infty} \\mathbb{P}(X_n = k) = \\mathbb{P}(X = k) $$\n",
    "\n",
    "for every integer $k$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solution**.\n",
    "\n",
    "If $X_n \\leadsto X$, then $\\lim _{n \\rightarrow \\infty} F_{X_n}(k) = F_X(k)$ for every integer $k$.  But since the variables are positive and integer valued,\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "\\mathbb{P}(X_n = k) = F_{X_n}(k) - F_{X_n}(k - 1)\n",
    "\\quad\\mathrm{and}\\quad \n",
    "\\mathbb{P}(X = k) = F_X(k) - F_X(k - 1)\n",
    "\\end{equation}\n",
    "$$\n",
    "\n",
    "Therefore,\n",
    "\n",
    "$$\n",
    "\\lim _{n \\rightarrow \\infty} \\mathbb{P}(X_n = k) \n",
    "= \\lim _{n \\rightarrow \\infty} F_{X_n}(k) - F_{X_n}(k - 1)\n",
    "= \\lim _{n \\rightarrow \\infty} F_{X_n}(k) - \\lim _{n \\rightarrow \\infty} F_{X_n}(k - 1)\n",
    "= F_X(k) - F_X(k - 1)\n",
    "= \\mathbb{P}(X = k)\n",
    "$$\n",
    "\n",
    "On the other direction, if  $ \\lim _{n \\rightarrow \\infty} \\mathbb{P}(X_n = k) = \\mathbb{P}(X = k) $, then\n",
    "\n",
    "$$ \\lim _{n \\rightarrow \\infty}\\left( F_{X_n}(k) - F_{X_n}(k - 1) \\right) = F_X(k) - F_X(k - 1) $$\n",
    "\n",
    "But the variables are positive and integer valued, so $F_{X_n}(k) = F_X(k) = 0$ for $k \\leq 0$.  We can then show that $\\lim _{n \\rightarrow \\infty} F_{X_n}(k) = F_X(k)$ for every integer valued $k$ by induction in $k$:\n",
    "\n",
    "$$ \\lim _{n \\rightarrow \\infty} \\left( F_{X_n}(k) - F_{X_n}(k - 1) \\right) = \\left( \\lim _{n \\rightarrow \\infty} F_{X_n}(k) \\right) - F_X(k - 1) = F_X(k) - F_X(k - 1)$$\n",
    "$$ \\Rightarrow \\lim _{n \\rightarrow \\infty} F_{X_n}(k) = F_X(k)$$\n",
    "\n",
    "Since the result holds for every integer variable $k$ and the random variables can only take integer values, it must hold for all values, therefore $X_n \\leadsto X$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 6.8.13**. Let $Z_1, Z_2, \\dots$ be iid random variables with density $f$.  Suppose that $\\mathbb{P}(Z_i > 0) = 1$ and that $\\lambda = \\lim _{x \\downarrow 0} f(x) > 0$.  Let\n",
    "\n",
    "$$ X_n = n \\min \\{ Z_1, \\dots, Z_n \\} $$\n",
    "\n",
    "Show that $X_n \\leadsto Z$ where $Z$ has and exponential distribution with mean $1 / \\lambda$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solution**.\n",
    "\n",
    "Since $\\mathbb{P}(Z_i > 0) = 1$, the cumulative density functions $F$ assume value 0 for values up until 0 inclusive.\n",
    "\n",
    "We have:\n",
    "\n",
    "$$\\mathbb{P}(X_n > x) = \\mathbb{P}(n \\min\\{Z_1, \\dots, Z_n\\} > x) = \\prod _{i=1}^n \\mathbb{P}(Z_i > x/n)$$\n",
    "\n",
    "Expanding the probability based on its density function,\n",
    "\n",
    "$$\n",
    "\\mathbb{P}(X_n > x)\n",
    "= \\prod _{i=1}^n \\mathbb{P}(Z_i > x/n)\n",
    "= \\prod _{i=1}^n \\int _0^{x/n} f(u) du\n",
    "= \\left(F\\left(\\frac{x}{n}\\right) \\right)^n\n",
    "= \\left(F(0) + F'(0)\\frac{x}{n} + F''(0)\\left(\\frac{x}{n}\\right)^2\\frac{1}{2!} + \\cdots \\right)^n\n",
    "$$\n",
    "\n",
    "Taking the limit as $n \\rightarrow \\infty$,\n",
    "\n",
    "$$\n",
    "\\lim _{n \\rightarrow \\infty}\\mathbb{P}(X_n > x)\n",
    "= \\lim _{n \\rightarrow \\infty} \\left(F(0) + F'(0)\\frac{x}{n} + F''(0)\\left(\\frac{x}{n}\\right)^2\\frac{1}{2!} + \\cdots \\right)^n\n",
    "= \\lim _{n \\rightarrow \\infty} \\left(F(0) + F'(0)\\frac{x}{n} \\right)^n\n",
    "= \\lim _{n \\rightarrow \\infty} \\left(0 + \\lambda\\frac{x}{n} \\right)^n\n",
    "= e^{-\\lambda x}\n",
    "$$\n",
    "\n",
    "On the other hand, $\\mathbb{P}(Z > x) = e^{-\\lambda x}$, so the limit of the CDF complements are the same, and so $X_n \\leadsto Z$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 6.8.14**.  Let $X_1, \\dots, X_n \\sim \\text{Uniform}(0, 1)$.  Let $Y_n = \\overline{X}_n^2$.  Find the limiting distribution of $Y_n$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solution**.\n",
    "\n",
    "Let $F_K(x)$ denote the CDF of random variable $K$.\n",
    "\n",
    "The sample mean $\\overline{X}_n$ has a limiting distribution of $X = 1/2$, by the (strong) law of large numbers.\n",
    "\n",
    "Then, for $x \\geq 0$,\n",
    "\n",
    "$$\\mathbb{P}(Y_n > x) = \\mathbb{P}(\\overline{X}_n^2 > x) = \\mathbb{P}(\\overline{X}_n > x^{1/2})$$\n",
    "\n",
    "$$F_{Y_n}(x) = F_{\\overline{X}_n}(x^{1/2}) $$\n",
    "\n",
    "Since $\\overline{X}_n \\leadsto 1/2$,\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\lim _{n \\rightarrow \\infty} F_{\\overline{X}_n}(x) & = F_{1/2}(x) \\\\\n",
    "\\lim _{n \\rightarrow \\infty} F_{\\overline{X}_n}(x^{1/2}) & = F_{1/2}(x^{1/2}) \\\\\n",
    "\\lim _{n \\rightarrow \\infty} F_{Y}(x) & = F_{1/2}(x^{1/2})\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "Therefore, $Y \\leadsto Z$, where $F_Z(x) = F_{1/2}(x^{1/2})$, that is,\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "  F_Z(t) =\n",
    "    \\begin{cases}\n",
    "      0   & \\text{if } t^{1/2} < 1/2 \\\\\n",
    "      1 & \\text{otherwise}\n",
    "    \\end{cases}   \n",
    "  = \\begin{cases}\n",
    "      0   & \\text{if } t < 1/4 \\\\\n",
    "      1 & \\text{otherwise}\n",
    "    \\end{cases} \n",
    "\\end{equation}\n",
    "$$\n",
    "\n",
    "so $Z$ assumes the constant value of $1/4$, and $Y \\leadsto 1/4$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 6.8.15**.  Let\n",
    "\n",
    "$$ \n",
    "\\begin{pmatrix} X_{11} \\\\ X_{21} \\end{pmatrix}, \\;\n",
    "\\begin{pmatrix} X_{12} \\\\ X_{22} \\end{pmatrix}, \\;\n",
    "\\cdots, \\;\n",
    "\\begin{pmatrix} X_{1n} \\\\ X_{1n} \\end{pmatrix}\n",
    "$$\n",
    "\n",
    "be iid random vectors with mean $\\mu = (\\mu_1, \\mu_2)$ and variance $\\Sigma$.  \n",
    "\n",
    "Let\n",
    "\n",
    "$$\n",
    "\\overline{X}_1 = \\frac{1}{n}\\sum_{i=1}^n X_{1i}, \\; \\; \\;\n",
    "\\overline{X}_2 = \\frac{1}{n}\\sum_{i=1}^n X_{2i}\n",
    "$$\n",
    "\n",
    "and define $Y_n = \\overline{X}_1 \\big/ \\overline{X}_2$.  Find the limiting distribution of $Y_n$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solution**.\n",
    "\n",
    "Let \n",
    "$$\\overline{X} = n^{-1} \\sum_{i=1}^n \\begin{pmatrix} X_{1i} \\\\ X_{2i} \\end{pmatrix} = \\begin{pmatrix} \\overline{X}_1 \\\\ \\overline{X}_2 \\end{pmatrix}$$.  \n",
    "\n",
    "By the multivariate central limit theorem,\n",
    "\n",
    "$$\n",
    "\\sqrt{n}(\\overline{X} - \\mu) \\leadsto N(0, \\Sigma)\n",
    "$$\n",
    "\n",
    "Define $g: \\mathbb{R} \\times \\mathbb{R}_{\\neq 0} \\rightarrow \\mathbb{R}$ as:\n",
    "\n",
    "$$\n",
    "g \\begin{pmatrix} y_1 \\\\ y_2 \\end{pmatrix} = y_1 / y_2\n",
    "$$\n",
    "\n",
    "Then, $Y_n = g(\\overline{X_n})$ in every scenario where $Y_n$ is defined.\n",
    "\n",
    "Applying the multivariate delta method,\n",
    "\n",
    "$$ \\nabla g \n",
    "= \\begin{pmatrix} \\frac{\\partial g}{\\partial y_1} \\\\ \\frac{\\partial g}{\\partial y_2} \\end{pmatrix} \n",
    "= \\begin{pmatrix} \\frac{1}{y_2} \\\\ -\\frac{y_1}{y_2^2} \\end{pmatrix} \n",
    "$$\n",
    "\n",
    "Then,\n",
    "\n",
    "$$ \\nabla_\\mu \n",
    "= \\begin{pmatrix} \\frac{1}{\\mu_2} \\\\ -\\frac{\\mu_1}{\\mu_2^2} \\end{pmatrix} \n",
    "$$\n",
    "\n",
    "and so\n",
    "\n",
    "$$ \n",
    "\\begin{align}\n",
    "\\sqrt{n}(Y_n - g(\\mu)) & \\leadsto N(0, \\nabla_\\mu^T \\Sigma \\nabla_\\mu)  \\\\\n",
    "Y_n & \\leadsto N(\\mu_1 / \\mu_2, n^{-1/2} \\nabla_\\mu^T \\Sigma \\nabla_\\mu) \n",
    "\\end{align}\n",
    "$$"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}