线性回归/20. 鲁棒学习.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "前面我们学习了通过最小二乘法求解回归模型，既可以求解线性回归，也可以扩展到非线性回归。最小二乘法是机器学习中的最基础的算法，这种求解损失函数最小值的思路可以延伸到更多的机器学习算法中，包括分类和聚类问题。\n",
    "\n",
    "介绍损失函数时我们提到了两种损失函数，一种是 **绝对损失（Absolute Loss）**，一种是 **平方损失（Squared Loss）**。\n",
    "\n",
    "绝对损失函数：$$ loss = |y - \\hat{y}| $$\n",
    "\n",
    "平方损失函数：$$ loss = (y - \\hat{y})^2 $$\n",
    "\n",
    "绝对损失函数因为不是处处可导，计算麻烦，所以我们选取了平方损失，最小二乘法就是求解平方损失函数的最小值。但是平方损失有一个很严重的问题，它对异常值非常敏感，在前面的例子中，我们使用的训练数据基本上都是正常的，也就是和模型大致是吻合的。如果我们在训练数据中加入异常值，这在实际场景中是经常遇到的，那么采用前面的最小二乘法，得到的模型会是什么样的呢？"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAErFJREFUeJzt3XGMnVWZx/HvY1t1wuoOaCF0WgKuTdEsgeIsW9PEKBgLaGyXSBbjSkOadDdBwyabarv/7Jq4AdPsoiSGbFfU4qpI2FIaJdSmhexfolOLVCgNlWXpzFRalWHXZcRSn/3jnoGhnTL3ztzpnbnn+0lu3vc999yZ5+SU333nvO+9RGYiSarDmzpdgCTpzDH0Jakihr4kVcTQl6SKGPqSVBFDX5Iq0lToR0RvRNwXEU9FxIGIeH9EnBMRuyLi6bI9u/SNiLgjIg5FxOMRcfnMDkGS1Kxmz/S/AjyUmRcDlwIHgI3A7sxcCuwuxwDXAEvLYz1wZ1srliRNWUz24ayIeDvwM+BdOa5zRBwEPpiZRyLifOCRzFwWEf9a9r97cr8ZG4UkqSnzm+jzLuAY8I2IuBTYC9wCnDcW5CX4zy39+4DD414/WNpeF/oRsZ7GXwKcddZZ77v44ounMw5Jqs7evXt/lZkLW3lNM6E/H7gc+GxmPhoRX+G1pZyJxARtp/w5kZlbgC0A/f39OTAw0EQpkqQxEfHfrb6mmTX9QWAwMx8tx/fReBN4vizrULZHx/VfMu71i4HhVguTJLXfpKGfmb8EDkfEstJ0FfAksANYW9rWAg+U/R3AjeUunhXAi67nS9Ls0MzyDsBngW9HxJuBZ4CbaLxh3BsR64DngOtL3weBa4FDwEulryRpFmgq9DPzMaB/gqeumqBvAjdPsy5J0gzwE7mSVBFDX5IqYuhLUkUMfUmqiKEvSRUx9CWpIoa+JFXE0Jekihj6klQRQ1+SKmLoS1JFDH1JqoihL0kVMfQlqSKGviRVxNCXpIoY+pJUEUNfkipi6EtSRQx9SaqIoS9JFTH0Jakihr4kVcTQl6SKGPqSVBFDX5IqYuhLUkWaCv2IeDYi9kfEYxExUNrOiYhdEfF02Z5d2iMi7oiIQxHxeERcPpMDkCQ1r5Uz/Q9l5mWZ2V+ONwK7M3MpsLscA1wDLC2P9cCd7SpWkjQ901neWQ1sLftbgTXj2u/Ohh8BvRFx/jR+jySpTZoN/QR+GBF7I2J9aTsvM48AlO25pb0PODzutYOlTZLUYfOb7LcyM4cj4lxgV0Q89QZ9Y4K2PKVT481jPcAFF1zQZBmSpOlo6kw/M4fL9ihwP3AF8PzYsk3ZHi3dB4El416+GBie4Gduycz+zOxfuHDh1EcgSWrapKEfEWdFxNvG9oGPAD8HdgBrS7e1wANlfwdwY7mLZwXw4tgykCSps5pZ3jkPuD8ixvp/JzMfioifAPdGxDrgOeD60v9B4FrgEPAScFPbq5YkTcmkoZ+ZzwCXTtD+a+CqCdoTuLkt1UmS2spP5EpSRQx9SaqIoS9JFTH0Jakihr4kVcTQl6SKGPqSVBFDX5IqYuhLUkUMfUmqiKEvSRUx9CWpIoa+JFXE0Jekihj6klQRQ1+SKmLoS1JFDH1JqoihL0kVMfQlqSKGviRVZH6nC5Ckk23fN8TmnQcZHhllUW8PG1YtY83yvk6X1RUMfUmzyvZ9Q2zatp/R4ycAGBoZZdO2/QAGfxu4vCNpVtm88+CrgT9m9PgJNu882KGKuouhL2lWGR4ZbaldrTH0Jc0qi3p7WmpXawx9SbPKhlXL6Fkw73VtPQvmsWHVsg5V1F28kCtpVhm7WOvdOzOj6dCPiHnAADCUmR+LiIuAe4BzgJ8Cn87M30fEW4C7gfcBvwb+MjOfbXvlkqZkLtwOuWZ536yrqVu0srxzC3Bg3PGXgNszcynwArCutK8DXsjMdwO3l36SZoGx2yGHRkZJXrsdcvu+oU6XpjOkqdCPiMXAR4GvleMArgTuK122AmvK/upyTHn+qtJfUod5O6SaPdP/MvA54A/l+B3ASGa+Uo4HgbG/xfqAwwDl+RdL/9eJiPURMRARA8eOHZti+ZJa4e2QmjT0I+JjwNHM3Du+eYKu2cRzrzVkbsnM/szsX7hwYVPFSpoeb4dUM2f6K4GPR8SzNC7cXknjzL83IsYuBC8Ghsv+ILAEoDz/x8Bv2lizpCnydkhNGvqZuSkzF2fmhcANwJ7M/BTwMPCJ0m0t8EDZ31GOKc/vycxTzvQlnXlrlvdx63WX0NfbQwB9vT3cet0l3ilTkencp/954J6I+CKwD7irtN8FfCsiDtE4w79heiVKaidvh6xbS6GfmY8Aj5T9Z4ArJujzO+D6NtQmSWozv4ZBkipi6EtSRQx9SaqIoS9JFTH0Jakihr4kVcTQl6SKGPqSVBFDX5IqYuhLUkUMfUmqiKEvSRUx9CWpIoa+JFXE0Jekihj6klQRQ1+SKmLoS1JFDH1JqoihL0kVMfQlqSKGviRVxNCXpIrM73QB0myxfd8Qm3ceZHhklEW9PWxYtYw1y/s6XZbUVoa+RCPwN23bz+jxEwAMjYyyadt+AINfXcXlHQnYvPPgq4E/ZvT4CTbvPNihiqSZYehLwPDIaEvt0lw1aehHxFsj4scR8bOIeCIivlDaL4qIRyPi6Yj4XkS8ubS/pRwfKs9fOLNDkKZvUW9PS+3SXNXMmf7LwJWZeSlwGXB1RKwAvgTcnplLgReAdaX/OuCFzHw3cHvpJ82o7fuGWHnbHi7a+ANW3raH7fuGWnr9hlXL6Fkw73VtPQvmsWHVsnaWKXXcpKGfDb8thwvKI4ErgftK+1ZgTdlfXY4pz18VEdG2iqWTjF2EHRoZJXntImwrwb9meR+3XncJfb09BNDX28Ot113iRVx1nabu3omIecBe4N3AV4FfACOZ+UrpMgiM/dfRBxwGyMxXIuJF4B3Ar076meuB9QAXXHDB9Eahqr3RRdhWQnvN8j5DXl2vqdDPzBPAZRHRC9wPvGeibmU70Vl9ntKQuQXYAtDf33/K86pbK/fMexFWal5Ld+9k5gjwCLAC6I2IsTeNxcBw2R8ElgCU5/8Y+E07ilUdWl2u8SKs1Lxm7t5ZWM7wiYge4MPAAeBh4BOl21rggbK/oxxTnt+TmZ7Jq2mt3jPvRVipec0s75wPbC3r+m8C7s3M70fEk8A9EfFFYB9wV+l/F/CtiDhE4wz/hhmoW12s1eWasWUfv0JBmtykoZ+ZjwPLJ2h/BrhigvbfAde3pTpVaVFvD0MTBPwbLdd4EVZqjp/I1azjco00c/zCNc06LtdIM8fQ16zkco00M1zekaSKGPqSVBFDX5IqYuhLUkUMfUmqiKEvSRUx9CWpIoa+JFXE0Jekihj6klQRQ1+SKmLoS1JFDH1JqoihL0kVMfQlqSKGviRVxNCXpIoY+pJUEUNfkipi6EtSRQx9SaqIoS9JFTH0Jakihr4kVWTS0I+IJRHxcEQciIgnIuKW0n5OROyKiKfL9uzSHhFxR0QciojHI+LymR6EJKk5zZzpvwL8XWa+B1gB3BwR7wU2ArszcymwuxwDXAMsLY/1wJ1tr1odt33fECtv28NFG3/Aytv2sH3fUKdLktSESUM/M49k5k/L/v8CB4A+YDWwtXTbCqwp+6uBu7PhR0BvRJzf9srVMdv3DbFp236GRkZJYGhklE3b9hv80hzQ0pp+RFwILAceBc7LzCPQeGMAzi3d+oDD4142WNpO/lnrI2IgIgaOHTvWeuXqmM07DzJ6/MTr2kaPn2DzzoMdqkhSs5oO/Yj4I+A/gL/NzP95o64TtOUpDZlbMrM/M/sXLlzYbBmaBYZHRltqlzR7NBX6EbGARuB/OzO3lebnx5ZtyvZoaR8Elox7+WJguD3lajZY1NvTUruk2aOZu3cCuAs4kJn/Mu6pHcDasr8WeGBc+43lLp4VwItjy0DqDhtWLaNnwbzXtfUsmMeGVcs6VJGkZs1vos9K4NPA/oh4rLT9PXAbcG9ErAOeA64vzz0IXAscAl4Cbmprxeq4Ncsbl2g27zzI8Mgoi3p72LBq2avtkmavyDxluf2M6+/vz4GBgU6XIUlzSkTszcz+Vl7jJ3IlqSKGviRVxNCXpIoY+pJUEUNfkipi6EtSRQx9SaqIoS9JFTH0Jakihr4kVcTQl6SKGPqSVBFDX5IqYuhLUkUMfUmqiKEvSRUx9CWpIoa+JFXE0Jekihj6klQRQ1+SKmLoS1JFDH1JqoihL0kVMfQlqSKGviRVxNCXpIpMGvoR8fWIOBoRPx/Xdk5E7IqIp8v27NIeEXFHRByKiMcj4vKZLF6S1JpmzvS/CVx9UttGYHdmLgV2l2OAa4Cl5bEeuLM9ZUqS2mHS0M/M/wR+c1LzamBr2d8KrBnXfnc2/AjojYjz21WsJGl6prqmf15mHgEo23NLex9weFy/wdImSZoF2n0hNyZoywk7RqyPiIGIGDh27Fiby5AkTWSqof/82LJN2R4t7YPAknH9FgPDE/2AzNySmf2Z2b9w4cIpliFJasVUQ38HsLbsrwUeGNd+Y7mLZwXw4tgykCSp8+ZP1iEivgt8EHhnRAwC/wDcBtwbEeuA54DrS/cHgWuBQ8BLwE0zULMkaYomDf3M/ORpnrpqgr4J3DzdoiRJM8NP5EpSRQx9SaqIoS9JFTH0Jakihr4kVcTQl6SKGPqSVBFDX5IqYuhLUkUMfUmqiKEvSRUx9CWpIoa+JFXE0Jekihj6klQRQ1+SKmLoS1JFJv0/Z0nSXLZ93xCbdx5keGSURb09bFi1jDXL+zpdVscY+pK61vZ9Q2zatp/R4ycAGBoZZdO2/QDVBr/LO5K61uadB18N/DGjx0+weefBDlXUeYa+pK41PDLaUnsNDH1JXWtRb09L7TUw9CV1rQ2rltGzYN7r2noWzGPDqmUdqqjzvJArqWuNXaz17p3XGPqSutqa5X1Vh/zJXN6RpIoY+pJUEUNfkioyI6EfEVdHxMGIOBQRG2fid0iSWtf20I+IecBXgWuA9wKfjIj3tvv3SJJaNxNn+lcAhzLzmcz8PXAPsHoGfo8kqUUzcctmH3B43PEg8Ocnd4qI9cD6cvhyRPx8BmqZLd4J/KrTRcygbh5fN48NHN9c1/KnzGYi9GOCtjylIXMLsAUgIgYys38GapkVHN/c1c1jA8c310XEQKuvmYnlnUFgybjjxcDwDPweSVKLZiL0fwIsjYiLIuLNwA3Ajhn4PZKkFrV9eSczX4mIzwA7gXnA1zPziUletqXddcwyjm/u6uaxgeOb61oeX2SestwuSepSfiJXkipi6EtSRc546EfEWyPixxHxs4h4IiK+UNoviohHI+LpiPheuQg8p7zB2L4ZEf8VEY+Vx2WdrnU6ImJeROyLiO+X4zk/d+NNML6umb+IeDYi9pdxDJS2cyJiV5m/XRFxdqfrnKrTjO8fI2Jo3Pxd2+k6pyIieiPivoh4KiIORMT7pzJ3nTjTfxm4MjMvBS4Dro6IFcCXgNszcynwArCuA7VN1+nGBrAhMy8rj8c6V2Jb3AIcGHfcDXM33snjg+6avw+VcYzdv74R2F3mb3c5nstOHh80/n2Ozd+DHatser4CPJSZFwOX0vg32vLcnfHQz4bflsMF5ZHAlcB9pX0rsOZM1zZdbzC2rhERi4GPAl8rx0EXzN2Yk8dXidU05g3m+Px1q4h4O/AB4C6AzPx9Zo4whbnryJp++fP5MeAosAv4BTCSma+ULoM0vs5hzjl5bJn5aHnqnyLi8Yi4PSLe0sESp+vLwOeAP5Tjd9Alc1ecPL4x3TJ/CfwwIvaWr0IBOC8zjwCU7bkdq276JhofwGfK/H19ji5fvQs4BnyjLD1+LSLOYgpz15HQz8wTmXkZjU/rXgG8Z6JuZ7aq9jh5bBHxp8Am4GLgz4BzgM93sMQpi4iPAUczc+/45gm6zsm5O834oEvmr1iZmZfT+BbcmyPiA50uqM0mGt+dwJ/QWHI9AvxzB+ubqvnA5cCdmbkc+D+muAzX0bt3yp8njwArgN6IGPuw2Jz/6oZxY7s6M4+UpZ+XgW/QeKObi1YCH4+IZ2l8e+qVNM6Mu2XuThlfRPx7F80fmTlctkeB+2mM5fmIOB+gbI92rsLpmWh8mfl8ORn7A/BvzM35GwQGx60c3EfjTaDluevE3TsLI6K37PcAH6ZxQeJh4BOl21rggTNd23SdZmxPjZuUoLHmNie/UTQzN2Xm4sy8kMbXa+zJzE/RBXMHpx3fX3XL/EXEWRHxtrF94CM0xrKDxrzBHJ6/041vbP6Kv2AOzl9m/hI4HBFj36p5FfAkU5i7mfiWzcmcD2yNxv9s5U3AvZn5/Yh4ErgnIr4I7KNcsJhjTje2PRGxkMZSyGPA33SyyBnweeb+3L2Rb3fJ/J0H3N9472I+8J3MfCgifgLcGxHrgOeA6ztY43ScbnzfKrfZJvAs8NedK3FaPkvj3+KbgWeAmyg508rc+TUMklQRP5ErSRUx9CWpIoa+JFXE0Jekihj6klQRQ1+SKmLoS1JF/h+c2fzKFtOGWgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.xlim(30, 60)\n",
    "plt.ylim(0, 600)\n",
    "\n",
    "X = np.array([39.93, 42.05, 43.18, 44.68, 49.87, 53.57, 55.28])\n",
    "Y = np.array([199,   290,   298,   310,   399,   420,   30])\n",
    "plt.scatter(X, Y)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到 (55.28, 30) 这个点明显有问题，对这个数据集进行训练："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a = [[370.98773744]\n",
      " [ -1.98111201]]\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "X = np.matrix([\n",
    "    [1, 1, 1, 1, 1, 1, 1], \n",
    "    [39.93, 42.05, 43.18, 44.68, 49.87, 53.57, 55.28]]).T\n",
    "Y = np.matrix([199,   290,   298,   310, 399,   420, 30]).T\n",
    "\n",
    "a = (X.T * X).I * X.T * Y\n",
    "print(\"a = {0}\".format(a))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "得到线性模型模型：\n",
    "\n",
    "$$\n",
    "y = -1.98111201x + 370.98773744\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAFt9JREFUeJzt3X9sXed93/H3VyQlUfIP2o7s2JKNOIsgJ2hmy+FSFwaK1i6q2AlqrYsRZ20iGB60AW6QYYMaq/+0GzI0gdClyVZ4U/OjdpbWMzz/QudFMeQYRbHFixS5VhJHs+I6NinVkn9QlixKIqnv/riHFEVdkffy1yXv834BxL3nuc8lnwfH/pzzPOecR5GZSJLKsKTVDZAkzR9DX5IKYuhLUkEMfUkqiKEvSQUx9CWpIA2FfkT0RMQjEfGziHgxIn4lIi6NiKcj4qXq9ZKqbkTE1yJif0S8EBE3zm0XJEmNavRM/6vAdzPzOuB64EXgPmBnZq4FdlbbALcBa6ufzcD9s9piSdK0xVQPZ0XERcDfAe/PcZUjYh/wa5l5MCKuBJ7NzHUR8V+r9381sd6c9UKS1JDOBuq8HzgMfCsirgd2A58HrhgN8ir4L6/qrwZeG/f9vqrsrNCPiM3URgKsXLnyI9ddd91M+iFJxdm9e/cbmbmqme80EvqdwI3A5zLzuYj4KmemcuqJOmXnDCcyczuwHaC3tzd37drVQFMkSaMi4hfNfqeROf0+oC8zn6u2H6F2EHi9mtahej00rv7V476/BjjQbMMkSbNvytDPzH8AXouIdVXRrcBPgSeBTVXZJuCJ6v2TwGeru3huAo44ny9JC0Mj0zsAnwO+ExFLgZeBu6kdMB6OiHuAV4E7q7pPAbcD+4HjVV1J0gLQUOhn5vNAb52Pbq1TN4F7Z9guSdIc8IlcSSqIoS9JBTH0Jakghr4kFcTQl6SCGPqSVBBDX5IKYuhLUkEMfUkqiKEvSQUx9CWpIIa+JBXE0Jekghj6klQQQ1+SCmLoS1JBDH1JKoihL0kFMfQlqSCGviQVxNCXpIIY+pJUEENfkgpi6EtSQQx9SSqIoS9JBTH0JakgDYV+RLwSEXsj4vmI2FWVXRoRT0fES9XrJVV5RMTXImJ/RLwQETfOZQckSY1r5kz/1zPzhszsrbbvA3Zm5lpgZ7UNcBuwtvrZDNw/W42VJM3MTKZ37gAeqN4/AGwcV/5g1vwA6ImIK2fwdyRJs6TR0E/gexGxOyI2V2VXZOZBgOr18qp8NfDauO/2VWWSpBbrbLDezZl5ICIuB56OiJ9NUjfqlOU5lWoHj80A11xzTYPNkCTNRENn+pl5oHo9BDwGfBR4fXTapno9VFXvA64e9/U1wIE6v3N7ZvZmZu+qVaum3wNJUsOmDP2IWBkRF46+B34T+DHwJLCpqrYJeKJ6/yTw2eounpuAI6PTQJKk1mpkeucK4LGIGK3/l5n53Yj4IfBwRNwDvArcWdV/Crgd2A8cB+6e9VZLkqZlytDPzJeB6+uUvwncWqc8gXtnpXWSpFnlE7mSVBBDX5IKYuhLUkEMfUkqiKEvSQUx9CWpIIa+JBXE0Jekghj6klQQQ1+SCmLoS1JBDH1JKoihL0kFMfQlqSCGviQVxNCXpIIY+pJUEENfkgpi6EtSQQx9SSqIoS9JBelsdQMkaaLH9/Szbcc+DgwMclVPN1s2rGPj+tWtblZbMPQlLSiP7+ln66N7GRwaAaB/YJCtj+4FMPhngdM7khaUbTv2jQX+qMGhEbbt2NeiFrUXQ1/SgnJgYLCpcjXH0Je0oFzV091UuZpj6EtaULZsWEd3V8dZZd1dHWzZsK5FLWovXsiVtKCMXqz17p250XDoR0QHsAvoz8xPRMS1wEPApcCPgM9k5qmIWAY8CHwEeBP4VGa+MustlzQti+F2yI3rVy+4NrWLZqZ3Pg+8OG77y8BXMnMt8DZwT1V+D/B2Zn4A+EpVT9ICMHo7ZP/AIMmZ2yEf39Pf6qZpnjQU+hGxBvg48PVqO4BbgEeqKg8AG6v3d1TbVJ/fWtWX1GLeDqlGz/T/FPh94HS1fRkwkJnD1XYfMDoWWw28BlB9fqSqf5aI2BwRuyJi1+HDh6fZfEnN8HZITRn6EfEJ4FBm7h5fXKdqNvDZmYLM7ZnZm5m9q1ataqixkmbG2yHVyJn+zcBvRcQr1C7c3kLtzL8nIkYvBK8BDlTv+4CrAarPLwbemsU2S5omb4fUlKGfmVszc01mvg+4C3gmM38H+D7wyaraJuCJ6v2T1TbV589k5jln+pLm38b1q/nj3/4wq3u6CWB1Tzd//Nsf9k6ZgszkPv0vAA9FxBeBPcA3qvJvAN+OiP3UzvDvmlkTJc0mb4csW1Ohn5nPAs9W718GPlqnzgngzllomyRplrkMgyQVxNCXpIIY+pJUEENfkgpi6EtSQQx9SSqIoS9JBTH0Jakghr4kFcTQl6SCGPqSVBBDX5IKYuhLUkEMfUkqiKEvSQUx9CWpIIa+JBXE0Jekghj6klQQQ1+SCmLoS1JBDH1JKoihL0kF6Wx1A6SF4vE9/WzbsY8DA4Nc1dPNlg3r2Lh+daubJc0qQ1+iFvhbH93L4NAIAP0Dg2x9dC+Awa+2siBC//+9fpR/dv//5qLlnVzc3cVF3V211+XVa3cnF5213cWFyzpZsiRa3XS1iW079o0F/qjBoRG27dhn6KutLIjQX97VwfKuJRw+dpKfH36XI4NDvHNiiMzzfycCLlzWycUrageD8QeIsQPGivrlF3V3sbyrY/46qAXvwMBgU+XSYjVl6EfEcuBvgGVV/Ucy8w8j4lrgIeBS4EfAZzLzVEQsAx4EPgK8CXwqM1+Z7G9cc+kKvvMvbjqr7PTp5NipYd4ZHKodBAaHxw4G7wwOnSk/cabOy28cG6s78axtoqWdS6qDwJnRRd0DR52RxwXLO+lwlNFWrurppr9OwF/V092C1khzp5Ez/ZPALZl5LCK6gL+NiP8F/BvgK5n5UET8F+Ae4P7q9e3M/EBE3AV8GfhUsw1bsiTGzuDXXNLst+Hk8AhHT1QHinEHiNHtd6oDyOhB4q13T/H3b7xblQ8zcnqSYQZw4fLOs6efzjlAdE4YaZw5cCzvWkKEB43ZNNOLsFs2rDtrTh+gu6uDLRvWzUVzpZaZMvQzM4Fj1WZX9ZPALcA/r8ofAP6IWujfUb0HeAT4zxER1e+ZN8s6O1h2QQfvuWBZ09/NTN49NXL2AWOSA8c7g8P84s3jYweR46emGGV0LKl7nWKyaxrjp6YcZZxtNi7Cjtbz7h21u4bm9COiA9gNfAD4M+DnwEBmDldV+oDR/ztWA68BZOZwRBwBLgPemPA7NwObAa655pqZ9WKWRQQXLOvkgmWdrJ7G8H5o5PTYiOHsA0b9qaqB46d49a3jHKnqTTXKuGBZ7SBwYTNTU1V5d1dH240yZusi7Mb1qw15tb2GQj8zR4AbIqIHeAz4YL1q1Wu9RDknxTJzO7AdoLe3d15HAXOtq2MJl12wjMumOco4fmrk3APEJAeO1946PnaQOXZyeNLf39URYyOGiaOLqQ4cFy7vpKtjfp7na2a6xouwUuOaunsnMwci4lngJqAnIjqrs/01wIGqWh9wNdAXEZ3AxcBbs9fk9hYRrFzWycplnVx5cfOjjOGR02euZdQ5QNSbqup/e3Dss6GRyY+/K5d2nDX9dNFk1zTOGml0sXJpY6OMZqdrvAgrNa6Ru3dWAUNV4HcDv0Ht4uz3gU9Su4NnE/BE9ZUnq+3/U33+zHzP55ess2MJl6xcyiUrlzb93czkxNDp+geIwSGODA6fU94/MMiLB2vbR6cYZXQsibrXLSYeOP7ke81N13gRVmpcTJXHEfGPqV2o7aC2Vs/DmfnvI+L9nLllcw/wu5l5srrF89vAempn+Hdl5suT/Y3e3t7ctWvXjDuj1ho5nRw9MdnI4nwjj9ptt6dGTk/5N9570fK6008HBgZ57uW3OHJiiEtXLOXTv3w1H//wVWN1L1jW2XbXMqSI2J2ZvU19ZyGchBv6AjgxNMI7g0N84j/9LYeOnjzn8wuWdXL7h99b92L40ZPDkz7MtySY8rrFRcs7x651TLx7almnD/Np4ZlO6C+IJ3IlGH0yu4M/uP2Ddadrvrjxl857MXfkdHLs5PBZ01FTXdN4/Z2TY9snhycfZSzvWjLpdQuXDNFiYehrwZnOPfMdS4KLqzC+ehp/88TQyNgzF+d78vvI8TMHjkNHT/DSoWpqqsklQ+o+gzHJcxkuGaLZZOhrQZrve+ZHRxmXX9j8d5tZMmR0ZPHzw8fGDiAnhiYfZdRbMuScBQldMkQNMvSlGZrvJUPePFZbMmT088me5YuoXQuZ6lqGS4aUw9CXWmw+lww5MjjEK28cHxuJTHfJkIsnjC5cMmTxMPSlRawVS4b84s13x+pPZ8mQydaVavclQxYCQ18qWCuWDPlJ9fm7U4wyJlsyZKprGvO5ZMhiY+hLmpZWLRkyujDh8BSjjPMtGVL/38ronNaSIYuRoS+pJWa6ZMjg0MjYLbPzvWRI3Yvh4+ou5FGGoS9p0YkIViztZMXSTt578fKmvz88cppjJ4fHRheNLBnSPzDY8JIh3V0d579baorVbed6yRBDX1JxOjuW0LNiKT0rpjfKODl8+vxPftc5cBw8coJ9rx/lyOAQR09MPspoaMmQ6sAxrb5P61uSVKiIGHuY74qLmh9ljJxOjp04e3Qxm0uGTMXQl6R51LEkuHhFFxevmOmSIUOs/XLz3zf0JWkRObNkSPOjDKitjy9JKoShL0kFMfQlqSCGviQVxNCXpIIY+pJUEENfkgpi6EtSQQx9SSqIoS9JBTH0Jakghr4kFcTQl6SCTBn6EXF1RHw/Il6MiJ9ExOer8ksj4umIeKl6vaQqj4j4WkTsj4gXIuLGue6EJKkxjZzpDwP/NjM/CNwE3BsRHwLuA3Zm5lpgZ7UNcBuwtvrZDNw/661Wyz2+p5+bv/QM1973P7n5S8/w+J7+VjdJUgOmDP3MPJiZP6reHwVeBFYDdwAPVNUeADZW7+8AHsyaHwA9EXHlrLdcLfP4nn62PrqX/oFBEugfGGTro3sNfmkRaGpOPyLeB6wHngOuyMyDUDswAJdX1VYDr437Wl9VNvF3bY6IXRGx6/Dhw823XC2zbcc+BodGziobHBph2459LWqRpEY1HPoRcQHwP4B/nZnvTFa1TlmeU5C5PTN7M7N31apVjTZDC8CBgcGmyiUtHA2FfkR0UQv872Tmo1Xx66PTNtXroaq8D876px/XAAdmp7laCK7q6W6qXNLC0cjdOwF8A3gxM//juI+eBDZV7zcBT4wr/2x1F89NwJHRaSC1hy0b1tHd1XFWWXdXB1s2rGtRiyQ1qpF/GP1m4DPA3oh4vir7A+BLwMMRcQ/wKnBn9dlTwO3AfuA4cPestlgtt3F97RLNth37ODAwyFU93WzZsG6sXNLCFZnnTLfPu97e3ty1a1ermyFJi0pE7M7M3ma+4xO5klQQQ1+SCmLoS1JBDH1JKoihL0kFMfQlqSCGviQVxNCXpIIY+pJUEENfkgpi6EtSQQx9SSqIoS9JBTH0Jakghr4kFcTQl6SCGPqSVBBDX5IKYuhLUkEMfUkqiKEvSQUx9CWpIIa+JBXE0Jekghj6klQQQ1+SCmLoS1JBpgz9iPhmRByKiB+PK7s0Ip6OiJeq10uq8oiIr0XE/oh4ISJunMvGS5Ka08iZ/l8AH5tQdh+wMzPXAjurbYDbgLXVz2bg/tlppiRpNkwZ+pn5N8BbE4rvAB6o3j8AbBxX/mDW/ADoiYgrZ6uxkqSZme6c/hWZeRCger28Kl8NvDauXl9VJklaAGb7Qm7UKcu6FSM2R8SuiNh1+PDhWW6GJKme6Yb+66PTNtXroaq8D7h6XL01wIF6vyAzt2dmb2b2rlq1aprNkCQ1Y7qh/ySwqXq/CXhiXPlnq7t4bgKOjE4DSZJar3OqChHxV8CvAe+JiD7gD4EvAQ9HxD3Aq8CdVfWngNuB/cBx4O45aLMkaZqmDP3M/PR5Prq1Tt0E7p1poyRJc8MnciWpIIa+JBXE0Jekghj6klQQQ1+SCmLoS1JBDH1JKoihL0kFMfQlqSCGviQVxNCXpIIY+pJUEENfkgpi6EtSQQx9SSqIoS9JBTH0JakgU/7LWZK0mD2+p59tO/ZxYGCQq3q62bJhHRvXr251s1rG0JfUth7f08/WR/cyODQCQP/AIFsf3QtQbPA7vSOpbW3bsW8s8EcNDo2wbce+FrWo9Qx9SW3rwMBgU+UlMPQlta2rerqbKi+BoS+pbW3ZsI7uro6zyrq7OtiyYV2LWtR6XsiV1LZGL9Z6984Zhr6ktrZx/eqiQ34ip3ckqSCGviQVxNCXpILMSehHxMciYl9E7I+I++bib0iSmjfroR8RHcCfAbcBHwI+HREfmu2/I0lq3lyc6X8U2J+ZL2fmKeAh4I45+DuSpCbNxS2bq4HXxm33Ab88sVJEbAY2V5snI+LHc9CWheI9wButbsQcauf+tXPfwP4tdk0/ZTYXoR91yvKcgsztwHaAiNiVmb1z0JYFwf4tXu3cN7B/i11E7Gr2O3MxvdMHXD1uew1wYA7+jiSpSXMR+j8E1kbEtRGxFLgLeHIO/o4kqUmzPr2TmcMR8XvADqAD+GZm/mSKr22f7XYsMPZv8WrnvoH9W+ya7l9knjPdLklqUz6RK0kFMfQlqSDzHvoRsTwi/m9E/F1E/CQi/l1Vfm1EPBcRL0XEf68uAi8qk/TtLyLi7yPi+ernhla3dSYioiMi9kTEX1fbi37fjVenf22z/yLilYjYW/VjV1V2aUQ8Xe2/pyPikla3c7rO078/ioj+cfvv9la3czoioiciHomIn0XEixHxK9PZd6040z8J3JKZ1wM3AB+LiJuALwNfycy1wNvAPS1o20ydr28AWzLzhurn+dY1cVZ8Hnhx3HY77LvxJvYP2mv//XrVj9H71+8Ddlb7b2e1vZhN7B/U/vsc3X9PtaxlM/NV4LuZeR1wPbX/Rpved/Me+llzrNrsqn4SuAV4pCp/ANg4322bqUn61jYiYg3wceDr1XbQBvtu1MT+FeIOavsNFvn+a1cRcRHwq8A3ADLzVGYOMI1915I5/Wr4/DxwCHga+DkwkJnDVZU+ass5LDoT+5aZz1Uf/YeIeCEivhIRy1rYxJn6U+D3gdPV9mW0yb6rTOzfqHbZfwl8LyJ2V0uhAFyRmQcBqtfLW9a6mavXP4Dfq/bfNxfp9NX7gcPAt6qpx69HxEqmse9aEvqZOZKZN1B7WvejwAfrVZvfVs2OiX2LiF8CtgLXAf8EuBT4QgubOG0R8QngUGbuHl9cp+qi3Hfn6R+0yf6r3JyZN1JbBffeiPjVVjdoltXr3/3AP6I25XoQ+JMWtm+6OoEbgfszcz3wLtOchmvp3TvV8ORZ4CagJyJGHxZb9Es3jOvbxzLzYDX1cxL4FrUD3WJ0M/BbEfEKtdVTb6F2Ztwu++6c/kXEf2uj/UdmHqheDwGPUevL6xFxJUD1eqh1LZyZev3LzNerk7HTwJ+zOPdfH9A3bubgEWoHgab3XSvu3lkVET3V+27gN6hdkPg+8Mmq2ibgiflu20ydp28/G7dTgtqc26JcUTQzt2bmmsx8H7XlNZ7JzN+hDfYdnLd/v9su+y8iVkbEhaPvgd+k1pcnqe03WMT773z9G91/lX/KItx/mfkPwGsRMbqq5q3AT5nGvpuLVTanciXwQNT+sZUlwMOZ+dcR8VPgoYj4IrCH6oLFInO+vj0TEauoTYU8D/yrVjZyDnyBxb/vJvOdNtl/VwCP1Y5ddAJ/mZnfjYgfAg9HxD3Aq8CdLWzjTJyvf9+ubrNN4BXgX7auiTPyOWr/LS4FXgbupsqZZvadyzBIUkF8IleSCmLoS1JBDH1JKoihL0kFMfQlqSCGviQVxNCXpIL8f6uvaiJfziPgAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.xlim(30, 60)\n",
    "plt.ylim(0, 600)\n",
    "\n",
    "X = np.array([39.93, 42.05, 43.18, 44.68, 49.87, 53.57, 55.28])\n",
    "Y = np.array([199,   290,   298,   310,   399,   420,   30])\n",
    "plt.scatter(X, Y)\n",
    "\n",
    "x = np.linspace(30, 60, 200)\n",
    "y = -1.98111201*x + 370.98773744\n",
    "plt.plot(x, y)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到，这个异常值直接导致模型完全变了样，这是因为在最小二乘法中我们使用的损失函数为平方损失：\n",
    "\n",
    "$$ loss = (y - \\hat{y})^2 $$\n",
    "\n",
    "其中，$y - \\hat{y}$ 叫做残差，表示训练样本和预测值之间的差值，显然异常值的残差比较大，为了让平方损失最小，模型不得不在正常值和异常值之间找一个平衡，这样残差越大，得出来的模型越不准确。有没有什么好的方法，能让学习算法对异常值不敏感？\n",
    "\n",
    "一般来说，有两种不同的思路，一种方法是在训练之前首先使用**异常检测**的方法将异常值剔除，这在我们后面学习无监督学习时会介绍到；另一种方法是保留异常值，但是采用结果不受异常值影响的算法，这种对异常值也能保持稳定可靠的性质，称为 **鲁棒性**，具有这样性质的学习算法称为 **鲁棒学习**。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看出，平方损失对异常值敏感的原因在于损失的大小随着残差呈平方级増长。所以鲁棒学习的关键就是找到某种损失函数，可以在残差增长的时候，保证损失不会变得太大。一个很容易下想到的损失函数就是绝对损失：\n",
    "\n",
    "$$ loss = |y - \\hat{y}| $$\n",
    "\n",
    "这里损失的大小随着残差呈线性增长，相比平方损失具有更好的鲁棒性。\n",
    "\n",
    "另一种鲁棒性很好的损失函数是 **Huber 损失**：\n",
    "\n",
    "$$\n",
    "Huber(r) = \n",
    "\\left \\{\n",
    "\\begin{align}\n",
    "& \\frac{r^2}{2} & (|r| \\leq \\eta) \\\\\n",
    "& \\eta |r| - \\frac{\\eta^2}{2} & (|r| > \\eta)\n",
    "\\end{align}\n",
    "\\right .\n",
    "$$\n",
    "\n",
    "如果残差的绝对值 $|r|$ 小于阈值 $\\eta$ 的话，上式就变成了平方损失；如果残差的绝对值 $|r|$ 大于阈值 $\\eta$ 的话，就变成了绝对损失，在绝对损失中减去常数项 $\\eta^2/2$ 是为了和平方损失平滑的连接。当阈值 $\\eta$ 非常小时，Huber 损失退化为绝对损失，所以求解绝对损失最小化可以转换为求解 Huber 损失最小化。\n",
    "\n",
    "Huber 损失最小化学习的求解方法是迭代算法，将其转换为加权最小二乘的形式：\n",
    "\n",
    "$$\n",
    "J(\\theta) = \\frac{1}{2} \\sum_{i=1}^n w_i r_i^2 + C\n",
    "$$\n",
    "\n",
    "其中，权重 $w$ 为：\n",
    "\n",
    "$$\n",
    "w_i =\n",
    "\\left \\{\n",
    "\\begin{align}\n",
    "& 1 & (|r_i| \\leq \\eta) \\\\\n",
    "& \\frac{\\eta}{|r_i|} & (|r_i| > \\eta)\n",
    "\\end{align}\n",
    "\\right .\n",
    "$$\n",
    "\n",
    "然后通过 **反复加权最小二乘学习法** 进行求解（参加《图解机器学习》p.61）。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "另一个具有很好鲁棒性的损失函数为 **Tukey 损失**（图基损失）：\n",
    "\n",
    "$$\n",
    "Tukey(r) = \n",
    "\\left \\{\n",
    "\\begin{align}\n",
    "& [1 - (1 - \\frac{r^2}{\\eta^2})^3] \\frac{\\eta^2}{6} & (|r| \\leq \\eta) \\\\\n",
    "& \\frac{\\eta^2}{6} & (|r| > \\eta)\n",
    "\\end{align}\n",
    "\\right .\n",
    "$$\n",
    "\n",
    "它也可以转换为加权最小二乘的形式，权重 $w$ 为：\n",
    "\n",
    "$$\n",
    "w =\n",
    "\\left \\{\n",
    "\\begin{align}\n",
    "& (1 - \\frac{r^2}{\\eta^2})^2 & (|r| \\leq \\eta) \\\\\n",
    "& 0 & (|r| > \\eta)\n",
    "\\end{align}\n",
    "\\right .\n",
    "$$"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}