notebooks/05.04-Feature-Engineering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--NAVIGATION-->\n",
    "< [超参数及模型验证](05.03-Hyperparameters-and-Model-Validation.ipynb) | [目录](Index.ipynb) | [深入：朴素贝叶斯分类](05.05-Naive-Bayes.ipynb) >\n",
    "\n",
    "<a href=\"https://colab.research.google.com/github/wangyingsm/Python-Data-Science-Handbook/blob/master/notebooks/05.04-Feature-Engineering.ipynb\"><img align=\"left\" src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\" title=\"Open and Execute in Google Colaboratory\"></a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Feature Engineering\n",
    "\n",
    "# 特征工程"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> The previous sections outline the fundamental ideas of machine learning, but all of the examples assume that you have numerical data in a tidy, ``[n_samples, n_features]`` format.\n",
    "In the real world, data rarely comes in such a form.\n",
    "With this in mind, one of the more important steps in using machine learning in practice is *feature engineering*: that is, taking whatever information you have about your problem and turning it into numbers that you can use to build your feature matrix.\n",
    "\n",
    "上几节中我们描述了机器学习的基本概念，但前面所有的例子都假定你的数据都是数值的，并且具有干净的形状为`[n_samples, n_features]`格式。在现实世界中，数据很少具有这样的格式。有了这个前提，要在实践中使用机器学习其中一个重要的步骤就是*特征工程*：也就是使用你拿到的数据，将它们转换为数值形式，以便你可以用来在特征矩阵中使用它们。\n",
    "\n",
    "> In this section, we will cover a few common examples of feature engineering tasks: features for representing *categorical data*, features for representing *text*, and features for representing *images*.\n",
    "Additionally, we will discuss *derived features* for increasing model complexity and *imputation* of missing data.\n",
    "Often this process is known as *vectorization*, as it involves converting arbitrary data into well-behaved vectors.\n",
    "\n",
    "在本节中我们会介绍一些特征工程任务的通用例子：表示*分类数据*的特征，表示*文字*的特征和表示*图像*的特征。除此之外我们还会讨论*派生特征*用于增加模型复杂度和对缺失值进行插值。通常这个过程被称为*向量化*，因为它意味着将任意数据转变成格式良好的向量。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Categorical Features\n",
    "\n",
    "## 分类特征\n",
    "\n",
    "> One common type of non-numerical data is *categorical* data.\n",
    "For example, imagine you are exploring some data on housing prices, and along with numerical features like \"price\" and \"rooms\", you also have \"neighborhood\" information.\n",
    "For example, your data might look something like this:\n",
    "\n",
    "非数值数据的一个常见类型是*分类*数据。例如，假设你在研究房价的数据，数据集中除了数值特征如“价格”和“房间数”之外，还有会有例如“邻近地区”这样的信息。下面例子展示了这个数据的可能情况："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = [\n",
    "    {'price': 850000, 'rooms': 4, 'neighborhood': 'Queen Anne'},\n",
    "    {'price': 700000, 'rooms': 3, 'neighborhood': 'Fremont'},\n",
    "    {'price': 650000, 'rooms': 3, 'neighborhood': 'Wallingford'},\n",
    "    {'price': 600000, 'rooms': 2, 'neighborhood': 'Fremont'}\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> You might be tempted to encode this data with a straightforward numerical mapping:\n",
    "\n",
    "你可能想要将这个数据直接进行数值类型的编码："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "{'Queen Anne': 1, 'Fremont': 2, 'Wallingford': 3};"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> It turns out that this is not generally a useful approach in Scikit-Learn: the package's models make the fundamental assumption that numerical features reflect algebraic quantities.\n",
    "Thus such a mapping would imply, for example, that *Queen Anne < Fremont < Wallingford*, or even that *Wallingford - Queen Anne = Fremont*, which (niche demographic jokes aside) does not make much sense.\n",
    "\n",
    "这在Scikit-Learn中不是一个实用的方法：包中的模型基本上假设数值特征表示的都是算术量。因此这样的映射会暗示比如*Queen Anne < Fremont < Wallingford*，甚至*Wallingford - Queen Anne = Fremont*，这种转换没有任何含义。\n",
    "\n",
    "> In this case, one proven technique is to use *one-hot encoding*, which effectively creates extra columns indicating the presence or absence of a category with a value of 1 or 0, respectively.\n",
    "When your data comes as a list of dictionaries, Scikit-Learn's ``DictVectorizer`` will do this for you:\n",
    "\n",
    "在这种情况下，有一种证明过的技巧可以使用被称为*one-hot encoding*，它能有效的创建额外的列代表一个类别的存在或缺失，分别使用数值1或0表示。如果你的数据是一个字典的列表格式，Scikit-Learn的`DictVectorizer`可以帮你完成这项工作："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[     0,      1,      0, 850000,      4],\n",
       "       [     1,      0,      0, 700000,      3],\n",
       "       [     0,      0,      1, 650000,      3],\n",
       "       [     1,      0,      0, 600000,      2]])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction import DictVectorizer\n",
    "vec = DictVectorizer(sparse=False, dtype=int)\n",
    "vec.fit_transform(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Notice that the 'neighborhood' column has been expanded into three separate columns, representing the three neighborhood labels, and that each row has a 1 in the column associated with its neighborhood.\n",
    "With these categorical features thus encoded, you can proceed as normal with fitting a Scikit-Learn model.\n",
    "\n",
    "上面的变换之后'neighborhood'列已经被扩展成为3个独立的列，分别代表三个邻近地区的标签，然后每行中1所在的列的位置与邻近地区相关。经过这样的分类特征编码后，你就可以使用Scikit-Learn模型进行拟合数据了。\n",
    "\n",
    "> To see the meaning of each column, you can inspect the feature names:\n",
    "\n",
    "要查看每个列的含义，你可以列出特征名称："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['neighborhood=Fremont',\n",
       " 'neighborhood=Queen Anne',\n",
       " 'neighborhood=Wallingford',\n",
       " 'price',\n",
       " 'rooms']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vec.get_feature_names()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> There is one clear disadvantage of this approach: if your category has many possible values, this can *greatly* increase the size of your dataset.\n",
    "However, because the encoded data contains mostly zeros, a sparse output can be a very efficient solution:\n",
    "\n",
    "这种方法有一个明显的缺点：如果你的分类特征有很多可能的取值，这会*极大*增加你的数据集的大小。但是因为编码后的数据大部分都是0值，因此输出结果作为稀疏矩阵是非常高效的："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<4x5 sparse matrix of type '<class 'numpy.int64'>'\n",
       "\twith 12 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vec = DictVectorizer(sparse=True, dtype=int)\n",
    "vec.fit_transform(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Many (though not yet all) of the Scikit-Learn estimators accept such sparse inputs when fitting and evaluating models. ``sklearn.preprocessing.OneHotEncoder`` and ``sklearn.feature_extraction.FeatureHasher`` are two additional tools that Scikit-Learn includes to support this type of encoding.\n",
    "\n",
    "许多（虽然不是全部）Scikit-Learn评估器接受这样的稀疏输入作为模型拟合及预测的参数。`sklearn.preprocessing.OneHotEncoder`和`sklearn.feature_extraction.FeatureHasher`是另外两个额外的工具支持这种编码。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text Features\n",
    "\n",
    "## 文字特征\n",
    "\n",
    "> Another common need in feature engineering is to convert text to a set of representative numerical values.\n",
    "For example, most automatic mining of social media data relies on some form of encoding the text as numbers.\n",
    "One of the simplest methods of encoding data is by *word counts*: you take each snippet of text, count the occurrences of each word within it, and put the results in a table.\n",
    "\n",
    "另外一种特征工程常见的需求是将文字转换成一组代表它们的数字值。例如大多数社交媒体数据的自动挖掘都依赖于某种形式的文字到数字的编码转换。其中最简单的方法是进行*单词计数*：选取每一小段文字，计算里面每个单词出现的次数，然后将它们放到表中。\n",
    "\n",
    "> For example, consider the following set of three phrases:\n",
    "\n",
    "以下面的三个短语为例："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample = ['problem of evil',\n",
    "          'evil queen',\n",
    "          'horizon problem']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> For a vectorization of this data based on word count, we could construct a column representing the word \"problem,\" the word \"evil,\" the word \"horizon,\" and so on.\n",
    "While doing this by hand would be possible, the tedium can be avoided by using Scikit-Learn's ``CountVectorizer``:\n",
    "\n",
    "想要将上面的数据使用单词计数进行向量化，我们可以构造一个列代表单词\"problem\"， 一个列代表单词\"evil\"，一个列代表单词\"horizon\"等等。虽然可以手工完成这项任务，但是你可以使用Scikit-Learn的`CountVectorizer`将自己从重复劳动中解放出来："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<3x5 sparse matrix of type '<class 'numpy.int64'>'\n",
       "\twith 7 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "vec = CountVectorizer()\n",
    "X = vec.fit_transform(sample)\n",
    "X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> The result is a sparse matrix recording the number of times each word appears; it is easier to inspect if we convert this to a ``DataFrame`` with labeled columns:\n",
    "\n",
    "结果是一个稀疏矩阵，它记录了每个单词出现的次数；我们将它转成一个`DataFrame`就可以很方便的看到数据："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>evil</th>\n",
       "      <th>horizon</th>\n",
       "      <th>of</th>\n",
       "      <th>problem</th>\n",
       "      <th>queen</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   evil  horizon  of  problem  queen\n",
       "0     1        0   1        1      0\n",
       "1     1        0   0        0      1\n",
       "2     0        1   0        1      0"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "pd.DataFrame(X.toarray(), columns=vec.get_feature_names())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> There are some issues with this approach, however: the raw word counts lead to features which put too much weight on words that appear very frequently, and this can be sub-optimal in some classification algorithms.\n",
    "One approach to fix this is known as *term frequency-inverse document frequency* (*TF–IDF*) which weights the word counts by a measure of how often they appear in the documents.\n",
    "The syntax for computing these features is similar to the previous example:\n",
    "\n",
    "然而这种处理方法有一些问题：原始的单词计数会导致特征在频繁出现的单词上放置了太多的权重，这对于一些分类算法来说是不够准确的。解决这个问题的其中一个办法是被称为*term frequency-inverse document frequency* (*TF–IDF*)的算法，它会将根据单词在文档中出现的频率单词计数的权重。计算这些特征的语法与前面的例子类似："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>evil</th>\n",
       "      <th>horizon</th>\n",
       "      <th>of</th>\n",
       "      <th>problem</th>\n",
       "      <th>queen</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.517856</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.680919</td>\n",
       "      <td>0.517856</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.605349</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.795961</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.795961</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.605349</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       evil   horizon        of   problem     queen\n",
       "0  0.517856  0.000000  0.680919  0.517856  0.000000\n",
       "1  0.605349  0.000000  0.000000  0.000000  0.795961\n",
       "2  0.000000  0.795961  0.000000  0.605349  0.000000"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "vec = TfidfVectorizer()\n",
    "X = vec.fit_transform(sample)\n",
    "pd.DataFrame(X.toarray(), columns=vec.get_feature_names())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> For an example of using TF-IDF in a classification problem, see [In Depth: Naive Bayes Classification](05.05-Naive-Bayes.ipynb).\n",
    "\n",
    "使用TF-IDF在分类问题中的例子，可参见[深入：朴素贝叶斯分类](05.05-Naive-Bayes.ipynb)。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Image Features\n",
    "\n",
    "## 图像特征\n",
    "\n",
    "> Another common need is to suitably encode *images* for machine learning analysis.\n",
    "The simplest approach is what we used for the digits data in [Introducing Scikit-Learn](05.02-Introducing-Scikit-Learn.ipynb): simply using the pixel values themselves.\n",
    "But depending on the application, such approaches may not be optimal.\n",
    "\n",
    "还有一种常见的需求是将*图像*编码成适合机器学习分析的数据。最简单的方法在[Scikit-Learn简介](05.02-Introducing-Scikit-Learn.ipynb)中已经看到过：直接使用图像的像素数据。但是根据应用场景不同，这种方法可能不是最优的。\n",
    "\n",
    "> A comprehensive summary of feature extraction techniques for images is well beyond the scope of this section, but you can find excellent implementations of many of the standard approaches in the [Scikit-Image project](http://scikit-image.org).\n",
    "For one example of using Scikit-Learn and Scikit-Image together, see [Feature Engineering: Working with Images](05.14-Image-Features.ipynb).\n",
    "\n",
    "图像中特征提取技术的完整介绍远远超出了本节的范围，但是你可以在[Scikit-Image项目](http://scikit-image.org)中找到一流的标准方法实现。参见[特征工程：使用图像数据](05.14-Image-Features.ipynb)中可以看到联合使用Scikit-Learn和Scikit-Image的例子。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Derived Features\n",
    "\n",
    "## 派生特征\n",
    "\n",
    "> Another useful type of feature is one that is mathematically derived from some input features.\n",
    "We saw an example of this in [Hyperparameters and Model Validation](05.03-Hyperparameters-and-Model-Validation.ipynb) when we constructed *polynomial features* from our input data.\n",
    "We saw that we could convert a linear regression into a polynomial regression not by changing the model, but by transforming the input!\n",
    "This is sometimes known as *basis function regression*, and is explored further in [In Depth: Linear Regression](05.06-Linear-Regression.ipynb).\n",
    "\n",
    "另一个有用的特征类型是从其他输入特征中进行数学计算并派生获得的特征。我们已经在[超参数与模型验证](05.03-Hyperparameters-and-Model-Validation.ipynb)中看到了一个例子，我们从输入数据中构造了*多项式特征*。该例中我们看到能够将一个线性回归转变成一个多项式回归，这不是通过改变模型实现的，而是通过转变输入数据实现的。这有时被称为*基本函数回归*，[深入：线性回归](05.06-Linear-Regression.ipynb)一节中会更加深入讨论这方面内容。\n",
    "\n",
    "> For example, this data clearly cannot be well described by a straight line:\n",
    "\n",
    "例如这个数据显然无法使用直线很好的拟合："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAO4ElEQVR4nO3dX2xk5X3G8eep10kmQGopO0pYL62VG0stNHhrIaKVEIU0hgYRi3JBpKQlUrX9E7WJWjmqc5GoveHCUkTaSo22kJY0kJASYxEEOFQURbnIVrOYxjSLKxptlLVpd0hl/jSjZDG/Xni87LrjnTMwZ87P+PuRrD0+5/WcR+/uPDPzzpm1I0IAgLx+oeoAAIALo6gBIDmKGgCSo6gBIDmKGgCS21fGje7fvz/GxsbKuGkAeEs6fvz4CxFR73SslKIeGxtTo9Eo46YB4C3J9o92OsbSBwAkR1EDQHIUNQAkR1EDQHIUNQAkR1EDQHJdL8+zPS7p/nN2vU/S5yLiztJSAcAusbC0qrnFFa2tt3RgpKaZqXFNT4z29RxdizoiViRdKUm2hyStSnqwrykAYBdaWFrV7PyyWmc2JEmr6y3Nzi9LUl/Lutelj+sl/WdE7HhhNgDsFXOLK2dLekvrzIbmFlf6ep5ei/o2SV/rdMD2EdsN241ms/nmkwFAcmvrrZ72v1GFi9r22yTdLOmfOh2PiKMRMRkRk/V6x4+rA8BbyoGRWk/736henlHfKOmpiPjvviYAgF1qZmpcteGh8/bVhoc0MzXe1/P08p8yfVQ7LHsAwF609YZh5Vd9SJLtd0r6TUm/39ezA8AuNz0x2vdi3q5QUUfETyW9u9QkAICO+GQiACRHUQNAchQ1ACRHUQNAchQ1ACRHUQNAchQ1ACRHUQNAchQ1ACRHUQNAchQ1ACRHUQNAchQ1ACRHUQNAchQ1ACRHUQNAchQ1ACRHUQNAchQ1ACRHUQNAchQ1ACRHUQNAchQ1ACRXqKhtj9h+wPaztk/Y/kDZwQAAm/YVHPdFSY9FxK223ybpnSVmAgCco2tR236XpGsk3S5JEfFzST8vNxYAYEuRpY/3SWpK+nvbS7bvsn3R9kG2j9hu2G40m82+BwWAvapIUe+TdEjS30bEhKT/lfTn2wdFxNGImIyIyXq93ueYALB3FSnqU5JORcSx9vcPaLO4AQAD0LWoI+K/JP3Y9nh71/WSflBqKgDAWUWv+vhjSfe2r/j4oaRPlBcJAHCuQkUdEU9Lmiw5CwCgAz6ZCADJUdQAkBxFDQDJUdQAkBxFDQDJUdQAkBxFDQDJUdQAkBxFDQDJUdQAkBxFDQDJUdQAkBxFDQDJUdQAkBxFDQDJUdQAkBxFDQDJUdQAkBxFDQDJUdQAkBxFDQDJUdQAkBxFDQDJUdQAkNy+IoNsn5T0sqQNSa9GxGSZoQAArytU1G2/EREvlJYEANARSx8AkFzRog5J37Z93PaRTgNsH7HdsN1oNpv9SwgAe1zRoj4cEYck3Sjpk7av2T4gIo5GxGRETNbr9b6GBIC9rFBRR8Ra+8/Tkh6UdFWZoQAAr+ta1LYvsn3J1rakD0l6puxgAIBNRa76eI+kB21vjb8vIh4rNRUA4KyuRR0RP5T0/gFkAQB0wOV5AJAcRQ0AyVHUAJAcRQ0AyVHUAJAcRQ0AyVHUAJAcRQ0AyVHUAJAcRQ0AyVHUAJAcRQ0AyVHUAJAcRQ0AyVHUAJAcRQ0AyVHUAJAcRQ0AyVHUAJAcRQ0AyVHUAJAcRQ0AyVHUAJAcRQ0Aye0rOtD2kKSGpNWIuKnfQRaWVjW3uKK19ZYOjNQ0MzWu6YnRfp8GAHadwkUt6VOSTkh6V79DLCytanZ+Wa0zG5Kk1fWWZueXJYmyBrDnFVr6sH1Q0ocl3VVGiLnFlbMlvaV1ZkNziytlnA4AdpWia9R3SvqMpNd2GmD7iO2G7Uaz2ewpxNp6q6f9ALCXdC1q2zdJOh0Rxy80LiKORsRkREzW6/WeQhwYqfW0HwD2kiLPqA9Lutn2SUlfl3Sd7a/2M8TM1Lhqw0Pn7asND2lmaryfpwGAXalrUUfEbEQcjIgxSbdJeiIiPtbPENMTo7rjlis0OlKTJY2O1HTHLVfwRiIAqLerPko1PTFKMQNABz0VdUQ8KenJUpIAADrik4kAkBxFDQDJUdQAkBxFDQDJUdQAkBxFDQDJUdQAkBxFDQDJUdQAkBxFDQDJUdQAkBxFDQDJUdQAkBxFDQDJUdQAkBxFDQDJUdQAkBxFDQDJUdQAkBxFDQDJUdQAkBxFDQDJUdQAkBxFDQDJ7es2wPY7JH1H0tvb4x+IiM+XHQxAdRaWVjW3uKK19ZYOjNQ0MzWu6YnRqmPtWV2LWtLPJF0XEa/YHpb0XduPRsT3Ss4GoAILS6uanV9W68yGJGl1vaXZ+WVJoqwr0nXpIza90v52uP0VpaYCUJm5xZWzJb2ldWZDc4srFSVCoTVq20O2n5Z0WtLjEXGsw5gjthu2G81ms985AQzI2nqrp/0oX6GijoiNiLhS0kFJV9m+vMOYoxExGRGT9Xq93zkBDMiBkVpP+1G+nq76iIh1SU9KuqGUNAAqNzM1rtrw0Hn7asNDmpkarygRuha17brtkfZ2TdIHJT1bdjAA1ZieGNUdt1yh0ZGaLGl0pKY7brmCNxIrVOSqj0sl3WN7SJvF/o2IeLjcWACqND0xSjEn0rWoI+L7kiYGkAUA0AGfTASA5ChqAEiOogaA5ChqAEiOogaA5ChqAEiOogaA5ChqAEiOogaA5ChqAEiOogaA5ChqAEiOogaA5ChqAEiOogaA5ChqAEiOogaA5ChqAEiOogaA5ChqAEiOogaA5ChqAEiOogaA5ChqAEhuX7cBti+T9BVJ75X0mqSjEfHFsoPhwhaWVjW3uKK19ZYOjNQ0MzWu6YnRqmMBKEHXopb0qqQ/i4inbF8i6bjtxyPiByVnww4WllY1O7+s1pkNSdLqekuz88uSRFkDb0Fdlz4i4vmIeKq9/bKkE5JogwrNLa6cLektrTMbmltcqSgRgDL1tEZte0zShKRjHY4dsd2w3Wg2m/1Jh47W1ls97QewuxUuatsXS/qmpE9HxEvbj0fE0YiYjIjJer3ez4zY5sBIraf9AHa3QkVte1ibJX1vRMyXGwndzEyNqzY8dN6+2vCQZqbGK0oEoExFrvqwpLslnYiIL5QfCd1svWHIVR/A3lDkqo/Dkj4uadn20+19n42IR8qLhW6mJ0YpZmCP6FrUEfFdSR5AFgBAB3wyEQCSo6gBIDmKGgCSo6gBIDmKGgCSo6gBIDmKGgCSo6gBIDmKGgCSo6gBIDmKGgCSo6gBIDmKGgCSo6gBIDmKGgCSo6gBIDmKGgCSo6gBIDmKGgCSo6gBIDmKGgCSo6gBIDmKGgCS29dtgO0vS7pJ0umIuLz8SED/LSytam5xRWvrLR0YqWlmalzTE6NVxwIKKfKM+h8k3VByDqA0C0urmp1f1up6SyFpdb2l2fllLSytVh0NKKRrUUfEdyT9zwCyAKWYW1xR68zGeftaZzY0t7hSUSKgN31bo7Z9xHbDdqPZbPbrZoE3bW291dN+IJu+FXVEHI2IyYiYrNfr/bpZ4E07MFLraT+QDVd94C1vZmpcteGh8/bVhoc0MzVeUSKgN12v+gB2u62rO7jqA7tVkcvzvibpWkn7bZ+S9PmIuLvsYEA/TU+MUszYtboWdUR8dBBBAACdsUYNAMlR1ACQHEUNAMlR1ACQHEUNAMk5Ivp/o3ZT0o/e4I/vl/RCH+P0C7l6Q67ekKs3GXO92Uy/HBEdP9ZdSlG/GbYbETFZdY7tyNUbcvWGXL3JmKvMTCx9AEByFDUAJJexqI9WHWAH5OoNuXpDrt5kzFVapnRr1ACA82V8Rg0AOAdFDQDJVVbUtr9s+7TtZ3Y4btt/Zfs529+3fShBpmttv2j76fbX58rO1D7vZbb/xfYJ2/9u+1MdxlQxX0VyDXzObL/D9r/a/rd2rr/oMObttu9vz9cx22NJct1uu3nOfP1e2bna5x2yvWT74Q7HBj5XBXNVNVcnbS+3z9nocLz/98WIqORL0jWSDkl6ZofjvyXpUUmWdLWkYwkyXSvp4Qrm6lJJh9rbl0j6D0m/kmC+iuQa+Jy15+Di9vawpGOSrt425o8kfam9fZuk+5Pkul3S31Twb+xPJd3X6e+qirkqmKuquTopaf8Fjvf9vljZM+ro/tvNPyLpK7Hpe5JGbF9acaZKRMTzEfFUe/tlSSckbf9f8KuYryK5Bq49B6+0vx1uf21/1/wjku5pbz8g6XrbTpBr4GwflPRhSXftMGTgc1UwV1Z9vy9mXqMelfTjc74/pQQlIOkD7Zeuj9r+1UGfvP2yc0Kbz8bOVel8XSCXVMGctV8yPy3ptKTHI2LH+YqIVyW9KOndCXJJ0m+3XzI/YPuysjNJulPSZyS9tsPxSuaqQC5p8HMlbT64ftv2cdtHOhzv+30xc1F3esSu+tnHU9r8PP77Jf21pIVBntz2xZK+KenTEfHS9sMdfmQg89UlVyVzFhEbEXGlpIOSrrJ9+bYhlcxXgVzfkjQWEb8m6Z/1+jPZUti+SdLpiDh+oWEd9pU6VwVzDXSuznE4Ig5JulHSJ21fs+143+crc1GfknTuI+RBSWsVZZEkRcRLWy9dI+IRScO29w/i3LaHtVmG90bEfIchlcxXt1xVzln7nOuSnpR0w7ZDZ+fL9j5Jv6gBLnvtlCsifhIRP2t/+3eSfr3kKIcl3Wz7pKSvS7rO9le3jalirrrmqmCuts671v7ztKQHJV21bUjf74uZi/ohSb/Tfgf1akkvRsTzVQay/d6ttTnbV2lz/n4ygPNa0t2STkTEF3YYNvD5KpKrijmzXbc90t6uSfqgpGe3DXtI0u+2t2+V9ES03wmqMte2tcybtbnuX5qImI2IgxExps03Cp+IiI9tGzbwuSqSa9Bz1T7nRbYv2dqW9CFJ268S6/t9sesvty2LO/x2c22+uaKI+JKkR7T57ulzkn4q6RMJMt0q6Q9tvyqpJem2sv/Bth2W9HFJy+31TUn6rKRfOifbwOerYK4q5uxSSffYHtLmA8M3IuJh238pqRERD2nzAeYfbT+nzWeHt5WcqWiuP7F9s6RX27luH0Cu/yfBXBXJVcVcvUfSg+3nHvsk3RcRj9n+A6m8+yIfIQeA5DIvfQAARFEDQHoUNQAkR1EDQHIUNQAkR1EDQHIUNQAk939cSUrhFQnAnwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "x = np.array([1, 2, 3, 4, 5])\n",
    "y = np.array([4, 2, 1, 3, 7])\n",
    "plt.scatter(x, y);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Still, we can fit a line to the data using ``LinearRegression`` and get the optimal result:\n",
    "\n",
    "我们仍然可以使用`LinearRegression`将直线拟合到数据上："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAVK0lEQVR4nO3df3Db9X3H8dcbx4BJSE0SW2ATYwLBlcmtmBkopUkp2DUtHM1u/YOudKNXmnXt+mPdzC29rr3tbn/sfNdru+3Wy9pu7fpj7WjIdVyLWxsopbfSOoQ1NI75tfDDprYTcEJAJLbz3h+SEseVLCnRV/pIej7ucsjSF+vNh+jlr7/6vvQ1dxcAIFxnlHsAAMDSCGoACBxBDQCBI6gBIHAENQAEblkU33TNmjXe3t4exbcGgKq0c+fO/e7elOmxSIK6vb1dIyMjUXxrAKhKZvZstsc49AEAgSOoASBwBDUABI6gBoDAEdQAEDiCGgACl/P0PDPrkPTdBXetk/RZd/9CZFMBQIXYsWtcA4NjmphJqKWxQf19Hdrc1VrU58gZ1O4+JukKSTKzOknjku4p6hQAUIF27BrX1u27lZidlySNzyS0dftuSSpqWBd66ONGSU+7e9YTswGgVgwMjh0P6bTE7LwGBseK+jyFBvVtkr6T6QEz22JmI2Y2Mj09ffqTAUDgJmYSBd1/qvIOajM7U9Ktkv4r0+Puvs3du929u6kpY10dAKpKS2NDQfefqkL2qN8p6VF3nyzqBABQofr7OtRQX3fSfQ31derv6yjq8xTyoUzvVZbDHgBQi9JvGJb9rA9JMrNzJPVK+tOiPjsAVLjNXa1FD+bF8gpqd39N0upIJwEAZEQzEQACR1ADQOAIagAIHEENAIEjqAEgcAQ1AASOoAaAwBHUABA4ghoAAkdQA0DgCGoACBxBDQCBI6gBIHAENQAEjqAGgMAR1AAQOIIaAAJHUANA4AhqAAgcQQ0AgSOoASBwBDUABI6gBoDA5RXUZtZoZneb2V4zGzWza6MeDACQtCzP7b4o6T53f4+ZnSnpnAhnAgAskDOozWylpE2S7pAkdz8q6Wi0YwEA0vI59LFO0rSkfzOzXWb2FTNbvngjM9tiZiNmNjI9PV30QQGgVuUT1MskXSnpX9y9S9Krkv568Ubuvs3du929u6mpqchjAkDtyieoX5D0grs/kvr6biWDGwBQAjmD2t1/K+l5M+tI3XWjpD2RTgUAOC7fsz4+JulbqTM+npH0gehGAgAslFdQu/tjkrojngUAkAHNRAAIHEENAIEjqAEgcAQ1AASOoAaAwBHUABA4ghoAAkdQA0DgCGoACBxBDQCBI6gBIHAENQAEjqAGgMAR1AAQOIIaAAJHUANA4AhqAAgcQQ0AgSOoASBwBDUABI6gBoDAEdQAEDiCGgACR1ADQOCW5bORme2T9IqkeUlz7t4d5VAAgBPyCuqUt7v7/sgmAQBkxKEPAAhcvkHtkn5sZjvNbEumDcxsi5mNmNnI9PR08SYEgBqXb1Bf5+5XSnqnpI+a2abFG7j7Nnfvdvfupqamog4JALUsr6B294nUP6ck3SPp6iiHAgCckDOozWy5mZ2bvi3pHZIej3owAEBSPmd9xCTdY2bp7b/t7vdFOhUA4LicQe3uz0h6UwlmAYCKNTd/THVnmFI7tUVVyHnUAIAFDiZm9dMnpjU8OqkH9k5p+0eu06XNK4r+PAQ1ABTguQOvaWh0UkOjk/rl/72kuWOu1cvP1DsuP18R7ExLIqgBYEnzx1yPPT+j4VQ4PzF5WJK0vnmFPrRpnXrizbpi7XmqOyOilBZBDQC/47Wjc/rZk/s1tGdSD4xNaf/ho6o7w3R1+yr9zS1t6ok366LVy0s2D0ENAJJ+e/B1De+d1NCeSf386QM6OndM5569TG/vaNaN8WZdf1mz3nBOfVlmI6gB1CR3128mDml4dEpDo5PaPX5QktS26hzdfs1F6uls1lXtq1RfV/6PRCKoAdSMI3Pz+p+nD2hodFLDo1N68eDrMpOubDtPd93Uod54TJc2r4jkFLvTQVADqGoHDh/R/XunNDw6pYeenNZrR+fVUF+nTZet0V/0XqYb3tisNSvOKveYSyKoAVQVd9fT04f1kz1TGh6d1M7nXpa7dP7Ks/UHXa3q6Yzp2nWrdXZ9XblHzRtBDaDizc4f06/2vXT8ePOzB16TJG1oXamP37BevZ0xXd6yMrhDGvkiqAFUpMWtwEOvz+nMujP0lktX686N63TjG5vV0thQ7jGLgqAGUDGWagX2xGPauH6Nlp9VfbFWff9FAKrGUq3AOzeuU29n9K3AEBDUAIISWiswBAQ1gLLL1gq8vqNZPWVuBYaAoAZQculWYLp4km4Frl3VkGwFxpt11cVhtAJDQFADKIlsrcCutY2666YO9cRjWh9gKzAEBDWAyGRrBW5cXzmtwBAQ1ACKJlsrMLbyrGQrMB7TtZdUViswBAQ1gNOSrRV4eUuyFdgTj2lDa+W2AkNAUAMoWLoVOLRnUg+OVXcrMAQENYC8ZGoFrjreCmzWxvVNVdkKDAGrCiCjbK3AS2usFRgCghrAcUu1Aj9z81r1xGNqX1NbrcAQ5B3UZlYnaUTSuLvfUuxBduwa18DgmCZmEmppbFB/X4c2d7UW+2kALJKtFfi2y5rU2xmr+VZgCArZo/6EpFFJK4s9xI5d49q6fbcSs/OSpPGZhLZu3y1JhDVQZEu1At93TZt64zFagYHJK6jN7EJJN0v6e0mfKvYQA4Njx0M6LTE7r4HBMYIaKIKlWoH9fR3q7aQVGLJ896i/IOkuSedm28DMtkjaIkltbW0FDTExkyjofgC50QqsHjmD2sxukTTl7jvN7Pps27n7NknbJKm7u9sLGaKlsUHjGUKZ8zCB/C3VCtzc1apeWoEVK5896usk3Wpm75J0tqSVZvZNd7+9WEP093WcdIxakhrq69Tf11GspwCq0uz8MY3sezl1SGNS+2gFVqWcQe3uWyVtlaTUHvVfFTOkpRNvGHLWB5DbUtcK/CCtwKoUzHnUm7taCWYgC1qBta2g/7Pu/qCkByOZBMBxx465HnthRkN7avtagUjiRzAQiHQrcHh0Uvfv5VqBOIGgBsqIawUiHwQ1UELpVmD6s5tpBSIfBDUQsXQrcHg0eX7zBNcKRIEIaiACBw4f0QNjyQ/W/9mT03p1QSvwk7QCUSCCGiiCXK1ArhWI00FQA6dobv6YfpWhFbihNdkK7O2M6fIWWoE4fQQ1UIBDr8/qp2PTGhqd1INj0zqYmKUViMgR1EAO6Vbg8N5JPfJMshW4evmZ6u2MqSce08b1a2gFIlL87QIWWdgKHB6d0tjkK5KSrcAPbVqnnjitQJQWQQ0o2Qp8+Mn9GsrYCuykFYiyIqhRs9KtwOHRKT381H5agQgWQY2aka0V2LbqHN1+zUXqiTfTCkSQCGpUtVytwN54TJfSCkTgCGpUnXQrcHh0Ug89caIVuOkyWoGoTAQ1Kl66FTg0OqWhPZN69LmXdcyl81eenWwFdsZ07TpagahcBDUqUroVOJy66snCVuDHaAWiyhDUqBi0AlGrCGoE7fmXTlwrkFYgahV/wxGUpVqBXCsQtYqgRtmd3Aqc1v7DR2gFAgsQ1CiLyUOvpz4edEo/f2q/jtAKBLIiqFES7q49Lx7S0J4pDe+d1K9fONEKfB+tQGBJBDUic2RuXr945qXU8WZagcCpyhnUZna2pIcknZXa/m53/1zUg6EyvfTqUd2/d+p3WoFcK7Cy7Ng1roHBMU3MJNTS2KD+vg5t7mot91g1K5896iOSbnD3w2ZWL+lhM/uRu/8i4tlQAZKtwFeTp9AtaAVyrcDKtWPXuLZu363E7LwkaXwmoa3bd0sSYV0mOYPa3V3S4dSX9ak/HuVQCNvc/DGNPPuyhvac3Aq8vIVWYDUYGBw7HtJpidl5DQyOEdRlktcxajOrk7RT0qWS/tndH8mwzRZJWySpra2tmDMiAOlW4PDopB6gFVjVJmYSBd2P6OUV1O4+L+kKM2uUdI+ZbXD3xxdts03SNknq7u5mj7sKZGoFrjreCmzWxvVNtAKrUEtjg8YzhDI/iMunoFeZu8+Y2YOSbpL0eI7NUWHSrcDh0UkN7aEVWKv6+zpOOkYtSQ31derv6yjjVLUtn7M+miTNpkK6QVKPpH+IfDKUxFKtwM/cHFdPPKb2NbQCa0n6ODRnfYQjnz3qCyR9PXWc+gxJ33P3e6MdC1GaPPT68ctR0QpEJpu7WgnmgORz1sevJXWVYBZEJFsrcO2qBv3RNW3qjcdoBQIB452gKpWrFdgTj2k9rUCgIhDUVeSlV4/qgb3JQxq0AoHqQVBXsIWtwOHRSe189kQr8N1dreqlFQhUBYK6wuRqBfbEY9rQSisQqCYEdQXI1gq89pLV+uBbL9aN8RhlBKCKEdSBytUKfOv6Jq2gFQjUBF7pgcjWCryUViBQ8wjqMkq3AodHpzS8d4pWIICMCOoSoxUIoFAEdcRoBQI4XQR1BJZqBfb3dai3k1YggPwR1EWy5LUCey7T29/YrKZzaQUCKBxBfYpoBQIoFYK6AEu1Av/8hvXqpRUIIAIEdQ5LtgK5ViCAEiCoM8jWCuyJx9TbSSsQQGmRNsrdCuyJN6urjVYggPKo2aDOdq3Aq9rPoxUIICg1FdSTh15PnaUxdaIVeNYyva2jSb2dMVqBAIJU1UGdqxXYE4/pqvZVOnMZrUAA4aq6oM7WCrwi1Qrsicd0WYxWIIDKURVBTSsQQDWryKBe2Aoc2jOpR587uRXYE2/WWy5ZQysQQFWomKCemz+mX+17OXkK3YJWYOcFyVZgT7xZG1reoDM4hQ5AlckZ1Ga2VtI3JJ0v6Zikbe7+xagHk060AodGJ/VghmsF3hCPqbVGW4E7do1rYHBMEzMJtTQ2qL+vQ5u7Wss9FoAI5LNHPSfpL939UTM7V9JOM/uJu++JYqBMrcDzzqmnFbjAjl3j2rp9txKz85Kk8ZmEtm7fLUmENVCFciaeu78o6cXU7VfMbFRSq6SiB/V9j/9WH/7mTknSJU3L9cGNF6s3HqMVuMjA4NjxkE5LzM5rYHCMoAaqUEG7pmbWLqlL0iMZHtsiaYsktbW1ndIw11ycvFbgjfGYLqYVmNXETKKg+wFUtrybHma2QtL3JX3S3Q8tftzdt7l7t7t3NzU1ndIw5y0/U3duXEdI55Dt0/r4FD+gOuUV1GZWr2RIf8vdt0c7EnLp7+tQw6JTDxvq69Tf11GmiQBEKZ+zPkzSVyWNuvvnox8JuaSPQ3PWB1Ab8jlGfZ2k90vabWaPpe77tLv/MLqxkMvmrlaCGagR+Zz18bAkTrkAgDLhY+MAIHAENQAEjqAGgMAR1AAQOIIaAAJHUANA4AhqAAgcQQ0AgSOoASBwBDUABI6gBoDAEdQAEDiCGgACR1ADQOAIagAIHEENAIEjqAEgcAQ1AASOoAaAwBHUABA4ghoAAkdQA0DgCGoACNyyXBuY2dck3SJpyt03RD8SUHw7do1rYHBMEzMJtTQ2qL+vQ5u7Wss9FpCXfPao/13STRHPAURmx65xbd2+W+MzCbmk8ZmEtm7frR27xss9GpCXnEHt7g9JeqkEswCRGBgcU2J2/qT7ErPzGhgcK9NEQGGKdozazLaY2YiZjUxPTxfr2wKnbWImUdD9QGiKFtTuvs3du929u6mpqVjfFjhtLY0NBd0PhIazPlD1+vs61FBfd9J9DfV16u/rKNNEQGFynvUBVLr02R2c9YFKlc/ped+RdL2kNWb2gqTPuftXox4MKKbNXa0EMypWzqB29/eWYhAAQGYcowaAwBHUABA4ghoAAkdQA0DgCGoACJy5e/G/qdm0pGdP8V9fI2l/EccpFuYqDHMVhrkKE+JcpzvTRe6esdYdSVCfDjMbcffucs+xGHMVhrkKw1yFCXGuKGfi0AcABI6gBoDAhRjU28o9QBbMVRjmKgxzFSbEuSKbKbhj1ACAk4W4Rw0AWICgBoDAlS2ozexrZjZlZo9nedzM7Etm9pSZ/drMrgxgpuvN7KCZPZb689moZ0o971oze8DMRs3sN2b2iQzblGO98pmr5GtmZmeb2S/N7H9Tc/1thm3OMrPvptbrETNrD2SuO8xsesF63Rn1XKnnrTOzXWZ2b4bHSr5Wec5VrrXaZ2a7U885kuHx4r8W3b0sfyRtknSlpMezPP4uST+SZJLeLOmRAGa6XtK9ZVirCyRdmbp9rqQnJHUGsF75zFXyNUutwYrU7XpJj0h686JtPiLpy6nbt0n6biBz3SHpn8rwd+xTkr6d6f9VOdYqz7nKtVb7JK1Z4vGivxbLtkftua9u/m5J3/CkX0hqNLMLyjxTWbj7i+7+aOr2K5JGJS3+FPxyrFc+c5Vcag0Op76sT/1Z/K75uyV9PXX7bkk3mpkFMFfJmdmFkm6W9JUsm5R8rfKcK1RFfy2GfIy6VdLzC75+QQGEgKRrU7+6/sjMLi/1k6d+7exScm9sobKu1xJzSWVYs9SvzI9JmpL0E3fPul7uPifpoKTVAcwlSX+Y+pX5bjNbG/VMkr4g6S5Jx7I8Xpa1ymMuqfRrJSV/uP7YzHaa2ZYMjxf9tRhyUGf6iV3uvY9Hlezjv0nSP0raUconN7MVkr4v6ZPufmjxwxn+lZKsV465yrJm7j7v7ldIulDS1Wa2YdEmZVmvPOb6b0nt7v57koZ0Yk82EmZ2i6Qpd9+51GYZ7ot0rfKcq6RrtcB17n6lpHdK+qiZbVr0eNHXK+SgfkHSwp+QF0qaKNMskiR3P5T+1dXdfyip3szWlOK5zaxeyTD8lrtvz7BJWdYr11zlXLPUc85IelDSTYseOr5eZrZM0htUwsNe2eZy9wPufiT15b9K+v2IR7lO0q1mtk/Sf0q6wcy+uWibcqxVzrnKsFbp551I/XNK0j2Srl60SdFfiyEH9Q8k/XHqHdQ3Szro7i+WcyAzOz99bM7MrlZy/Q6U4HlN0lcljbr757NsVvL1ymeucqyZmTWZWWPqdoOkHkl7F232A0l/krr9Hkn3e+qdoHLOtehY5q1KHvePjLtvdfcL3b1dyTcK73f32xdtVvK1ymeuUq9V6jmXm9m56duS3iFp8VliRX8t5ry4bVQsw9XNlXxzRe7+ZUk/VPLd06ckvSbpAwHM9B5Jf2Zmc5ISkm6L+i9synWS3i9pd+r4piR9WlLbgtlKvl55zlWONbtA0tfNrE7JHwzfc/d7zezvJI24+w+U/AHzH2b2lJJ7h7dFPFO+c33czG6VNJea644SzPU7AlirfOYqx1rFJN2T2vdYJunb7n6fmX1Yiu61SIUcAAIX8qEPAIAIagAIHkENAIEjqAEgcAQ1AASOoAaAwBHUABC4/we6Rt8BZqyKAwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.linear_model import LinearRegression\n",
    "X = x[:, np.newaxis]\n",
    "model = LinearRegression().fit(X, y)\n",
    "yfit = model.predict(X)\n",
    "plt.scatter(x, y)\n",
    "plt.plot(x, yfit);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> It's clear that we need a more sophisticated model to describe the relationship between $x$ and $y$.\n",
    "\n",
    "很显然我们需要更加复杂的模型来描述x和y之间的关系。\n",
    "\n",
    "> One approach to this is to transform the data, adding extra columns of features to drive more flexibility in the model.\n",
    "For example, we can add polynomial features to the data this way:\n",
    "\n",
    "一种方案是转换数据，通过增加额外的特征列来增加模型的灵活性。例如，我们可以如下方式增加多项式特征："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[  1.   1.   1.]\n",
      " [  2.   4.   8.]\n",
      " [  3.   9.  27.]\n",
      " [  4.  16.  64.]\n",
      " [  5.  25. 125.]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "poly = PolynomialFeatures(degree=3, include_bias=False)\n",
    "X2 = poly.fit_transform(X)\n",
    "print(X2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> The derived feature matrix has one column representing $x$, and a second column representing $x^2$, and a third column representing $x^3$.\n",
    "Computing a linear regression on this expanded input gives a much closer fit to our data:\n",
    "\n",
    "派生的矩阵中第一列代表$x$，第二列代表$x^2$，第三列代表$x^3$。在这个扩增输入上计算得到的线性回归得到对数据更好的一个拟合："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3dd3hUZcL+8e+TBgkkhJBQQgvN0GtoYtcVBFexo2svoOta3nVRcetv931FZXXV3XUFsZdVUWRXl6ZiQymGGkpCb6GFQBJC+uT5/ZGAyAaYgZk5Zyb357pyMcwc5tw8kHtOTnuMtRYREXGvCKcDiIjIiamoRURcTkUtIuJyKmoREZdTUYuIuFxUIN40OTnZpqWlBeKtRUTC0pIlS/ZZa1Pqei0gRZ2WlkZmZmYg3lpEJCwZY7Ye7zXt+hARcTkVtYiIy6moRURcTkUtIuJyKmoREZdTUYuIuNxJi9oYk26MWX7UV5Ex5sFghBMRES/Oo7bW5gB9AYwxkUAu8FGAc4mIhIQZy3J5cnY2uwrLaJnQkEcv6crofq39ug5fL3i5ENhorT3uidkiIvXFjGW5TJieRWmlB4DdRWVMmJ4F4Ney9nUf9Rjgn35bu4hICJs0J+dISR9WWulh0pwcv67H66I2xsQAlwHTjvP6WGNMpjEmMy8vz1/5RERca2dBqU/PnypftqgvAZZaa/fU9aK1doq1NsNam5GSUud9RUREwkpKfIM6n09NjPXrenwp6uvRbg8RkSOaxsX813Ox0ZGMH57u1/V4VdTGmDjgJ8B0v65dRCREfbUuj5w9BxndN5XWibEYoHViLBOv7OXMWR/W2hKgmV/XLCISojzVlokz19IuKY6nru5DTFRgrx3UlYkiIj76cOkOsncf5OER6QEvaVBRi4j4pLTCw9Nzc+jbNpFRvVoFZZ0qahERH7w8fxN7isr59ahuGGOCsk4VtYiIl/YVl/PiV5u4uHsLBqYlBW29KmoRES8999l6Sis9PHJJ16CuV0UtIuKFjXnFvLN4GzcMakenlMZBXbeKWkTEC0/OyiY2OpIHLuoS9HWrqEVETuL7LfuZu2YPd5/bkeTGdV82HkgqahGRE7DW8vjMtbRMaMgdZ3V0JIOKWkTkBGZm7WbZtgJ+efEZxMZEOpJBRS0ichwVVdU8NSebri3juap/G8dyqKhFRI7jrYVb2ZpfwoSR3YiMCM7FLXVRUYuI1KGwtJLn563n7C7JnHuGs/fYV1GLiNThhS83UFhayaNBvrilLipqEZFj7DhQwqvfbuGKfq3pkdrE6TgqahGRYz09dx0G+NXF/p2p5VSpqEVEjrIqt5CPluVy+1kd/D734alSUYuI1Dp8cUtSoxjuOa+T03GOUFGLiNT6MieP7zbmc/8FnUloGO10nCNU1CIiQJWnmomz1pLWLI4bBrd3Os6PqKhFRIAPluxg3Z5iHhnRNSjzIPrCXWlERBxQUlHFM5+uY0D7pozo2dLpOP9FRS0i9d5LX29m78FyHhvZNWjzIPpCRS0i9dreg2VM/nojl/RsyYD2wZsH0RdeFbUxJtEY84ExJtsYs9YYMzTQwUREguHZz9ZTUVXNwyOcv1T8eKK8XO45YLa19mpjTAwQF8BMIiJBsWHvQd77fjs3DWlPh+RGTsc5rpMWtTEmATgHuBXAWlsBVAQ2lohI4D0xK5u46EjuvzD48yD6wptdHx2BPOBVY8wyY8xUY8x/ffQYY8YaYzKNMZl5eXl+Dyoi4k8LN+Xz2dq93HN+J5IaxTgd54S8KeoooD/wD2ttP+AQ8OixC1lrp1hrM6y1GSkpzt67VUTkRKqray4VT23SkNuHdXA6zkl5U9Q7gB3W2kW1v/+AmuIWEQlJH6/cycodhTx0cToNo52ZB9EXJy1qa+1uYLsx5vD9/i4E1gQ0lYhIgJRXeZg0J4furRK4ol9rp+N4xduzPu4D3q4942MTcFvgIomIBM4b321lx4FS3rqjNxEOzoPoC6+K2lq7HMgIcBYRkYAqKKngr/PWc+4ZKZzVJdnpOF7TlYkiUm/8bd4GisurmDDSvRe31EVFLSL1wvb9JbyxYCtXD2hD15YJTsfxiYpaROqFp+bkEBEBv/yJO+ZB9IWKWkTC3ortBXy8Yid3nd2Rlk0aOh3HZypqEQlr1lr+b+ZakhvHMO5c98yD6AsVtYiEtc/W7mXx5v08cNEZNG7g7RnJ7qKiFpGwVeWp5olZa+mY0ogxA9s6HeeUqahFJGy9+/12NuYd4tERXYmODN26C93kIiInUFxexbOfrWNQWhI/6d7C6TinJTR32IiInMSUrzayr7iCqbd0c+U8iL7QFrWIhJ09RWW89M1mLu3dir5tE52Oc9pU1CISdp6Zu46q6moeHh5al4ofj4paRMJKzu6DTFuynZuHptGuWXhM76qiFpGwMnHWWho3iOK+Czo7HcVvVNQiEja+3bCPL3Py+MUFnUmMc/c8iL5QUYtIWDg8D2LrxFhuHprmdBy/UlGLSFiYsTyX1TuLeHhEaMyD6AsVtYiEvLJKD3+ek0Ov1k34ae9Up+P4nYpaRELeq99uYWdhGY+N7BYy8yD6QkUtIiFt/6EKXvhiAxd2bc7QTs2cjhMQKmoRCWnPf76eQxVVPHpJeFzcUhcVtYiErC37DvHWwq1cN7AdXVrEOx0nYFTUIhKynpqTTUxUBP/zky5ORwkoFbWIhKQlWw8wM2s3Y8/pSPP40JsH0Rde3ebUGLMFOAh4gCprbUYgQ4mInIi1NRe3pMQ34K6zOzodJ+B8uR/1+dbafQFLIiLipTmrd7Nk6wEmXtmLRiE6D6IvtOtDREJKpaeaJ2fn0KV5Y64Z0MbpOEHhbVFbYK4xZokxZmxdCxhjxhpjMo0xmXl5ef5LKCJylHcWbWPzvkNMGNmVqBCeB9EX3v4th1lr+wOXAPcaY845dgFr7RRrbYa1NiMlJcWvIUVEAIrKKnnu8/UM7diM89ObOx0naLwqamvtztpf9wIfAYMCGUpEpC4vfrmR/YcqeGxk6M+D6IuTFrUxppExJv7wY+BiYFWgg4mIHG1nQSkvz9/M6L6p9GrTxOk4QeXN4dIWwEe1n15RwDvW2tkBTSUicoyn567DAr8anu50lKA7aVFbazcBfYKQRUSkTmt2FjF92Q7Gnt2RNk3DYx5EX9SPQ6YiEtImzlpLk9hofn5++MyD6AsVtYi42lfr8vhm/T7uu6ALTWKjnY7jCBW1iLiWp9oyceZa2iXFcdOQ9k7HcYyKWkRc68OlO8jefZCHR6QTE1V/66r+/s1FxNVKKzw8PTeHvm0TGdWrldNxHKWiFhFXenn+JvYUlfPrUfXr4pa6qKhFxHX2FZfz4lebuLh7CwamJTkdx3EqahFxnec+W09ppYdHwngeRF+oqEXEVTbmFfPO4m3cMKgdnVIaOx3HFVTUIuIqT87KJjY6kgcuCu95EH2hohYR11i8eT9z1+zh7nM7kty4gdNxXENFLSKucHgexJYJDbnjrPCfB9EXKmoRcYX/ZO1i+fYCfnnxGcTGRDodx1VU1CLiuPIqD0/NzqFry3iu6l8/5kH0hYpaRBz31sJtbNtfwoSR3YiMqN8Xt9RFRS0ijiosreSv89Zzdpdkzj1D863WRUUtIo564YsNFJZWMuGSbk5HcS0VtYg4ZseBEl79bgtX9mtD99QEp+O4lopaRBzz5zk5GOBXw89wOoqrqahFxBFZOwqZsXwnd5zVgVZNYp2O42oqahEJusMXtyQ1iuHu8zo5Hcf1VNQiEnRf5OxlwaZ8HriwCwkN6+c8iL5QUYtIUFV5qpk4M5sOyY24YXA7p+OEBK+L2hgTaYxZZoz5JJCBRCS8TVuyg/V7i3lkRDrRkdpW9EaUD8s+AKwFAnIOzYxluTwxK5vdRWW0TGjIo5d0ZXS/1oFYlYg45FB5Fc98uo6M9k0Z3qOl03FChlcfZ8aYNsAoYGogQsxYlsuE6VnsLioDYHdRGY9+uJIZy3IDsToRcchL32wi72A5E0ZqHkRfePtzx7PAw0B1IEJMmpNDaaXnR8+VVVUzcdbaQKxORByw92AZU77exMheLRnQvqnTcULKSYvaGHMpsNdau+Qky401xmQaYzLz8vJ8CrGzoLTO5/cUlbNl3yGf3ktE3Okvn66n0lPNw8M1D6KvvNmiHgZcZozZArwLXGCMeevYhay1U6y1GdbajJQU326skppY98nuEQaumbyAdXsO+vR+IuIu6/cc5L3vt/Gzwe1JS27kdJyQc9KittZOsNa2sdamAWOAedbaG/0ZYvzwdGKjf3yj8NjoSB4e3hUDXDd5AatyC/25ShEJoidmZdMoJor7L9Q8iKfCFefGjO7XmolX9qJ1YiwGaJ0Yy8Qre3H3eZ14f9xQ4mKiuP6lhSzZut/pqCLiowUb8/k8ey8/P78zSY1inI4Tkoy11u9vmpGRYTMzM/32frkFpdw4dRF7isqYenMGZ3ZO9tt7i0jgVFdbLv/7t+QXlzPvV+fRMFpTbB2PMWaJtTajrtdcsUV9Mq0TY3lv3BDaNo3j1te+Z172HqcjiYgXPl65k6zcQn41PF0lfRpCoqgBmsc35N2xQ0hvEc+4N5cwM2uX05FE5ATKKmvmQeyRmsDovrp47XSETFEDNG0Uw9t3DaZPm0R+8c5SPlyyw+lIInIcbyzYQm5BKY+N7EaE5kE8LSFV1AAJDaN5445BnNkpmYemreDNhVudjiQixygoqeBv8zZwXnoKw3RM6bSFXFEDxMVEMfWWDC7s2pzfzljFlK83Oh1JRI7y13kbKC6v0jyIfhKSRQ3QMDqSF28awKjerXh8ZjZ/+XQdgTiDRUR8sy2/hDcWbOGaAW1JbxnvdJyw4Mvd81wnOjKC58f0IzY6kuc+X09JRRWP6WYvIo56ak42URER/PJizYPoLyFd1ACREYanrupNo5hIXvpmMyUVHv50eU8dvBBxwPLtBXyychf3X9CZFgkNnY4TNkK+qAEiIgx/uKwHsTFRvPjVRkorPDx1dW+idFNykaCx1vL4f9aS3DiGsedqHkR/CouiBjDG8MiIdBrFRPL0p+sorfTw3Jh+xESprEWC4dM1e1i8ZT//O7onjRuETbW4Qli1mDGG+y7swm9GdWPWqt2MezOTsmPucy0i/lfpqeaJ2dl0SmnEmIFtnY4TdsKqqA+78+yOPH5FL75cl8dtr37PofIqpyOJhLV3v9/OprxDPHpJN+1yDICwHdEbBrfjmWv7sHjLfm56eRGFpZVORxIJS8XlVTz32ToGdUjiom7NnY4TlsK2qAGu6NeGv9/Qn6zcQq6fspD84nKnI4mEnclfbWRfcQW/1qmxARPWRQ0womdLXro5g415xYyZspA9tRPoisjp211YxkvfbOKnfVLp0zbR6ThhK+yLGuC89Oa8fvsgdhaUcu3kBew4UOJ0JJGw8MynOVRXw8PD052OEtbqRVEDDOnYjLfuHMyBQxVc++ICNuUVOx1JJKRl7y5i2pId3Dy0PW2T4pyOE9bqTVED9GvXlHfHDqW8qpprJy8kZ7cmzRU5VRNnZhPfIIpfXNDZ6Shhr14VNUD31ATeGzeEyAi4bsoCVu4ocDqSSMiZv34fX63L474LupAYp3kQA63eFTVA5+bxTBt3Jo0bRHHDS4v4fosmzRXxVnW15fGZa2nTNJabz2zvdJx6oV4WNUC7ZnFMu3sozeMbcPPLi5m/fp/TkURCwkfLclmzq4jxw9NpEKV5EIOh3hY1QKsmsbw3bijtm8Vx+2vf89kaTZorciJllR6enptD7zZN+GnvVKfj1Bv1uqgBUuIb8O7YIXRrFc/dby3h4xU7nY4k4lqvfLuZnYVlmgcxyOp9UQMkxsXw1p2D6d+uKQ+8u4z3M7c7HUnEdfKLy/nHFxu5qFtzhnRs5nSceuWk9yI0xjQEvgYa1C7/gbX294EOFmzxDaN5/fZBjH0zk4c/WElphYdbzkxzOpaII2Ysy2XSnBx2FpSSmhjL+OHpLN9eQEmlh0cv6ep0vHrHm5vGlgMXWGuLjTHRwHxjzCxr7cIAZwu62JhIpt6SwS/eWcbv/72akgoP95ynG6BL/TJjWS4TpmdRWnuL4NyCUh75cCWVnmrGDGpH5+aaBzHYTrrrw9Y4fBlfdO1X2M4i2yAqkhd+1p/L+qTy5Oxsnp6bo0lzpV6ZNCfnSEkfVl5VjbXw4EVdHEpVv3k1DYMxJhJYAnQG/m6tXVTHMmOBsQDt2rXzZ8agi46M4C/X9SUuJpK/zttASYWH34zSncGkfthZUFrn8xZoHq95EJ3g1cFEa63HWtsXaAMMMsb0rGOZKdbaDGttRkpKir9zBl1khGHilb24bVgaL8/fzGMfZeGp1pa1hL/UxNg6n2/VRCXtFJ/O+rDWFgBfAiMCksZljDH87tLu3Ht+J/65eDsPvb+cKk+107FEAmr88HRio398IUt0pOGRETqI6BRvzvpIASqttQXGmFjgIuDJgCdzCWMM44d3JS4m6si+u+ev76crsiRsje7XGoCnZmezs7CMqAjDk1f2PvK8BJ83W9StgC+MMSuB74FPrbWfBDaW+9x7fmd+/9PuzFm9h7FvLKG0QpPmSvga3a81Y8/pCMBLN2dw5YA2Dieq3066RW2tXQn0C0IW17ttWAfiYiJ5dHoWt766mJdvHUjjBl4djxUJKfOy9/DMp+s4s1MzzksP/WNOoU5XJvrouoHtePa6vmRuPcDPpi6isEST5kr4KCqrZPy0Fdz+WiapibE8fkUvne3kAtocPAWX921NbHQkv3hnGWNeWsibdwwiuXEDp2OJnJb56/fx8Acr2F1Uxr3nd+L+C7voWIxLaIv6FF3coyVTb8lg875irpu8gN2FmjRXQtOh8ip+MyOLG19eRMOYSD6850zGD++qknYRFfVpOOeMFN64fTB7isq5ZvJ3bN+vSXMltCzalM8lz33D24u2cedZHZh5/9n0a9fU6VhyDBX1aRrUIYm37xxMUWkV17y4gI2aNFdCQFmlhz99soYxL9Xcsue9sUP5zaXdaRitrWg3UlH7QZ+2ibw7dghV1dVcN3kBa3cVOR1J5LiWbTvAyOe/4eX5m7lxcHtmPXA2gzokOR1LTkBF7SfdWiXw3rihREdGMGbKQpZv16S54i7lVR6emp3NVf/4jrIKD2/dMZg/je5JI51i6noqaj/qlNKY98cNpUlsNDdOXcSiTflORxIBYFVuIZf/7Vte+HIjVw9ow+z/OYezuiQ7HUu8pKL2s7ZJcbw/bigtEhpwy6uL+WpdntORpB6r9FTz3GfrGf33b8k/VMErt2bw1NV9SGgY7XQ08YGKOgBaNmnI++OG0jG5MXe9nsmc1budjiT10Lo9B7nyhe/4y2frGNW7FZ/+zzlc0LWF07HkFKioA6RZ4wb8864hdE9N4OdvL+Vfy3OdjiT1hKfa8uJXG7n0+fnkFpTyj5/157kx/UiMi3E6mpwiHUUIoCZx0bx152DufP17HnxvOaUVHsYMCu1JFcTdNuUV86tpK1i6rYARPVryv1f01FWzYUBFHWCNG0Tx2m2DuPutJTw6PYuSCg+3n9XB6VgSZqqrLa8v2MKTs7OJiYzg2ev6cnnfVN2nI0yoqIOgYXQkk28awAP/XM4fP1lDaaWHe8/v7HQsCRPb95cw/oMVLNy0n/PTU3jiqt60SNBsLOFERR0kDaIi+dsN/Rj/wUomzcnhUHkV44ena4tHTpm1ln8u3s7//WcNxhievKoX12a01f+pMKSiDqKoyAievqYPsTGRvPDlRkoqPPzu0u5EROgbS3yzq7CURz7M4ut1eQzr3Iwnr+pNm6ZxTseSAFFRB1lEhOH/RvckNjqSl+dvprTCw+NX9iJSZS1esNYyfWkuf/h4NVUey58u78HPBrfXh32YU1E7wBjDb0Z1o1GDKJ7/fD0llR6eubYP0ZE6W1KOb+/BMh6bvorP1u5hYFpTJl3dh7TkRk7HkiBQUTvEGMMvf3IGcTGRPDErm9IKD3+7oZ/uXiZ1+mTlTn47YxWHKjz8ZlQ3bhvWQT+F1SMqaofdfW4n4mIi+d2/VnPXG5lMvmkAcTH6Z5Ea+w9V8Nt/reI/K3fRp00Tnr62D52bxzsdS4JMjeACNw9NIzY6kkc+XMktryzmlVsHEq97MdR7n67Zw4TpWRSWVjB+eDrjzulIlHaP1Usqape4JqMtsTGRPPjucn42dRFv3D5Il/zWU4Wllfy/j1czfWku3Vol8Mbtg+iemuB0LHGQitpFLu2dSmx0JPe8vZQxUxby5h2DSYnX5b/1yVfr8njkg5XkFZdz3wWdue+CLsREaSu6vtP/AJe5sFsLXr11IFvzS7hu8gJ2FpQ6HUmCoLi8ignTs7jllcU0bhjF9HvO5KGL01XSAnhR1MaYtsaYL4wxa40xq40xDwQjWH02rHMyb94xiLyD5Vzz4gK25h/6r2VmLMtl2BPz6PDofxj2xDxmLNPd+ULVgo35jHj2a979fhvjzunIJ/edRZ+2iU7HEhfx5uO6CnjIWtsNGALca4zpHthYkpGWxDt3DeFQRRXXTl7Ahr0Hj7w2Y1kuE6ZnkVtQigVyC0qZMD1LZR1iSis8/OHfq7n+pYVERRimjRvKhJHddIqm/JeTFrW1dpe1dmnt44PAWqB1oIMJ9GrThPfGDqXawrWTF7J6ZyEAk+bkUFrp+dGypZUeJs3JcSKmnIIlW/cz8vlveO27Ldx6ZhozHzibjDRNMCt182kHmDEmDegHLKrjtbHGmExjTGZenqaf8pf0lvG8P24oDaMiuH7KQpZuO3Dc/dban+1+ZZUeJs5ayzUvLqCiqpp37hzMHy7roXPn5YS8LmpjTGPgQ+BBa23Rsa9ba6dYazOstRkpKSn+zFjvdUhuxPt3DyWpUQw3Tl1Es0Z1n7aXmhgb5GTii6wdhfz0r/OZ/NUmrhvYltkPns2ZnTXBrJycV0VtjImmpqTfttZOD2wkqUubpjWT5rZOjKWwrJKYYy58iI2OZPzwdIfSyYlUVFXzzKfrGP3CtxSVVfLqbQOZeGVvXdQkXvPmrA8DvAystdY+E/hIcjzNExry3rihnNEiHo+1JMXFYIDWibFMvLIXo/vp0IHbZO8u4ooXvuX5z9dzeZ9U5j54LuenN3c6loQYb3aMDQNuArKMMctrn3vMWjszcLHkeJIaxfDOXUO4/bXvWb69gD+O7snovqnaOnOZKk81k7/exLOfraNJbDSTbxrA8B4tnY4lIcpYa/3+phkZGTYzM9Pv7ys/OFRexV1vZPLdxnwiDPRs3YTBHZIY3KEZAzsk0SRWxe2UjXnFPPT+CpZvL2BUr1b88fIeNNMEs3ISxpgl1tqMOl9TUYeuKk81CzftZ9HmfBZt2s/y7QVUeKoxBrq1TGBwx5riHtwhiabHOQAp/lNdbXnl281MmpNDbEwkf7q8Jz/tk+p0LAkRKup6oqzSw7JtBUeKe+m2A5RXVQOQ3iL+h+LumESytvD8amv+IcZPW8niLfu5qFtzHr+yF83jNcGseE9FXU+VV3lYuaOQRZvyWbR5P5lbDhy5UKZTSiMGd6zZ2h7SsZlmrT5F1lreWrSNiTPXEmkMv7+sB1f1b60JZsVnKmoBoNJTTVZuIYtqd5dkbjlAcXkVAGnN4hjSsdmRrW6dk31yuQWlPPLBSuZv2MfZXZJ58qreGjc5ZSpqqVOVp5o1u4qOFPfizfspKqsp7rZJsUf2bw/p2Iw2TWO1lVjLWsu0JTv408dr8FjLr0d144ZB7TQ+clpU1OIVT7Ule/ePi/tASSUAqU0aHtlVMrhjM9KaxdXLYtpbVMaE6Vl8nr2XQR2S+PPVfWjXLM7pWBIGVNRySqqrLev3Fh85OLlocz77iisAaB7f4Kh93El0Smkc1sVtreXfK3byu3+tpqzSw8MjunLbmWlEaIJZ8RMVtfiFtZaNeYdYtDm/5rTATfnsPVgOQHLjGAZ1+OGskjOax4dNieUXl/Pbf61iZtZu+rVL5M/X9KFTSmOnY0mYUVFLQFhr2ZJfcuSskkWb8tlZWAZA07hoBqYlHdnq7tYqgcgQLO7Zq3bz64+yOFhWxYM/6cLYszXBrATGiYpa91aUU2aMoUNyIzokN2LMoHZYa9lxoJSFh4t7cz5z1+wBIKFhVG1x12x190hNcHXhFZZU8oePV/PRslx6pCbwzl19SW8Z73QsqadU1OI3xhjaJsXRNimOazLaAjX3yP5hH/d+Ps/eC0DjBlEMaN/0SHH3btOEaJcU9xc5e3n0w5XkF1fw4EVduPf8zq7JJvWTdn1IUO0pKjuym2TR5v1s2FsM1NymNSOt6ZGzSnq3aUKDqOBOSXWwrJL//WQt72Vu54wWjXnm2r70bN0kqBmk/tI+anGtfcXlLD6quLN318wN2SAqgv7tftji7tcuMaBzCX67YR8Pf7CSXYWljDu3Ew9e1CXoHxRSv6moJWQcOFTB4i37j5wOuGZXEdZCTGQEfdsmHinu/u0T/TJ9VUlFFU/MyuaNBVvpmNyIP1/bh/7tmvrhbyLiGxW1hKzC0koyt+w/srtk1c4iPNWWqAhD7zZNjpxVkpGWROMGvhV35pb9PDRtBVvzS7h9WAfGD08nNkZb0eIMFbWEjeLyqh8V98odhVRVWyIjDD1TE35U3Me7J3dZpYen5+Ywdf5m2jSNZdLVfRjSsVmQ/yYiP6ailrBVUlHF0q0Fdd6Tu3urhCMX4AzukERiXAwrthfw0LQVbNhbzA2D2/HYyG4+b4mLBIKKWuqNE92Tu0vzxmzad4jm8Q148qrenHNGisNpRX6gC16k3mgYHcnQTs0Y2qlmV8bR9+RevOUAgzsmMX54V01VJiFFRS1hrUFUJAPTkhiYluR0FJFTpsutRERcTkUtIuJyKmoREZc76T5qY8wrwKXAXmttz8BHEvG/GctymTQnh50FpaQmxjJ+eDqj+7V2OpaIV7zZon4NGBHgHCIBM2NZLhOmZ5FbUIqlZlLaCdOzmLEs1+loIl45aVFba78G9gchi0hATJqTQ2ml50fPlVZ6mDQnx6FEIr7x2z5qY8xYY0ymMSYzLy/PX28rctp2FpT69LyI2/itqAEHMGAAAAT2SURBVK21U6y1GdbajJQUXfEl7pGaGOvT8yJuo7M+JOyNH55O7DH3so6NjmT88HSHEon4RlcmStg7fHaHzvqQUOXN6Xn/BM4Dko0xO4DfW2tfDnQwEX8a3a+1illC1kmL2lp7fTCCiIhI3bSPWkTE5VTUIiIup6IWEXE5FbWIiMupqEVEXC4gcyYaY/KAraf4x5OBfX6M4y/K5Rvl8o1y+caNuU43U3trbZ2XdQekqE+HMSbzeBM8Okm5fKNcvlEu37gxVyAzadeHiIjLqahFRFzOjUU9xekAx6FcvlEu3yiXb9yYK2CZXLePWkREfsyNW9QiInIUFbWIiMs5VtTGmFeMMXuNMauO87oxxjxvjNlgjFlpjOnvgkznGWMKjTHLa79+F+hMtetta4z5whiz1hiz2hjzQB3LODFe3uQK+pgZYxoaYxYbY1bU5vp/dSzTwBjzXu14LTLGpLkk163GmLyjxuvOQOeqXW+kMWaZMeaTOl4L+lh5mcupsdpijMmqXWdmHa/7/3vRWuvIF3AO0B9YdZzXRwKzAAMMARa5INN5wCcOjFUroH/t43hgHdDdBePlTa6gj1ntGDSufRwNLAKGHLPMz4EXax+PAd5zSa5bgb858H/sl8A7df1bOTFWXuZyaqy2AMkneN3v34uObVHbk89ufjnwhq2xEEg0xrRyOJMjrLW7rLVLax8fBNYCx94F34nx8iZX0NWOQXHtb6Nrv449an458Hrt4w+AC40xxgW5gs4Y0wYYBUw9ziJBHysvc7mV378X3byPujWw/ajf78AFJQAMrf3RdZYxpkewV177Y2c/arbGjuboeJ0gFzgwZrU/Mi8H9gKfWmuPO17W2iqgEGjmglwAV9X+yPyBMaZtoDMBzwIPA9XHed2RsfIiFwR/rKDmw3WuMWaJMWZsHa/7/XvRzUVd1ye201sfS6m5Hr8P8FdgRjBXboxpDHwIPGitLTr25Tr+SFDG6yS5HBkza63HWtsXaAMMMsb0PGYRR8bLi1wfA2nW2t7AZ/ywJRsQxphLgb3W2iUnWqyO5wI6Vl7mCupYHWWYtbY/cAlwrzHmnGNe9/t4ubmodwBHf0K2AXY6lAUAa23R4R9drbUzgWhjTHIw1m2MiaamDN+21k6vYxFHxutkuZwcs9p1FgBfAiOOeenIeBljooAmBHG31/FyWWvzrbXltb99CRgQ4CjDgMuMMVuAd4ELjDFvHbOME2N10lwOjNXh9e6s/XUv8BEw6JhF/P696Oai/jdwc+0R1CFAobV2l5OBjDEtD++bM8YMomb88oOwXgO8DKy11j5znMWCPl7e5HJizIwxKcaYxNrHscBFQPYxi/0buKX28dXAPFt7JMjJXMfsy7yMmv3+AWOtnWCtbWOtTaPmQOE8a+2NxywW9LHyJlewx6p2nY2MMfGHHwMXA8eeJeb378WTTm4bKKaO2c2pObiCtfZFYCY1R083ACXAbS7IdDVwjzGmCigFxgT6P2ytYcBNQFbt/k2Ax4B2R2UL+nh5mcuJMWsFvG6MiaTmg+F9a+0nxpg/ApnW2n9T8wHzpjFmAzVbh2MCnMnbXPcbYy4Dqmpz3RqEXP/FBWPlTS4nxqoF8FHttkcU8I61drYx5m4I3PeiLiEXEXE5N+/6EBERVNQiIq6nohYRcTkVtYiIy6moRURcTkUtIuJyKmoREZf7/ynV9Gkz5OoKAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "model = LinearRegression().fit(X2, y)\n",
    "yfit = model.predict(X2)\n",
    "plt.scatter(x, y)\n",
    "plt.plot(x, yfit);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> This idea of improving a model not by changing the model, but by transforming the inputs, is fundamental to many of the more powerful machine learning methods.\n",
    "We explore this idea further in [In Depth: Linear Regression](05.06-Linear-Regression.ipynb) in the context of *basis function regression*.\n",
    "More generally, this is one motivational path to the powerful set of techniques known as *kernel methods*, which we will explore in [In-Depth: Support Vector Machines](05.07-Support-Vector-Machines.ipynb).\n",
    "\n",
    "上面这种不通过改变模型本身而是通过转换输入数据的方法，是很多强大的机器学习算法的基础。我们会在[深入：线性回归](05.06-Linear-Regression.ipynb)一节中在*基础函数回归*部分更加详细的讨论它。更普遍的情况下，这种方法属于一组强大的被称为*核方法*的行动步骤的一部分，我们会在[深入：支持向量机](05.07-Support-Vector-Machines.ipynb)中深入讨论它们。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imputation of Missing Data\n",
    "\n",
    "## 缺失数据插值\n",
    "\n",
    "> Another common need in feature engineering is handling of missing data.\n",
    "We discussed the handling of missing data in ``DataFrame``s in [Handling Missing Data](03.04-Missing-Values.ipynb), and saw that often the ``NaN`` value is used to mark missing values.\n",
    "For example, we might have a dataset that looks like this:\n",
    "\n",
    "特征工程中还有一个普遍需求是处理缺失数据。我们在`DataFrame`的[处理缺失数据](03.04-Missing-Values.ipynb)一节中讨论过它，而且我们也知道通常我们使用`NaN`来代表缺失值。例如我们有如下的数据集："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "from numpy import nan\n",
    "X = np.array([[ nan, 0,   3  ],\n",
    "              [ 3,   7,   9  ],\n",
    "              [ 3,   5,   2  ],\n",
    "              [ 4,   nan, 6  ],\n",
    "              [ 8,   8,   1  ]])\n",
    "y = np.array([14, 16, -1,  8, -5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> When applying a typical machine learning model to such data, we will need to first replace such missing data with some appropriate fill value.\n",
    "This is known as *imputation* of missing values, and strategies range from simple (e.g., replacing missing values with the mean of the column) to sophisticated (e.g., using matrix completion or a robust model to handle such data).\n",
    "\n",
    "如果我们想要将典型机器学习模型应用到这个数据上时，我们需要首先将缺失数据填充上值。这被成为缺失数据的*插值*，它的策略从简单（例如使用列均值填充缺失值）到复杂（例如使用矩阵补全或一个健壮的模型来处理这些数据）都有。\n",
    "\n",
    "> The sophisticated approaches tend to be very application-specific, and we won't dive into them here.\n",
    "For a baseline imputation approach, using the mean, median, or most frequent value, Scikit-Learn provides the ``Imputer`` class:\n",
    "\n",
    "复杂的方法一般都是应用场景相关的，我们在这里不会深入研究它们。对于插值的基础方法，如使用均值、中位数或最常见值，Scikit-Learn提供了`Imputer`类：\n",
    "\n",
    "译者注：`Imputer`类已经过时，0.22版本会被移除，下面使用了`sklearn.impute.SimpleImputer`替换了原代码中的`Imputer`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[4.5, 0. , 3. ],\n",
       "       [3. , 7. , 9. ],\n",
       "       [3. , 5. , 2. ],\n",
       "       [4. , 5. , 6. ],\n",
       "       [8. , 8. , 1. ]])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.impute import SimpleImputer\n",
    "imp = SimpleImputer(strategy='mean')\n",
    "X2 = imp.fit_transform(X)\n",
    "X2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> We see that in the resulting data, the two missing values have been replaced with the mean of the remaining values in the column. This imputed data can then be fed directly into, for example, a ``LinearRegression`` estimator:\n",
    "\n",
    "我们可以从结果看到，两个缺失值被替换成了该列的平均值。处理完后的数据就能直接被传递给评估器模型处理，例如线性回归`LinearRegression`："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([13.14869292, 14.3784627 , -1.15539732, 10.96606197, -5.33782027])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = LinearRegression().fit(X2, y)\n",
    "model.predict(X2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Pipelines\n",
    "\n",
    "## 特征管道操作\n",
    "\n",
    "> With any of the preceding examples, it can quickly become tedious to do the transformations by hand, especially if you wish to string together multiple steps.\n",
    "For example, we might want a processing pipeline that looks something like this:\n",
    "\n",
    "> 1. Impute missing values using the mean\n",
    "2. Transform features to quadratic\n",
    "3. Fit a linear regression\n",
    "\n",
    "看完前面的例子之后，如果每次我们都要手动处理特征并且实例化模型的话，这项工作会变得很乏味，尤其是如果你希望将多个步骤串联在一起的情况下。例如我们可能希望按顺序完成下列任务：\n",
    "\n",
    "1. 使用均值插入缺失值\n",
    "2. 将特征转换为二阶多项式\n",
    "3. 选择和实例化线性回归模型\n",
    "\n",
    "> To streamline this type of processing pipeline, Scikit-Learn provides a ``Pipeline`` object, which can be used as follows:\n",
    "\n",
    "你可以将这些操作使用管道连接起来，Scikit-Learn提供了一个`Pipeline`对象，使用方式如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.pipeline import make_pipeline\n",
    "\n",
    "model = make_pipeline(SimpleImputer(strategy='mean'),\n",
    "                      PolynomialFeatures(degree=2),\n",
    "                      LinearRegression())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> This pipeline looks and acts like a standard Scikit-Learn object, and will apply all the specified steps to any input data.\n",
    "\n",
    "管道对象看起来就像标准的Scikit-Learn对象一样，它能将其中的所有操作步骤应用在（拟合）任意数据数据。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[14 16 -1  8 -5]\n",
      "[14. 16. -1.  8. -5.]\n"
     ]
    }
   ],
   "source": [
    "model.fit(X, y)  # X with missing values, from above\n",
    "print(y)\n",
    "print(model.predict(X))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> All the steps of the model are applied automatically.\n",
    "Notice that for the simplicity of this demonstration, we've applied the model to the data it was trained on; this is why it was able to perfectly predict the result (refer back to [Hyperparameters and Model Validation](05.03-Hyperparameters-and-Model-Validation.ipynb) for further discussion of this).\n",
    "\n",
    "所有的模型操作步骤都会自动被应用。注意这里为了简单起见，我们将模型预测应用到了训练它的数据上；这也是为什么模型能完美的预测结果的原因（参见[超参数和模型验证](05.03-Hyperparameters-and-Model-Validation.ipynb)）。\n",
    "\n",
    "> For some examples of Scikit-Learn pipelines in action, see the following section on naive Bayes classification, as well as [In Depth: Linear Regression](05.06-Linear-Regression.ipynb), and [In-Depth: Support Vector Machines](05.07-Support-Vector-Machines.ipynb).\n",
    "\n",
    "要学习更多Scikit-Learn管道的实际例子，参看接下来的小节朴素贝叶斯分类，以及[深入：线性回归](05.06-Linear-Regression.ipynb)和[深入：支持向量机](05.07-Support-Vector-Machines.ipynb)。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--NAVIGATION-->\n",
    "< [超参数及模型验证](05.03-Hyperparameters-and-Model-Validation.ipynb) | [目录](Index.ipynb) | [深入：朴素贝叶斯分类](05.05-Naive-Bayes.ipynb) >\n",
    "\n",
    "<a href=\"https://colab.research.google.com/github/wangyingsm/Python-Data-Science-Handbook/blob/master/notebooks/05.04-Feature-Engineering.ipynb\"><img align=\"left\" src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\" title=\"Open and Execute in Google Colaboratory\"></a>\n"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}