010-评估算法.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "74c6c3d1",
   "metadata": {},
   "source": [
    "# 选择模型\n",
    "\n",
    "## 评估算法\n",
    "\n",
    "### 评估算法的方法\n",
    "\n",
    "在评估机器学习算法的时候，不将训练数据集直接作为评估数据集，最直接原因计算过度拟合，不能有效的发现算法模型不足。    \n",
    "假设算法记住了所有的训练数据集，采用相同的数据集来评价算法时，会得到很高的评分，但是在预测新数据集时会表现的很糟糕。    \n",
    "四种不同的分离数据集方法，用来分离训练数据集和评估数据集，并用其评估算法模型：\n",
    "- 分离训练数据集和评估数据集\n",
    "- k折交叉验证分离\n",
    "- 弃一交叉验证分离\n",
    "- 重复随机评估、训练数据集分离"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "9c8fa314",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-01T13:14:04.467655Z",
     "start_time": "2021-12-01T13:14:04.433745Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "算法评估结果： 80.469%\n"
     ]
    }
   ],
   "source": [
    "# 分离数据集和评估数据集\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "filename = 'pima_data.csv'\n",
    "names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class']\n",
    "data = pd.read_csv(filename, names=names)\n",
    "array = data.values\n",
    "X = array[:, 0:8]\n",
    "Y = array[:, 8]\n",
    "test_size = 0.3333\n",
    "seed = 4\n",
    "X_train, X_test, Y_traing, Y_test = train_test_split(X, Y, test_size=test_size, random_state=seed)\n",
    "model = LogisticRegression()\n",
    "model.fit(X_train, Y_traing)\n",
    "result = model.score(X_test, Y_test)\n",
    "print('算法评估结果： %.3f%%' % (result * 100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "b34baeec",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-01T13:31:12.735974Z",
     "start_time": "2021-12-01T13:31:12.559452Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "算法评估结果： 77.475% (5.206%)\n"
     ]
    }
   ],
   "source": [
    "# k折交叉验证分离\n",
    "from sklearn.model_selection import KFold,cross_val_score\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "num_folds = 9\n",
    "seed = 7\n",
    "kfold = KFold(n_splits=num_folds, random_state=seed)\n",
    "model = LogisticRegression()\n",
    "result = cross_val_score(model, X, Y, cv=kfold)\n",
    "print('算法评估结果： %.3f%% (%.3f%%)' % (result.mean() * 100, result.std() * 100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "59cb3fa2",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-01T13:33:54.489578Z",
     "start_time": "2021-12-01T13:33:45.794716Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "算法评估结果： 76.823% (42.196%)\n"
     ]
    }
   ],
   "source": [
    "# 弃一交叉验证分离\n",
    "\n",
    "from sklearn.model_selection import LeaveOneOut\n",
    "\n",
    "loocv = LeaveOneOut()\n",
    "result = cross_val_score(model, X, Y, cv=loocv)\n",
    "print('算法评估结果： %.3f%% (%.3f%%)' % (result.mean() * 100, result.std() * 100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "1e01e910",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-01T13:36:39.077620Z",
     "start_time": "2021-12-01T13:36:38.952955Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "算法评估结果： 76.535% (1.672%)\n"
     ]
    }
   ],
   "source": [
    "# 重复随机分离评估数据集与训练数据集\n",
    "from sklearn.model_selection import ShuffleSplit\n",
    "\n",
    "test_size = 0.33\n",
    "n_splits = 10\n",
    "kfold = ShuffleSplit(n_splits=n_splits, test_size=test_size, random_state=seed)\n",
    "result = cross_val_score(model, X, Y, cv=kfold)\n",
    "print('算法评估结果： %.3f%% (%.3f%%)' % (result.mean() * 100, result.std() * 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "156fc106",
   "metadata": {},
   "source": [
    "## 总结\n",
    "\n",
    "- k折交叉验证一般选3,5,10；不知道如何设置就选10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ea6c24f",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}