Day_044_HW.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [作業重點]\n",
    "確保你了解隨機森林模型中每個超參數的意義，並觀察調整超參數對結果的影響"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 作業\n",
    "\n",
    "1. 試著調整 RandomForestClassifier(...) 中的參數，並觀察是否會改變結果？\n",
    "2. 改用其他資料集 (boston, wine)，並與回歸模型與決策樹的結果進行比較"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1.調整參數練習"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import datasets, metrics\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 讀取鳶尾花資料集\n",
    "iris = datasets.load_iris()\n",
    "\n",
    "# 切分訓練集/測試集\n",
    "x_train, x_test, y_train, y_test = train_test_split(iris.data, iris.target, test_size=0.25, random_state=4)\n",
    "\n",
    "# 建立模型 (使用 20 顆樹，每棵樹的最大深度為 4)\n",
    "clf = RandomForestClassifier(n_estimators=20, max_depth=4)\n",
    "\n",
    "# 訓練模型\n",
    "clf.fit(x_train, y_train)\n",
    "\n",
    "# 預測測試集\n",
    "y_pred = clf.predict(x_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy:  0.9736842105263158\n",
      "['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)']\n",
      "Feature importance:  [0.09805454 0.03176943 0.45623413 0.41394189]\n"
     ]
    }
   ],
   "source": [
    "acc = metrics.accuracy_score(y_test, y_pred)\n",
    "print(\"Accuracy: \", acc)\n",
    "print(iris.feature_names)\n",
    "print(\"Feature importance: \", clf.feature_importances_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(150, 4)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "iris.data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy:  0.9473684210526315\n",
      "['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)']\n",
      "Feature importance:  [0.07537658 0.01321336 0.43323539 0.47817466]\n"
     ]
    }
   ],
   "source": [
    "clf = (RandomForestClassifier(n_estimators=20, \n",
    "                              max_depth=3,\n",
    "                              criterion='gini',\n",
    "                              min_samples_split=2,\n",
    "                              min_samples_leaf=1,\n",
    "                              min_weight_fraction_leaf=0.0,\n",
    "                              max_features='auto',\n",
    "                              max_leaf_nodes=None,\n",
    "                              min_impurity_decrease=0.0,\n",
    "                              min_impurity_split=None,\n",
    "                              bootstrap=True,\n",
    "                              oob_score=False,\n",
    "                              n_jobs=None,\n",
    "                              random_state=None,\n",
    "                              verbose=0,\n",
    "                              warm_start=False,\n",
    "                              class_weight=None,\n",
    "                              ccp_alpha=0.0,\n",
    "                              max_samples=None))\n",
    "                              \n",
    "# 訓練模型\n",
    "clf.fit(x_train, y_train)\n",
    "\n",
    "# 預測測試集\n",
    "y_pred = clf.predict(x_test)\n",
    "\n",
    "\n",
    "acc = metrics.accuracy_score(y_test, y_pred)\n",
    "print(\"Accuracy: \", acc)\n",
    "print(iris.feature_names)\n",
    "print(\"Feature importance: \", clf.feature_importances_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy:  0.9736842105263158\n",
      "['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)']\n",
      "Feature importance:  [0.18924754 0.02661471 0.28769217 0.49644557]\n"
     ]
    }
   ],
   "source": [
    "clf = (RandomForestClassifier(n_estimators=10, \n",
    "                              max_depth=3,\n",
    "                              criterion='gini',\n",
    "                              min_samples_split=2,\n",
    "                              min_samples_leaf=1,\n",
    "                              min_weight_fraction_leaf=0.0,\n",
    "                              max_features='auto',\n",
    "                              max_leaf_nodes=None,\n",
    "                              min_impurity_decrease=0.0,\n",
    "                              min_impurity_split=None,\n",
    "                              bootstrap=True,\n",
    "                              oob_score=False,\n",
    "                              n_jobs=None,\n",
    "                              random_state=None,\n",
    "                              verbose=0,\n",
    "                              warm_start=False,\n",
    "                              class_weight=None,\n",
    "                              ccp_alpha=0.0,\n",
    "                              max_samples=None))\n",
    "                              \n",
    "# 訓練模型\n",
    "clf.fit(x_train, y_train)\n",
    "\n",
    "# 預測測試集\n",
    "y_pred = clf.predict(x_test)\n",
    "\n",
    "\n",
    "acc = metrics.accuracy_score(y_test, y_pred)\n",
    "print(\"Accuracy: \", acc)\n",
    "print(iris.feature_names)\n",
    "print(\"Feature importance: \", clf.feature_importances_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "由於特徵數不多，多幾棵樹的情形下都能建立出和單一決策樹一樣的效果，除非對深度進行調整才有可能調差些。但基本上要再改進很難了。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2.改用回歸模型測試並與單純回歸與決策樹回歸模型比較"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(506, 13)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "array(['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD',\n",
       "       'TAX', 'PTRATIO', 'B', 'LSTAT'], dtype='<U7')"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "boston = datasets.load_boston()\n",
    "display(boston.data.shape)\n",
    "display(boston.feature_names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 切分訓練集/測試集\n",
    "x_train, x_test, y_train, y_test = train_test_split(boston.data, boston.target, test_size=0.25, random_state=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MSE: 19.846619219455487\n",
      "Feature importance:  [0.04753098 0.00056047 0.00342789 0.00148296 0.0173673  0.47918477\n",
      " 0.00563211 0.04371651 0.00112268 0.01547843 0.0213062  0.00415309\n",
      " 0.35903662]\n"
     ]
    }
   ],
   "source": [
    "# 建立模型 (使用 20 顆樹，每棵樹的最大深度為 4)\n",
    "clf = RandomForestRegressor(n_estimators=20, max_depth=4)\n",
    "\n",
    "# 訓練模型\n",
    "clf.fit(x_train, y_train)\n",
    "\n",
    "# 預測測試集\n",
    "y_pred = clf.predict(x_test)\n",
    "\n",
    "mse = metrics.mean_squared_error(y_test, y_pred)\n",
    "print(\"MSE:\", mse)\n",
    "print(\"Feature importance: \", clf.feature_importances_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MSE:  29.00897637795276\n",
      "['CRIM' 'ZN' 'INDUS' 'CHAS' 'NOX' 'RM' 'AGE' 'DIS' 'RAD' 'TAX' 'PTRATIO'\n",
      " 'B' 'LSTAT']\n",
      "Feature importance:  [4.77016529e-02 2.27280782e-03 1.90079095e-03 9.55872684e-03\n",
      " 3.29277723e-02 5.42938641e-01 1.55779585e-02 5.87100761e-02\n",
      " 5.08254858e-04 9.78586774e-03 2.95016216e-02 2.72660375e-02\n",
      " 2.21349792e-01]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.tree import DecisionTreeRegressor\n",
    "# 建立模型\n",
    "clf = (DecisionTreeRegressor())\n",
    "\n",
    "# 訓練模型\n",
    "clf.fit(x_train, y_train)\n",
    "\n",
    "# 預測測試集\n",
    "y_pred = clf.predict(x_test)\n",
    "\n",
    "mse = metrics.mean_squared_error(y_test, y_pred)\n",
    "print(\"MSE: \", mse)\n",
    "print(boston.feature_names)\n",
    "print(\"Feature importance: \", clf.feature_importances_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MSE:  26.95142562423582\n"
     ]
    }
   ],
   "source": [
    "from sklearn.linear_model import LinearRegression\n",
    "\n",
    "Reg = LinearRegression()\n",
    "Reg.fit(x_train, y_train)\n",
    "y_pred = Reg.predict(x_test)\n",
    "\n",
    "mse = metrics.mean_squared_error(y_test, y_pred)\n",
    "\n",
    "print(\"MSE: \", mse)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用RF, DT, LR下各自的MSE一目瞭然\n",
    "分別為19.84, 29.00, 26.95\n",
    "可見RF的效果最佳(而且可以發現選用的特徵權重上不一樣)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}