t81_558_class_05_2_kfold.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_05_2_kfold.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 5: Regularization and Dropout**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 5 Material\n",
    "\n",
    "* Part 5.1: Part 5.1: Introduction to Regularization: Ridge and Lasso [[Video]](https://www.youtube.com/watch?v=jfgRtCYjoBs&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_05_1_reg_ridge_lasso.ipynb)\n",
    "* **Part 5.2: Using K-Fold Cross Validation with Keras** [[Video]](https://www.youtube.com/watch?v=maiQf8ray_s&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_05_2_kfold.ipynb)\n",
    "* Part 5.3: Using L1 and L2 Regularization with Keras to Decrease Overfitting [[Video]](https://www.youtube.com/watch?v=JEWzWv1fBFQ&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_05_3_keras_l1_l2.ipynb)\n",
    "* Part 5.4: Drop Out for Keras to Decrease Overfitting [[Video]](https://www.youtube.com/watch?v=bRyOi0L6Rs8&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_05_4_dropout.ipynb)\n",
    "* Part 5.5: Benchmarking Keras Deep Learning Regularization Techniques [[Video]](https://www.youtube.com/watch?v=1NLBwPumUAs&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_05_5_bootstrap.ipynb)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running the correct version of TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: not using Google CoLab\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    %tensorflow_version 2.x\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 5.2: Using K-Fold Cross-validation with Keras\n",
    "\n",
    "Cross-validation can be used for a variety of purposes in predictive modeling.  These include:\n",
    "\n",
    "* Generating out-of-sample predictions from a neural network\n",
    "* Estimate a good number of epochs to train a neural network for (early stopping)\n",
    "* Evaluate the effectiveness of certain hyperparameters, such as activation functions, neuron counts, and layer counts\n",
    "\n",
    "Cross-validation uses a number of folds, and multiple models, to provide each segment of data a chance to serve as both the validation and training set. Cross validation is shown in Figure 5.CROSS.\n",
    "\n",
    "**Figure 5.CROSS: K-Fold Crossvalidation**\n",
    "![K-Fold Crossvalidation](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_1_kfold.png \"K-Fold Crossvalidation\")\n",
    "\n",
    "It is important to note that there will be one model (neural network) for each fold. To generate predictions for new data, which is data not present in the training set, predictions from the fold models can be handled in several ways:\n",
    "\n",
    "* Choose the model that had the highest validation score as the final model.\n",
    "* Preset new data to the 5 models (one for each fold) and average the result (this is an [ensemble](https://en.wikipedia.org/wiki/Ensemble_learning)).\n",
    "* Retrain a new model (using the same settings as the cross-validation) on the entire dataset.  Train for as many epochs, and with the same hidden layer structure.\n",
    "\n",
    "Generally, I prefer the last approach and will retrain a model on the entire data set once I have selected hyper-parameters.  Of course, I will always set aside a final holdout set for model validation that I do not use in any aspect of the training process.\n",
    "\n",
    "### Regression vs Classification K-Fold Cross-Validation\n",
    "\n",
    "Regression and classification are handled somewhat differently with regards to cross-validation.  Regression is the simpler case where you can simply break up the data set into K folds with little regard for where each item lands.  For regression it is best that the data items fall into the folds as randomly as possible.  It is also important to remember that not every fold will necessarily have exactly the same number of data items.  It is not always possible for the data set to be evenly divided into K folds.  For regression cross-validation we will use the Scikit-Learn class **KFold**.\n",
    "\n",
    "Cross validation for classification could also use the **KFold** object; however, this technique would not ensure that the class balance remains the same in each fold as it was in the original.  It is very important that the balance of classes that a model was trained on remains the same (or similar) to the training set.  A drift in this distribution is one of the most important things to monitor after a trained model has been placed into actual use.  Because of this, we want to make sure that the cross-validation itself does not introduce an unintended shift. This is referred to as stratified sampling and is accomplished by using the Scikit-Learn object **StratifiedKFold** in place of **KFold** whenever you are using classification.  In summary, the following two objects in Scikit-Learn should be used:\n",
    "\n",
    "* **KFold** When dealing with a regression problem.\n",
    "* **StratifiedKFold** When dealing with a classification problem.\n",
    "\n",
    "The following two sections demonstrate cross-validation with classification and regression. \n",
    "\n",
    "### Out-of-Sample Regression Predictions with K-Fold Cross-Validation\n",
    "\n",
    "The following code trains the simple dataset using a 5-fold cross-validation.  The expected performance of a neural network, of the type trained here, would be the score for the generated out-of-sample predictions.  We begin by preparing a feature vector using the jh-simple-dataset to predict age.  This is a regression problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from scipy.stats import zscore\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Read the data set\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/jh-simple-dataset.csv\",\n",
    "    na_values=['NA','?'])\n",
    "\n",
    "# Generate dummies for job\n",
    "df = pd.concat([df,pd.get_dummies(df['job'],prefix=\"job\")],axis=1)\n",
    "df.drop('job', axis=1, inplace=True)\n",
    "\n",
    "# Generate dummies for area\n",
    "df = pd.concat([df,pd.get_dummies(df['area'],prefix=\"area\")],axis=1)\n",
    "df.drop('area', axis=1, inplace=True)\n",
    "\n",
    "# Generate dummies for product\n",
    "df = pd.concat([df,pd.get_dummies(df['product'],prefix=\"product\")],axis=1)\n",
    "df.drop('product', axis=1, inplace=True)\n",
    "\n",
    "# Missing values for income\n",
    "med = df['income'].median()\n",
    "df['income'] = df['income'].fillna(med)\n",
    "\n",
    "# Standardize ranges\n",
    "df['income'] = zscore(df['income'])\n",
    "df['aspect'] = zscore(df['aspect'])\n",
    "df['save_rate'] = zscore(df['save_rate'])\n",
    "df['subscriptions'] = zscore(df['subscriptions'])\n",
    "\n",
    "# Convert to numpy - Classification\n",
    "x_columns = df.columns.drop('age').drop('id')\n",
    "x = df[x_columns].values\n",
    "y = df['age'].values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the feature vector is created a 5-fold cross-validation can be performed to generate out of sample predictions.  We will assume 500 epochs, and not use early stopping.  Later we will see how we can estimate a more optimal epoch count."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fold #1\n",
      "Fold score (RMSE): 0.6245484893737087\n",
      "Fold #2\n",
      "Fold score (RMSE): 0.5802295511082306\n",
      "Fold #3\n",
      "Fold score (RMSE): 0.6300965769274195\n",
      "Fold #4\n",
      "Fold score (RMSE): 0.4550931884841248\n",
      "Fold #5\n",
      "Fold score (RMSE): 1.0517027192572377\n",
      "Final, out of sample score (RMSE): 0.6981314007708873\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import os\n",
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "from scipy.stats import zscore\n",
    "from sklearn.model_selection import KFold\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "# Cross-Validate\n",
    "kf = KFold(5, shuffle=True, random_state=42) # Use for KFold classification\n",
    "    \n",
    "oos_y = []\n",
    "oos_pred = []\n",
    "\n",
    "fold = 0\n",
    "for train, test in kf.split(x):\n",
    "    fold+=1\n",
    "    print(f\"Fold #{fold}\")\n",
    "        \n",
    "    x_train = x[train]\n",
    "    y_train = y[train]\n",
    "    x_test = x[test]\n",
    "    y_test = y[test]\n",
    "    \n",
    "    model = Sequential()\n",
    "    model.add(Dense(20, input_dim=x.shape[1], activation='relu'))\n",
    "    model.add(Dense(10, activation='relu'))\n",
    "    model.add(Dense(1))\n",
    "    model.compile(loss='mean_squared_error', optimizer='adam')\n",
    "    \n",
    "    model.fit(x_train,y_train,validation_data=(x_test,y_test),verbose=0,\n",
    "              epochs=500)\n",
    "    \n",
    "    pred = model.predict(x_test)\n",
    "    \n",
    "    oos_y.append(y_test)\n",
    "    oos_pred.append(pred)    \n",
    "\n",
    "    # Measure this fold's RMSE\n",
    "    score = np.sqrt(metrics.mean_squared_error(pred,y_test))\n",
    "    print(f\"Fold score (RMSE): {score}\")\n",
    "\n",
    "# Build the oos prediction list and calculate the error.\n",
    "oos_y = np.concatenate(oos_y)\n",
    "oos_pred = np.concatenate(oos_pred)\n",
    "score = np.sqrt(metrics.mean_squared_error(oos_pred,oos_y))\n",
    "print(f\"Final, out of sample score (RMSE): {score}\")    \n",
    "    \n",
    "# Write the cross-validated prediction\n",
    "oos_y = pd.DataFrame(oos_y)\n",
    "oos_pred = pd.DataFrame(oos_pred)\n",
    "oosDF = pd.concat( [df, oos_y, oos_pred],axis=1 )\n",
    "#oosDF.to_csv(filename_write,index=False)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, the above code also reports the average number of epochs needed.  A common technique is to then train on the entire dataset for the average number of epochs needed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Classification with Stratified K-Fold Cross-Validation\n",
    "\n",
    "The following code trains and fits the jh-simple-dataset dataset with cross-validation to generate out-of-sample .  It also writes out the out of sample (predictions on the test set) results.\n",
    "\n",
    "It is good to perform a stratified k-fold cross validation with classification data.  This ensures that the percentages of each class remains the same across all folds.  To do this, make use of the **StratifiedKFold** object, instead of the **KFold** object used in regression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from scipy.stats import zscore\n",
    "\n",
    "# Read the data set\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/jh-simple-dataset.csv\",\n",
    "    na_values=['NA','?'])\n",
    "\n",
    "# Generate dummies for job\n",
    "df = pd.concat([df,pd.get_dummies(df['job'],prefix=\"job\")],axis=1)\n",
    "df.drop('job', axis=1, inplace=True)\n",
    "\n",
    "# Generate dummies for area\n",
    "df = pd.concat([df,pd.get_dummies(df['area'],prefix=\"area\")],axis=1)\n",
    "df.drop('area', axis=1, inplace=True)\n",
    "\n",
    "# Missing values for income\n",
    "med = df['income'].median()\n",
    "df['income'] = df['income'].fillna(med)\n",
    "\n",
    "# Standardize ranges\n",
    "df['income'] = zscore(df['income'])\n",
    "df['aspect'] = zscore(df['aspect'])\n",
    "df['save_rate'] = zscore(df['save_rate'])\n",
    "df['age'] = zscore(df['age'])\n",
    "df['subscriptions'] = zscore(df['subscriptions'])\n",
    "\n",
    "# Convert to numpy - Classification\n",
    "x_columns = df.columns.drop('product').drop('id')\n",
    "x = df[x_columns].values\n",
    "dummies = pd.get_dummies(df['product']) # Classification\n",
    "products = dummies.columns\n",
    "y = dummies.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will assume 500 epochs, and not use early stopping.  Later we will see how we can estimate a more optimal epoch count."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fold #1\n",
      "Fold score (accuracy): 0.6766169154228856\n",
      "Fold #2\n",
      "Fold score (accuracy): 0.6691542288557214\n",
      "Fold #3\n",
      "Fold score (accuracy): 0.6907730673316709\n",
      "Fold #4\n",
      "Fold score (accuracy): 0.6733668341708543\n",
      "Fold #5\n",
      "Fold score (accuracy): 0.654911838790932\n",
      "Final score (accuracy): 0.673\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import os\n",
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "\n",
    "# np.argmax(pred,axis=1)\n",
    "# Cross-validate\n",
    "# Use for StratifiedKFold classification\n",
    "kf = StratifiedKFold(5, shuffle=True, random_state=42) \n",
    "    \n",
    "oos_y = []\n",
    "oos_pred = []\n",
    "fold = 0\n",
    "\n",
    "# Must specify y StratifiedKFold for\n",
    "for train, test in kf.split(x,df['product']):  \n",
    "    fold+=1\n",
    "    print(f\"Fold #{fold}\")\n",
    "        \n",
    "    x_train = x[train]\n",
    "    y_train = y[train]\n",
    "    x_test = x[test]\n",
    "    y_test = y[test]\n",
    "    \n",
    "    model = Sequential()\n",
    "    model.add(Dense(50, input_dim=x.shape[1], activation='relu')) # Hidden 1\n",
    "    model.add(Dense(25, activation='relu')) # Hidden 2\n",
    "    model.add(Dense(y.shape[1],activation='softmax')) # Output\n",
    "    model.compile(loss='categorical_crossentropy', optimizer='adam')\n",
    "\n",
    "    model.fit(x_train,y_train,validation_data=(x_test,y_test),verbose=0,epochs=500)\n",
    "    \n",
    "    pred = model.predict(x_test)\n",
    "    \n",
    "    oos_y.append(y_test)\n",
    "    # raw probabilities to chosen class (highest probability)\n",
    "    pred = np.argmax(pred,axis=1) \n",
    "    oos_pred.append(pred)  \n",
    "\n",
    "    # Measure this fold's accuracy\n",
    "    y_compare = np.argmax(y_test,axis=1) # For accuracy calculation\n",
    "    score = metrics.accuracy_score(y_compare, pred)\n",
    "    print(f\"Fold score (accuracy): {score}\")\n",
    "\n",
    "# Build the oos prediction list and calculate the error.\n",
    "oos_y = np.concatenate(oos_y)\n",
    "oos_pred = np.concatenate(oos_pred)\n",
    "oos_y_compare = np.argmax(oos_y,axis=1) # For accuracy calculation\n",
    "\n",
    "score = metrics.accuracy_score(oos_y_compare, oos_pred)\n",
    "print(f\"Final score (accuracy): {score}\")    \n",
    "    \n",
    "# Write the cross-validated prediction\n",
    "oos_y = pd.DataFrame(oos_y)\n",
    "oos_pred = pd.DataFrame(oos_pred)\n",
    "oosDF = pd.concat( [df, oos_y, oos_pred],axis=1 )\n",
    "#oosDF.to_csv(filename_write,index=False)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training with both a Cross-Validation and a Holdout Set\n",
    "\n",
    "If you have a considerable amount of data, it is always valuable to set aside a holdout set before you cross-validate.  This hold out set will be the final evaluation before you make use of your model for its real-world use. Figure 5.HOLDOUT shows this division.\n",
    "\n",
    "**Figure 5.HOLDOUT: Cross Validation and a Holdout Set**\n",
    "![Cross Validation and a Holdout Set](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_3_hold_train_val.png \"Cross Validation and a Holdout Set\")\n",
    "\n",
    "The following program makes use of a holdout set, and then still cross-validates.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from scipy.stats import zscore\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Read the data set\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/jh-simple-dataset.csv\",\n",
    "    na_values=['NA','?'])\n",
    "\n",
    "# Generate dummies for job\n",
    "df = pd.concat([df,pd.get_dummies(df['job'],prefix=\"job\")],axis=1)\n",
    "df.drop('job', axis=1, inplace=True)\n",
    "\n",
    "# Generate dummies for area\n",
    "df = pd.concat([df,pd.get_dummies(df['area'],prefix=\"area\")],axis=1)\n",
    "df.drop('area', axis=1, inplace=True)\n",
    "\n",
    "# Generate dummies for product\n",
    "df = pd.concat([df,pd.get_dummies(df['product'],prefix=\"product\")],axis=1)\n",
    "df.drop('product', axis=1, inplace=True)\n",
    "\n",
    "# Missing values for income\n",
    "med = df['income'].median()\n",
    "df['income'] = df['income'].fillna(med)\n",
    "\n",
    "# Standardize ranges\n",
    "df['income'] = zscore(df['income'])\n",
    "df['aspect'] = zscore(df['aspect'])\n",
    "df['save_rate'] = zscore(df['save_rate'])\n",
    "df['subscriptions'] = zscore(df['subscriptions'])\n",
    "\n",
    "# Convert to numpy - Classification\n",
    "x_columns = df.columns.drop('age').drop('id')\n",
    "x = df[x_columns].values\n",
    "y = df['age'].values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fold #1\n",
      "Fold score (RMSE): 24.299626704604506\n",
      "Fold #2\n",
      "Fold score (RMSE): 0.6609159891625663\n",
      "Fold #3\n",
      "Fold score (RMSE): 0.4997884237817687\n",
      "Fold #4\n",
      "Fold score (RMSE): 1.1084218284103058\n",
      "Fold #5\n",
      "Fold score (RMSE): 0.614899992174395\n",
      "\n",
      "Cross-validated score (RMSE): 10.888206072135832\n",
      "Holdout score (RMSE): 0.6283593821273058\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "import pandas as pd\n",
    "import os\n",
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "from scipy.stats import zscore\n",
    "from sklearn.model_selection import KFold\n",
    "\n",
    "# Keep a 10% holdout\n",
    "x_main, x_holdout, y_main, y_holdout = train_test_split(    \n",
    "    x, y, test_size=0.10) \n",
    "\n",
    "\n",
    "# Cross-validate\n",
    "kf = KFold(5)\n",
    "    \n",
    "oos_y = []\n",
    "oos_pred = []\n",
    "fold = 0\n",
    "for train, test in kf.split(x_main):        \n",
    "    fold+=1\n",
    "    print(f\"Fold #{fold}\")\n",
    "        \n",
    "    x_train = x_main[train]\n",
    "    y_train = y_main[train]\n",
    "    x_test = x_main[test]\n",
    "    y_test = y_main[test]\n",
    "    \n",
    "    model = Sequential()\n",
    "    model.add(Dense(20, input_dim=x.shape[1], activation='relu'))\n",
    "    model.add(Dense(5, activation='relu'))\n",
    "    model.add(Dense(1))\n",
    "    model.compile(loss='mean_squared_error', optimizer='adam')\n",
    "    \n",
    "    model.fit(x_train,y_train,validation_data=(x_test,y_test),\n",
    "              verbose=0,epochs=500)\n",
    "    \n",
    "    pred = model.predict(x_test)\n",
    "    \n",
    "    oos_y.append(y_test)\n",
    "    oos_pred.append(pred) \n",
    "\n",
    "    # Measure accuracy\n",
    "    score = np.sqrt(metrics.mean_squared_error(pred,y_test))\n",
    "    print(f\"Fold score (RMSE): {score}\")\n",
    "\n",
    "\n",
    "# Build the oos prediction list and calculate the error.\n",
    "oos_y = np.concatenate(oos_y)\n",
    "oos_pred = np.concatenate(oos_pred)\n",
    "score = np.sqrt(metrics.mean_squared_error(oos_pred,oos_y))\n",
    "print()\n",
    "print(f\"Cross-validated score (RMSE): {score}\")    \n",
    "    \n",
    "# Write the cross-validated prediction (from the last neural network)\n",
    "holdout_pred = model.predict(x_holdout)\n",
    "\n",
    "score = np.sqrt(metrics.mean_squared_error(holdout_pred,y_holdout))\n",
    "print(f\"Holdout score (RMSE): {score}\")    \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.6 (tensorflow)",
   "language": "python",
   "name": "rga"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}