t81_558_reference.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Course Reference**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), School of Engineering and Applied Science, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pandas\n",
    "\n",
    "This section describes Pandas, which is used to load/access data.  For more information, refer to [class 2](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class1_intro_python.ipynb).\n",
    "\n",
    "## Primary Imports\n",
    "```\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "```\n",
    "\n",
    "## Load a Dataframe from CSV\n",
    "\n",
    "```\n",
    "path = \"./data/\"\n",
    "filename_read = os.path.join(path,\"auto-mpg.csv\")\n",
    "df = pd.read_csv(filename_read)\n",
    "```\n",
    "\n",
    "## Write a Dataframe to CSV\n",
    "\n",
    "```\n",
    "df = pd.read_csv(filename_read,na_values=['NA','?'])\n",
    "df = df.reindex(np.random.permutation(df.index))\n",
    "df.to_csv(filename_write,index=False) \n",
    "```\n",
    "\n",
    "## Shuffle a Dataframe\n",
    "\n",
    "```\n",
    "filename_read = os.path.join(path,\"auto-mpg.csv\")\n",
    "df = pd.read_csv(filename_read,na_values=['NA','?'])\n",
    "np.random.seed(42) \n",
    "df = df.reindex(np.random.permutation(df.index))\n",
    "df.reset_index(inplace=True, drop=True)\n",
    "```\n",
    "\n",
    "## Access a Row and Column\n",
    "\n",
    "```\n",
    "print(\"The first car is: {}\".format(df['name'].iloc[0]))\n",
    "```\n",
    "\n",
    "## Add Calculated Field\n",
    "\n",
    "```\n",
    "df.insert(1,'weight_kg',(df['weight']*0.45359237).astype(int))\n",
    "```\n",
    "\n",
    "## ZScore\n",
    "\n",
    "```\n",
    "df['mpg'] = zscore(df['mpg'])\n",
    "```\n",
    "\n",
    "## Fill Missing Values\n",
    "\n",
    "```\n",
    "med = df['horsepower'].median()\n",
    "df['horsepower'] = df['horsepower'].fillna(med)\n",
    "```\n",
    "\n",
    "## Create New Dataset with Just Some Columns\n",
    "\n",
    "```\n",
    "col_horsepower = df['horsepower']\n",
    "col_name = df['name']\n",
    "result = pd.concat([col_name,col_horsepower],axis=1)\n",
    "```\n",
    "\n",
    "# Training/Validation Split\n",
    "\n",
    "```\n",
    "mask = np.random.rand(len(df)) < 0.8\n",
    "trainDF = pd.DataFrame(df[mask])\n",
    "validationDF = pd.DataFrame(df[~mask])\n",
    "\n",
    "print(\"Training DF: {}\".format(len(trainDF)))\n",
    "print(\"Validation DF: {}\".format(len(validationDF)))\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Keras\n",
    "\n",
    "The following imports allow access to Keras."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "# Imports\n",
    "from keras.models import Sequential\n",
    "from keras.layers.core import Dense, Activation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Keras Feedforward Regression\n",
    "\n",
    "The following is a basic Keras feedforward neural network for regression:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_size = 5 # How many input neurons do you need?\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(25, input_dim=input_size, activation='relu')) # Hidden 1\n",
    "model.add(Dense(10, activation='relu')) # Hidden 2\n",
    "model.add(Dense(1)) # Output\n",
    "model.compile(loss='mean_squared_error', optimizer='adam')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ever neural network used in this class will be of type [**Sequential**](https://keras.io/models/sequential/), even the recurrent neural networks.\n",
    "\n",
    "For feedforward, all layers are [**Dense**](https://keras.io/layers/core/).  The first **Dense** layer is must specify how many input neurons are needed.  Anny additional hidden layers will also be **Dense**, but should not have an **input_dim** specification.  \n",
    "\n",
    "Dense has the following parameters:\n",
    "\n",
    "* **units** - A number above that specifies how many neurons on that layer.  Above it is 25, 10 and 1.\n",
    "* **input_dim** - ONLY valid on the first Dense layer, how many input neurons you need.  \n",
    "* **activation** - For this class should always be **relu**, unless it is the output for a regression network.  If this is the output layer of a regression, then units should be 1 and **activation** should be **linear** or omitted (it defaults to **linear**). There are quite a few additional [activations](https://keras.io/activations/) you can use, just not in this course.  Relu is the most common.\n",
    "\n",
    "The **compile** function builds the neural network.  It has two parameters:\n",
    "\n",
    "* **loss** - How are errors calculated for training optimization.  The **mean_squared_error** option should always be used for regression in this course.  However, there are other [losses](https://keras.io/losses/) that can be used. \n",
    "* **optimizer** - The optimizer specifies the algorithm that will be used to adjust neural network weights during training.  Typically **adam** is a good choice.  However, other [optimizers](https://keras.io/optimizers/) are available.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Keras Feedforward Classification\n",
    "\n",
    "The following is a basic Keras feedforward neural network for regression:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_size = 5 # How many input neurons do you need?\n",
    "output_classes = 3 # How many output classes are there?\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(50, input_dim=input_size, activation='relu')) # Hidden 1\n",
    "model.add(Dense(25, activation='relu')) # Hidden 2\n",
    "model.add(Dense(output_classes,activation='softmax')) # Output\n",
    "model.compile(loss='categorical_crossentropy', optimizer='adam')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most of the information from the previous section for regression also applies to the classification example above.  However, for classification, the output layer should use **softmax** as the activation function.  This ensures that all of the output neurons specify class probabilities and sum to 1 (or at least close to it).\n",
    "\n",
    "The **compile** call should use **categorical_crossentropy** for a classification neural network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.6 (wustl)",
   "language": "python",
   "name": "wustl"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}