11_installing-python-modules.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**_pySpark Basics: Installing Python Modules_**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_by Jeff Levy (jlevy@urban.org)_\n",
    "\n",
    "_Last Updated: 2 Aug 2016, Spark v1.6.1_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Abstract: When a new cluster is spun up on AWS, it comes with only a few Python modules installed.  This guide will go over adding more as necessary._\n",
    "\n",
    "_Main operations used:_ `pip`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Those with experience in Python are probably accustomed to using a distribution with many modules pre-packaged, for example from Anaconda or Enthought.  However, every time a cluster is spun up on AWS the installation of Python must be ported over from permanent storage, and the more it has to move and configure the longer the spinup process becomes.  Based on the assumption that most users will not need very many modules not already provided by pySpark, we have opted to minimize startup time by not pre-configuring Python with lots of modules.\n",
    "\n",
    "# Installing Modules\n",
    "\n",
    "The current boostrap script installs `pip`, (module installation manager), `numpy` (many math operations), `requests` (tools for accessing the web) and `matplotlib` (graphing).  It also comes with the [Python standard library](https://docs.python.org/2/library/) that all installs have access to, such as `datetime`, `random`, `collections` and so on.  Any other modules you may need can be installed as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting pandas\n",
      "  Downloading pandas-0.18.1.tar.gz (7.3MB)\n",
      "Requirement already satisfied (use --upgrade to upgrade): python-dateutil in ./venv/lib/python2.7/site-packages (from pandas)\n",
      "Requirement already satisfied (use --upgrade to upgrade): pytz>=2011k in ./venv/lib/python2.7/site-packages (from pandas)\n",
      "Requirement already satisfied (use --upgrade to upgrade): numpy>=1.7.0 in ./venv/lib64/python2.7/site-packages (from pandas)\n",
      "Requirement already satisfied (use --upgrade to upgrade): six>=1.5 in ./venv/lib/python2.7/site-packages (from python-dateutil->pandas)\n",
      "Installing collected packages: pandas\n",
      "  Running setup.py install for pandas: started\n",
      "    Running setup.py install for pandas: still running...\n",
      "    Running setup.py install for pandas: still running...\n",
      "    Running setup.py install for pandas: finished with status 'done'\n",
      "Successfully installed pandas-0.18.1\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pip.main(['install', 'pandas'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note in the output that this command also checked on all the dependencies for `pandas`, and would have installed them if they had been lacking.  You can do this for any package listed in the PyPi index, the official repository for Pthon modules.\n",
    "\n",
    "# Installing Specific Versions\n",
    "\n",
    "We can also use this to get specific versions of modules; by default it installs the newest:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting statsmodels==0.6.0\n",
      "  Downloading statsmodels-0.6.0.zip (7.3MB)\n",
      "Collecting scipy (from statsmodels==0.6.0)\n",
      "  Downloading scipy-0.17.1-cp27-cp27mu-manylinux1_x86_64.whl (39.5MB)\n",
      "Requirement already satisfied (use --upgrade to upgrade): pandas in ./venv/lib64/python2.7/site-packages (from statsmodels==0.6.0)\n",
      "Collecting patsy (from statsmodels==0.6.0)\n",
      "  Downloading patsy-0.4.1-py2.py3-none-any.whl (233kB)\n",
      "Requirement already satisfied (use --upgrade to upgrade): python-dateutil in ./venv/lib/python2.7/site-packages (from pandas->statsmodels==0.6.0)\n",
      "Requirement already satisfied (use --upgrade to upgrade): pytz>=2011k in ./venv/lib/python2.7/site-packages (from pandas->statsmodels==0.6.0)\n",
      "Requirement already satisfied (use --upgrade to upgrade): numpy>=1.7.0 in ./venv/lib64/python2.7/site-packages (from pandas->statsmodels==0.6.0)\n",
      "Requirement already satisfied (use --upgrade to upgrade): six in ./venv/lib/python2.7/site-packages (from patsy->statsmodels==0.6.0)\n",
      "Installing collected packages: scipy, patsy, statsmodels\n",
      "  Running setup.py install for statsmodels: started\n",
      "    Running setup.py install for statsmodels: finished with status 'done'\n",
      "Successfully installed patsy-0.4.1 scipy-0.17.1 statsmodels-0.6.0\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pip.main(['install', 'statsmodels==0.6.0'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that this ended up installing two dependencies, `patsy` and `scipy`.  \n",
    "\n",
    "# Upgrading Modules\n",
    "\n",
    "And finally, we can use `pip` to upgrade modules if necessary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting statsmodels\n",
      "  Downloading statsmodels-0.6.1.tar.gz (7.0MB)\n",
      "Requirement already up-to-date: pandas in ./venv/lib64/python2.7/site-packages (from statsmodels)\n",
      "Requirement already up-to-date: python-dateutil in ./venv/lib/python2.7/site-packages (from pandas->statsmodels)\n",
      "Requirement already up-to-date: pytz>=2011k in ./venv/lib/python2.7/site-packages (from pandas->statsmodels)\n",
      "Requirement already up-to-date: numpy>=1.7.0 in ./venv/lib64/python2.7/site-packages (from pandas->statsmodels)\n",
      "Requirement already up-to-date: six>=1.5 in ./venv/lib/python2.7/site-packages (from python-dateutil->pandas->statsmodels)\n",
      "Installing collected packages: statsmodels\n",
      "  Found existing installation: statsmodels 0.6.0\n",
      "    Uninstalling statsmodels-0.6.0:\n",
      "      Successfully uninstalled statsmodels-0.6.0\n",
      "  Running setup.py install for statsmodels: started\n",
      "    Running setup.py install for statsmodels: finished with status 'done'\n",
      "Successfully installed statsmodels-0.6.1\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pip.main(['install', '--upgrade', 'statsmodels'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All of the packages you install this way will remain installed as long as this cluster is spun up.  You can open multiple notebooks, or close all of them and open a new one, and they will all have access to the modules you've installed.  Only when the cluster is spun down through the AWS Console will you need to start over.\n",
    "\n",
    "And finally, it's important to keep in mind that some packages may not be compatible with distributed data.  The `pandas` module, for example, is for working with dataframes in a standard desktop environment - if you try to load a very large distributed dataset into a Pandas dataframe, *it will attempt to put all the data in one location*, and it will fail.  If you have reduced your data down to a reasonable size, however, you can load it into a Pandas dataframe.  Whether a module will work for you or not depends entirely on the module and the situation, so feel free to consult with Research Programming if in doubt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}