05_numerical_data_in_numpy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Numerical data in `numpy`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[`numpy`](https://docs.scipy.org/doc/numpy/) is a very powerful library for working with numerical data in Python. It introduces the __Array__ data structure, which can contain multi-dimensional numerical data. The `numpy` library is __not__ part of the Python standard library. However, it comes bundles with [Anaconda](01_anaconda.ipynb) so you should already have it installed. The usual way to import `numpy` is as follows:\n",
    "\n",
    "    import numpy as np\n",
    "    \n",
    "This gives us access to all the `numpy`-functions using the prefix `np`. This is a convention, and you should do the same in your own code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "np.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reading data with `numpy`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `numpy` library comes with a few functions that can read numerical data from text files. The `np.loadtxt`-function is a fast reader with some basic functionality for skipping header lines, converting values to floats, etc. For more sophisticated files the `np.genfromtxt`-function can be used. It also supports handling missing values, but is slower.\n",
    "\n",
    "Note that these functions take a filename as input, so that you do not need to use the built-in `open`-function to open a file beforehand. Assume we have a text-file containing data of the following form:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!cat data/numpy_simple.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These data can be loaded with a very simple `np.loadtxt`-command as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data = np.loadtxt('data/numpy_simple.txt')\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Arrays in `numpy` have a shape specifying how big the dataset is. In this case we have 6 rows and 8 columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Indexing and vectorization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly to lists and other sequences in Python, `numpy` arrays can be indexed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data[0]    # First row (= row 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data[2:5]    # Rows 2, 3 and 4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, with multi-dimensional arrays (in this case 2-dimensional), we can specify each dimension in the index separated by commas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data[4, 5]    # Element in row 4, column 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data[:, :2]    # All rows, first two columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With `numpy`-arrays most operations are vectorized. That means that we do not need to explicitly loop over the elements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data[:, 4] + data[:, 6]    # Add columns 4 and 6 together"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "np.exp(data[:, -1])    # Exponentiate the last (-1) column"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## More advanced reading of data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most datafiles are not as clean as the simple datafile we have been working with above. Let us instead try to load the following file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!cat data/numpy_header.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A naive use of `np.loadtxt` will fail because `numpy` tries to interpret the header as data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data2 = np.loadtxt('data/numpy_header.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead we must give the `np.loadtxt` some more information. To get some help about a function and which parameters it takes, you can write a question mark after its name,\n",
    "\n",
    "    np.loadtxt?\n",
    "    \n",
    "or press `<shift>` and `<tab>` inside the paranthesis. Pressing `<shift>` and `<tab>` twice will give even more information.\n",
    "\n",
    "In this case, we notice that there is an argument called `skiprows` that can be used to ignore the header."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data2 = np.loadtxt('data/numpy_header.txt', skiprows=13)\n",
    "np.allclose(data, data2)    # Test if data and data2 contains the same elements (within a tolerance)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking more closely at the data, we also notice that there is one datapoint with the value of -999 that probably designates a missing data point. We can convert this to a `nan`-value to handle it properly as we are reading the data. However, to do so we need to use the more sophisticated `np.genfromtxt`-function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data3 = np.genfromtxt('data/numpy_header.txt', skip_header=13, missing_values='-999', usemask=True).filled(np.nan)\n",
    "data3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Actually handling the missing data is a two-step process. First we create a masked array, where the mask denotes which data are missing. This can be seen if we look directly at the output from `np.genfromtxt` (note the single `True`-value in the mask)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "np.genfromtxt('data/numpy_simple.txt', missing_values='-999', usemask=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See the [`numpy` docs](https://docs.scipy.org/doc/numpy-dev/user/basics.io.genfromtxt.html) for more information about reading data."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}