2b_arrays.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Arrays - Part 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Structured arrays\n",
    "\n",
    "A structured array consists of a number of columns, where each column can be a different datatype. \n",
    "\n",
    "Full information about structured arrays: \n",
    "http://docs.scipy.org/doc/numpy-1.10.1/user/basics.rec.html#structured-arrays\n",
    "\n",
    "One of the possible ways to specify a structured array is to use a list of tuples as `dtype`:\n",
    "For every column in the array a tuple is specified with the name of the column and the type of data in it. For example: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dtype = [('Name', 'U10'), ('Country', 'U10'), ('Area', 'float64')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The content of the array can then be given as a list of tuples, like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('Amsterdam', 'Netherland', 219.3) ('Paris', 'France', 105.4)\n",
      " ('Barcelona', 'Spain', 101.9)]\n"
     ]
    }
   ],
   "source": [
    "city = numpy.array([('Amsterdam', 'Netherlands', 219.3),\n",
    "                    ('Paris',     'France',      105.4 ),\n",
    "                    ('Barcelona', 'Spain',       101.9 )],\n",
    "                     dtype=dtype)\n",
    "print(city)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question\n",
    "What happens when you use list of lists instead of list of tuples to build a structured array?\n",
    "\n",
    "### Indexing structured arrays\n",
    "The rows in a structured array can be accessed by regular indexing. The columns of the array by using the column names that are specified when the array was created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('Amsterdam', 'Netherland', 219.3)\n",
      "[('Amsterdam', 'Netherland', 219.3) ('Paris', 'France', 105.4)]\n",
      "[ 219.3  105.4  101.9]\n",
      "[('Amsterdam', 219.3) ('Paris', 105.4) ('Barcelona', 101.9)]\n",
      "(3,) [('Name', '<U10'), ('Country', '<U10'), ('Area', '<f8')]\n"
     ]
    }
   ],
   "source": [
    "# Access first row\n",
    "print(city[0])\n",
    "\n",
    "# Access first two rows\n",
    "print(city[0:2])\n",
    "\n",
    "# Access column by name\n",
    "print(city['Area'])\n",
    "\n",
    "# Access two columns using list of names\n",
    "print(city[['Name', 'Area']])\n",
    "\n",
    "# Print information about the array\n",
    "print(city.shape, city.dtype)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that this structured array, even though it has rows and columns, \n",
    "is treated as one-dimensional.\n",
    "\n",
    "### Accessing and modifying column names\n",
    "\n",
    "For example:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('Name', 'Country', 'Area')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "city.dtype.names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 219.3  105.4  101.9]\n"
     ]
    }
   ],
   "source": [
    "city.dtype.names = ('name', 'country', 'area')\n",
    "print(city['area'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading data into structured arrays\n",
    "\n",
    "Structured arrays are useful for loading and working with tabular data with heterogeneous column types. \n",
    "\n",
    "#### Exercise 2b.1\n",
    "\n",
    "Complete the following code loading the data from file [populations.txt](populations.txt). Load the year column as an `int`, and the other columns as `float`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dtype = [('year',  ...\n",
    "         ('hare',  ...\n",
    "         ...\n",
    "          ] \n",
    "population = numpy.loadtxt(\"populations.txt\", dtype=...)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An alternative way of loading tabular data using `genfromtxt`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[  4000.   6100.   9800.  35200.  59400.  41700.  19000.  13000.   8300.\n",
      "   9100.   7400.   8000.  12300.  19500.  45700.  51100.  29700.  15800.\n",
      "   9700.  10100.   8600.]\n"
     ]
    }
   ],
   "source": [
    "population = numpy.genfromtxt(\"populations.txt\", \n",
    "             names=True,\n",
    "             dtype=['int','float','float','float'])\n",
    "# Access lynx column\n",
    "\n",
    "print(population['lynx'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Record arrays\n",
    "The is a special interface to structyred arrays called **record arrays**. For details, see https://docs.scipy.org/doc/numpy-1.10.1/user/basics.rec.html#record-arrays"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Array Indexing\n",
    "\n",
    "For complete information  about indexing see\n",
    "http://docs.scipy.org/doc/numpy/user/basics.indexing.html\n",
    "\n",
    "You have already seen how to access content of the array by using an index for each dimension. This method is know as matrix indexing. In addition to matrix indexing, there are other ways to address content in an array\n",
    "\n",
    "- Linear indexing transform the n-dimensional array to a 1-dimensional list. This linear index is returned when the `argmin` and `argmax` function are applied to an n-dimensional array. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.29493108 -0.70327901  0.97174343 -0.62764773  0.22290627]\n",
      " [-0.35018568  0.93124911 -0.47113303  0.37153914  0.37635051]\n",
      " [-0.76357905 -0.6857155   0.16636888 -0.26510055  0.94908343]\n",
      " [-0.39150909  0.90375192 -0.49675443  0.62174875  0.52339078]\n",
      " [ 0.83510306  0.40782966  0.66064087 -0.90445933  0.7370534 ]]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = numpy.random.uniform(-1,1,(5,5))\n",
    "print(a)\n",
    "# Return the index of the maximum value\n",
    "numpy.argmax(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Boolean indexing, which returns all values in the array for which the index is True."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ True False  True False  True]\n",
      " [False  True False  True  True]\n",
      " [False False  True False  True]\n",
      " [False  True False  True  True]\n",
      " [ True  True  True False  True]]\n",
      "[ 0.29493108  0.97174343  0.22290627  0.93124911  0.37153914  0.37635051\n",
      "  0.16636888  0.94908343  0.90375192  0.62174875  0.52339078  0.83510306\n",
      "  0.40782966  0.66064087  0.7370534 ]\n"
     ]
    }
   ],
   "source": [
    "# Create a boolean index for positive numbers in array a\n",
    "index = a > 0.0\n",
    "print(index)\n",
    "# Return all the positive numbers\n",
    "print(a[index])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Indexing with an array of indices. In this case you specify a separate array in which you store the indices as integers and you will return exactly the elements of the array with these indices. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.          0.11111111  0.22222222  0.33333333  0.44444444  0.55555556\n",
      "  0.66666667  0.77777778  0.88888889  1.        ]\n",
      "[ 0.22222222  0.33333333  0.55555556  0.77777778]\n"
     ]
    }
   ],
   "source": [
    "b = numpy.linspace(0,1,10)\n",
    "print(b)\n",
    "# Return numbers at prime indices\n",
    "index = numpy.array([ 2, 3, 5, 7])\n",
    "print(b[index])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Linear and matrix indexing\n",
    "\n",
    "Indexing in a 1-dimensional matrix is the same as the indexing in a Python list. And if you want to apply something to every element of the array then one simple for-loop over the items can do the trick.\n",
    "\n",
    "Indexing in a n-dimensional matrix has one index for every dimension. To access one element of the array, the index of every dimension should be given. When accessing more than one element, the slicing `\":\"` can be used, and this works similar as it works with lists, but then you can use the `\":\"` for every dimension. If no index is given for a dimension, then the `\":\"` will be given.\n",
    "If the index is `[a:b]` then indices that are used are `a` up to but not including `b`.\n",
    "\n",
    "If you have the linear index and you want to convert it to a matrix index, you can use the function `numpy.unravel_index()`.\n",
    "\n",
    "The first argument is the linear index and the second argument is the shape of the array for which you want to transform the index. For example: `numpy.unravel_index(linear_index, (2,3))`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[[ 0  1  2  3]\n",
      "  [ 4  5  6  7]\n",
      "  [ 8  9 10 11]]\n",
      "\n",
      " [[12 13 14 15]\n",
      "  [16 17 18 19]\n",
      "  [20 21 22 23]]]\n",
      "[[ 7 11]\n",
      " [19 23]]\n",
      "[[ 8  9 10 11]\n",
      " [20 21 22 23]]\n",
      "\n",
      " For a matrix with dimensions (2, 3, 4), the linear index:  10  is equal to matrix index:  (0, 2, 2)\n"
     ]
    }
   ],
   "source": [
    "# indexing in a 3-dimensional array\n",
    "z = numpy.arange(24).reshape((2, 3, 4))\n",
    "print(z)\n",
    "\n",
    "# slices\n",
    "print(z[0:2, 1:3, 3])\n",
    "print(z[:, 2, :])\n",
    "\n",
    "# linear indexing\n",
    "linear_index = 10\n",
    "print(\"\\n For a matrix with dimensions (2, 3, 4), the linear index: \", linear_index, \" is equal to \\\n",
    "matrix index: \", numpy.unravel_index(linear_index, z.shape))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 2b.2\n",
    "\n",
    "Create a $4\\times3$ matrix of random numbers between $0$ and $1$. \n",
    "Find the row and column position of the minimum and the maximum value."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 2b.3 \n",
    "\n",
    "Complete the following code to print years with the smallest number of hares, lynxes and carrots in the \n",
    "populations dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "for species in [....]:\n",
    "    year = ...\n",
    "    print(\"Least # of {} in year {}\".format(species, year))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Boolean indexing\n",
    "\n",
    "A boolean index can be created directly, but most often it is built by specifying a certain condition.\n",
    "\n",
    "The condition will return a True or False for every position in the array and when the condition is True then the corresponding element will be retrieved."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Only elements of x for which the value in y is True:  [1 3 5]\n",
      "Only elements of x for which the condition is True:  [4 5]\n"
     ]
    }
   ],
   "source": [
    "# Boolean indexing\n",
    "x = numpy.arange(1, 6)\n",
    "y = numpy.array([True, False, True, False, True ])\n",
    "print(\"Only elements of x for which the value in y is True: \", x[y])\n",
    "\n",
    "# boolean indexing by using a condition\n",
    "print(\"Only elements of x for which the condition is True: \", x[x>3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 2b.4\n",
    "Use the population data to\n",
    "\n",
    "1. Select all the years in which there are more than 50000 lynxes;\n",
    "2. Select all the years in which there are more lynxes than hares."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Indexing with an array of indices\n",
    "\n",
    "In this case you specify a separate array in which you store the indices as integers and you will return exactly the elements of the array with these indices.\n",
    "\n",
    "One advantage of this is that you can explicitly specify the order in which you want to have the values and you can return multiple times the value at a certain position. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Array x:  [100 101 102 103 104 105 106 107 108 109 110]\n",
      "Array with indices:  [8 3 8 4 9 3]\n",
      "Indexing with an array of indices will give: [108 103 108 104 109 103]\n"
     ]
    }
   ],
   "source": [
    "x = numpy.arange(100, 111)\n",
    "y = numpy.array([8, 3, 8, 4, 9, 3])\n",
    "print(\"Array x: \", x)\n",
    "print(\"Array with indices: \", y)\n",
    "print(\"Indexing with an array of indices will give:\", x[y])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 2b.5\n",
    "\n",
    "Indexing with an array is often useful when we want to randomize the order of items in some data. Complete the following code which creates a scrambled version of the population data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Create an index for the rows of population (from 0 to population.shape[0])\n",
    "index = ...\n",
    "# Shuffle the index\n",
    "numpy.random.shuffle(index)\n",
    "# Create a scrambled version\n",
    "population_rand = ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vector stacking\n",
    "\n",
    "Sometimes you want to combine two or more vectors to create an array. This is called vector stacking. Vector stacking can be done in two different ways horizontal and vertical. \n",
    "- horizontal stack: `numpy.hstack([x, y, z])`\n",
    "- vertical stack: `numpy.vstack([x, y, z])`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Horizontal stack:  [ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14]\n",
      "Vertical stack: \n",
      "[[ 0  1  2  3  4]\n",
      " [ 5  6  7  8  9]\n",
      " [10 11 12 13 14]]\n"
     ]
    }
   ],
   "source": [
    "x = numpy.arange(0,5)                     \n",
    "y = numpy.arange(5, 10)   \n",
    "z = numpy.arange(10, 15)\n",
    "print(\"Horizontal stack: \",  numpy.hstack([x,y, z]) )\n",
    "print(\"Vertical stack: \")\n",
    "print( numpy.vstack([x,y, z]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save data set to file\n",
    "\n",
    "When you want to save an array from numpy as a separate file you always have to specify the filename and the array you want to save and you can use the following functions:\n",
    "- `numpy.savetxt(filename, array)` : save an array to a text file. Some optional arguments are: delimiter=' ', newline = '\\n', header = ' '. http://docs.scipy.org/doc/numpy-1.10.0/reference/generated/numpy.savetxt.html#numpy.savetxt\n",
    "- `numpy.save(filename, array)` : save an array to a binary file in numpy `.npy` format. http://docs.scipy.org/doc/numpy-1.10.0/reference/generated/numpy.save.html#numpy.save\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 2b.6 \n",
    "\n",
    "Save the population data to a `.npy` file. Figure out how to load it back into a numpy array.\n",
    "\n",
    "#### Exercise 2b.7\n",
    "The files\n",
    "\n",
    "- [irisa.txt](irisa.txt)\n",
    "- [irisb.txt](irisb.txt)\n",
    "- [irisc.txt](irisc.txt)\n",
    "\n",
    "contain the data for the iris dataset. Each file has these columns:\n",
    "\n",
    "- `SepalLength` \n",
    "- `SepalWidth`\n",
    "- `PetalLength` \n",
    "- `PetalWidth` \n",
    "- `Species`\n",
    "\n",
    "Load this data, and create a single array with all the species."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3+"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}