tu017-Pandas-DataWranglingAutomation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c7240a29",
   "metadata": {},
   "source": [
    "# Automating data wrangling"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36232670",
   "metadata": {},
   "source": [
    "Sometimes we require a \"one off\" solution to a unique data analysis problem. In this situation, we write code to do a particular analysis on a particular data set. Then, if the analysis is part of a publication, we make the code and data publically available and... we're done.\n",
    "\n",
    "Often, however, we require a ***reusable*** solution that operates on data of a given format even though some of the particulars, such as sample size or variable names, might change. In this case, we want our code to be \"dynamic\" in the sense that it should be able to handle any anticipated changes to the details of the input data.\n",
    "\n",
    "Here, we'll tackle the same problem as last time – reformatting a data set from a cumbersome format into a more useful and \"tidy\" format."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e341163a",
   "metadata": {},
   "source": [
    "### Learning goals:\n",
    "\n",
    "* write reusable code for a data wrangling problem\n",
    "* create a function to make the code handy to use"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbe70474",
   "metadata": {},
   "source": [
    "## Import pandas and look at the data from last time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5212a9a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54942888",
   "metadata": {},
   "source": [
    "Read in the data from last time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "337ca959",
   "metadata": {},
   "outputs": [],
   "source": [
    "my_input_data = pd.read_csv('datasets/017DataFile.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3eaff4c1",
   "metadata": {},
   "source": [
    "Take a peek to remind ourselves of the data format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ab53d09c",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "my_input_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31aaeeae",
   "metadata": {},
   "source": [
    "In this data set, there are two \"independent variables\", sex and genotype of laboratory rats, and one \"dependent variable\", response time. The data are formatted such that each column contains the data from a unique combination of the two independent variables, *i.e.* a \"cell\" of the experimental design. Like this:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dbb5da10",
   "metadata": {},
   "source": [
    "|     | male | female |\n",
    "| --- | --- | --- |\n",
    "| **mutant** | mm | fm |\n",
    "| **wildtype** | mw | fw |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bcc027e",
   "metadata": {},
   "source": [
    "This format might seem to make sense, but it's actually not very flexible. For analysis purposes, it's generally better to have data in a format that obeys a couple of rules:\n",
    "\n",
    "* *each row should correspond to a single observation (measurement)*\n",
    "* *each column should correspond to a single variable*\n",
    "\n",
    "Data in this format are also referred to as \"tidy\"."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9be409d0",
   "metadata": {},
   "source": [
    "So in this case, our goal is to take the above data and put it into a format like this:\n",
    "\n",
    "| response time | sex | genotype |\n",
    "| ---| --- | --- |\n",
    "| rt value | male or female | wild or not |\n",
    "\n",
    "Once the data are in this format, we can easily use our tools to do things like compare wild to mutant, or compare wild to mutant only in females, etc."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "526890e5",
   "metadata": {},
   "source": [
    "Last time, we stacked the reaction time values into a single column using pandas functions. This relied on us knowing and \"hard coding\" the column names (\"Male Mutant\", etc.). If we're going to automate things, we want our code to be agnostic about these. One way would be to somehow read the column names into variables and work with them somehow... \n",
    "\n",
    "But what about numpy arrays? We already know how to manipulate those and, since they are just numbers, there are no column names or pesky row indexes to worry about. So let's try using numpy!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a3597f52",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84aca30f",
   "metadata": {},
   "source": [
    "Pandas dataframes know how to convert themselves to numpy arrays. They have a `to_numpy()` method that will pull *just the numbers* out of our dataframe, ignoring the column labels and row indexs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8418feb3",
   "metadata": {},
   "outputs": [],
   "source": [
    "raw_data = my_input_data.to_numpy() "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9ca648e",
   "metadata": {},
   "source": [
    "Let's take a look!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2d3c1ec4",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "raw_data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "905549fe",
   "metadata": {},
   "source": [
    "## Get some useful information from the original data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a14dd660",
   "metadata": {},
   "source": [
    "So far so good! Now we are going to put the data into the format we want. To automate this, we are going to get \n",
    "\n",
    "* the number of observations in each group (which is the number of rows), and \n",
    "* the number of groups (which is the number of columns)\n",
    "\n",
    "and store them in variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "174b3e3b",
   "metadata": {},
   "outputs": [],
   "source": [
    "obs_per_grp, grps = raw_data.shape\n",
    "print(\"We have \", obs_per_grp, \" observations per group and \", grps, \" groups.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "090bbebb",
   "metadata": {},
   "source": [
    "Now we'll calculate the total number of observations, which is also how long we want our new data frame to be."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b903a938",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "new_length = obs_per_grp*grps\n",
    "print(\"We have \", new_length, \" total observations.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "851ff17a",
   "metadata": {},
   "source": [
    "$\\color{blue}{\\text{Complete the following exercise.}}$\n",
    "\n",
    "  - Use the cell below and explain in your own words why we used Numpy Arrays in the previous cells. What was our final goal? Why did we dump the data into a Numpy Array?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e609377",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "78c4d1c1",
   "metadata": {},
   "source": [
    "## Build our response time (dependent variable) column"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "138f494f",
   "metadata": {},
   "source": [
    "We could now play legos \"by hand\", stacking the columns of our numpy array on top of each other to make a new array (and we already know how to do that). \n",
    "\n",
    "Or we could take advantage of the fact that one of the things numpy arrays know how to do – one of the methods they have – is to change their shape. So we'll take our `obs` by `cols` array and `numpy.reshape()` into a `new_length` by 1 array.\n",
    "\n",
    "What this command does (effectively) is read out the data values from the original array one-by-one, and places them in the cells of a new array of a shape you specify. The only catch is that the total number of cells in the new array has to be the same as in the old array – in other words, each and every data value has to have one and only one place to go in the new array. Which makes sense."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da1521a9",
   "metadata": {},
   "outputs": [],
   "source": [
    "values_col = np.reshape(raw_data, (new_length, 1))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cdc1100b",
   "metadata": {},
   "source": [
    "I called it `values_col` because it will eventually become the values column of our new pandas data frame."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "339c88f5",
   "metadata": {},
   "source": [
    "Let's see if that worked:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "80376fbe",
   "metadata": {},
   "outputs": [],
   "source": [
    "values_col"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e2d5240",
   "metadata": {},
   "source": [
    "Nice! But let's make absolutely sure that worked. What we want is for the columns of the original data to be stacked on top of one another. Is that what we have?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e1ab08a",
   "metadata": {},
   "source": [
    "Nope, it's not right. What happened is that the values got read out *left to right, top to bottom* (or row-wise) and placed into the new array one-by-one. But what we want is for the values to be read *top to bottom, left to right* (or columnwise). We can make this happen with the `order=` argument of `numpy.reshape()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c5f6282b",
   "metadata": {},
   "outputs": [],
   "source": [
    "values_col = np.reshape(raw_data, (new_length, 1), order = 'F')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29be7bee",
   "metadata": {},
   "source": [
    "Let's make sure that worked:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4b8b7d19",
   "metadata": {},
   "outputs": [],
   "source": [
    "values_col"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8ea7f1c",
   "metadata": {},
   "source": [
    "**Yay!** It did!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92c17aa4",
   "metadata": {},
   "source": [
    "**Useless trivia**: Two of Ye Olde Major Programming Languages are **C** (used mainly by programmers) and **Fortran** (used mainly by scientists). C (the language used to write Python) uses row-wise indexing, whereas Fortran uses columnwise indexing. That's why \"F\" is used to specify columnwise indexing above: the \"F\" is for \"Fortran\"."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "153762c3",
   "metadata": {},
   "source": [
    "Minor annoying thing: (there is always at least one that pops up in any coding task, amirite?) `values_col` is a (40x1) 2-dimensional numpy array but, when we go to build our new data frame, we'll need it to be a 40 long (40,) 1-dimensional array. \n",
    "\n",
    "This actually comes up so often that `numpy` has a `squeeze()` function to squeeze the dimension of length one into nothingness. It turns (n, 1) things into (n,) things."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b8722f5",
   "metadata": {},
   "source": [
    "Let's check the shape of our new array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed3e1de0",
   "metadata": {},
   "outputs": [],
   "source": [
    "values_col.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0bd62495",
   "metadata": {},
   "source": [
    "Now let's squeeze the (uneeded and unwanted) column dimension into oblivion:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ec0537d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "values_col = np.squeeze(values_col)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "187f6777",
   "metadata": {},
   "source": [
    "And check the shape again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a723fd0",
   "metadata": {},
   "outputs": [],
   "source": [
    "values_col.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69bf7b26",
   "metadata": {},
   "source": [
    "Okay, that worked, now onto..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4b2bb61",
   "metadata": {},
   "source": [
    "$\\color{blue}{\\text{Complete the following exercise.}}$\n",
    "\n",
    "  - Use the next cell to explaing what happened to the numpy array after the squeeze operation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dba4f11a",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "51cc2d68",
   "metadata": {},
   "source": [
    "  - Type below code demonstrating how you could explore the help for the method `.shape()` to explore what it does:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "93e2fbe3",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "854dd05b",
   "metadata": {},
   "source": [
    "   - Use the cell below to explain the use of the method `.resape()`:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "062d0172",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "0c4321fe",
   "metadata": {},
   "source": [
    "## Building the independent variable columns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44152c80",
   "metadata": {},
   "source": [
    "What we require is that the levels our two independent variables repeat themselves in the right order down their respective columns. We could certainly type this in by hand, but that would be really annoying to change if we required new labels later on or something. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55df6597",
   "metadata": {},
   "source": [
    "We could also use `for()` loops; they are designed for exactly such repetitive tasks after all. That might look something like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dfd72bab",
   "metadata": {},
   "outputs": [],
   "source": [
    "gen_var = list()                     # create a python list \n",
    "for i in range(new_length) :         # loop through all observations\n",
    "    if i < new_length/2 :            # for the first half, ...\n",
    "        gen_var.append(\"wildtype\")   # set to male\n",
    "    else :                           # otherwise...\n",
    "        gen_var.append(\"mutant\")     # set to female"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "080eb73f",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(gen_var)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b1a07be",
   "metadata": {},
   "source": [
    "We'd have to get a little bit more fancy with our `if...` to create the sex variable, that'd be the idea."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3bced34d",
   "metadata": {},
   "source": [
    "But pandas provides easy ways to repeat and stack things (numpy does too), so let's try those. The two will use are\n",
    "\n",
    "* `pandas.Series.repeat()` \n",
    "* `pandas.concat()`\n",
    "\n",
    "Note: When you see `pandas.Series.somefunction()` or `pandas.DataFrame.somefunction()` in the documentation, that means that all Series or DataFrames know how to do `somefunction()`. So if you had a Series named `Phred`, you would say `Phred.somefunction()` to use `somefunction()`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc4a6f61",
   "metadata": {},
   "source": [
    "$\\color{blue}{\\text{Complete the following exercise.}}$\n",
    "\n",
    "  - Use the cell below to explain what the variable `new_length` contain:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7edd8950",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "3c14094a",
   "metadata": {},
   "source": [
    "   - Use the cell below to explain the reason why we use `new_length/2` in combination with the `if, else`:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2afc8dff",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "ff985a74",
   "metadata": {},
   "source": [
    "### Make the genetic strain variable"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c915d0e1",
   "metadata": {},
   "source": [
    "In the way we have formatted the data, genetic strain is the \"outer\" variable, in that it only changes once as we go down the data set: all the wildtypes are on top, and all mutants are on the bottom. The sex variable is the \"inner\" variable, because it changes once within each value of strain, so it needs to three times as we go down the data set.\n",
    "\n",
    "This is arbitrary and has nothing to do with the experimental design; we could have formatted the data such that the roles were reversed."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7edd2079",
   "metadata": {},
   "source": [
    "What we will do is \n",
    "\n",
    "* make a short series containing the two levels of our variable\n",
    "* repeat each value to make the long series \n",
    "* deal with annoying index values (there's always something...)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b8d898fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "strain = pd.Series(['wildtype', 'mutant'])  # make the short series\n",
    "strain = strain.repeat(2*obs_per_grp)       # repeat each over two cell's worth of data\n",
    "strain = strain.reset_index(drop=True)      # reset the series's index value"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e25b7580",
   "metadata": {},
   "source": [
    "$\\color{blue}{\\text{Complete the following exercise.}}$\n",
    "\n",
    "  - Use the cell below to explain what is and what it is contained by the variable `strain`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d27feab7",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "0cea2651",
   "metadata": {},
   "source": [
    "Let's see if that worked:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f1607ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(strain)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59b6726d",
   "metadata": {},
   "source": [
    "$\\color{blue}{\\text{Complete the following exercise.}}$\n",
    "\n",
    "  - Use the cell below to explain why `mutants` appear at the bottom of the previous `Pandas Series`, who decided that order?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "849648f0",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "7f3b6c27",
   "metadata": {},
   "source": [
    "### Make the sex variable"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "775e276f",
   "metadata": {},
   "source": [
    "As the sex variable is the inner variable, we need it have `['male'..., 'female'...]` within each outer block of genotype. So what we'll do is make one block of `['male'..., 'female'...]` and then just stack two copies of that to make our variable. So the steps are\n",
    "\n",
    "* make a short series containing the two levels of our variable (just like above)\n",
    "* repeat it (just like above)\n",
    "* stack two copies on top of each other (dropping the annoying indexes in the process)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "621e0177",
   "metadata": {},
   "outputs": [],
   "source": [
    "sexes = pd.Series(['male', 'female'])             # make the short series\n",
    "sexes = sexes.repeat(obs_per_grp)                 # repeat each over one cell's worth of data\n",
    "sexes = pd.concat([sexes]*2, ignore_index=True)   # stack or \"concatonate\" two copies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7ec33914",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(sexes)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac9faacd",
   "metadata": {},
   "source": [
    "$\\color{blue}{\\text{Complete the following exercise.}}$\n",
    "\n",
    "  - Use the cell below to explain in your own words what happened in the previous cell:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc9ace3a",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "4398af96",
   "metadata": {},
   "source": [
    "  - Use the cell below to show your code to create a pandas series called `unicorns` comprising of 20 mistical equines half of which are `white` and half `pearl-white` in color (well ... what what do you want, they are unicorns):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1497f14c",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "e739471e",
   "metadata": {},
   "source": [
    "  - Use the cell below to show your code to create a pandas series called `Three trees` comprising of 30 trees 1/3 of which are `Live Oaks`, 1/3 `White Oaks` and 1/3 `Red Oaks`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ba4bbe3",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "c0da6b0e",
   "metadata": {},
   "source": [
    "### Build our new data frame!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ccc767b0",
   "metadata": {},
   "source": [
    "Data frames are created in pandas by handing it data it can make sense of. There are various ways to accomplish this, and one handy one is to hand it data in a \"column label 1 : data 1, column label 2 : data 2, ...\" format. \n",
    "\n",
    "We can accomplish this with a python \"dictionary\" (remember those?). A python `dict` associates a label (the \"word\") with a value or set of values or whatever (the \"definition\"). They are very useful, so let's take a look at a simple example before we use one to build out data frame. You create a dictionary using curly braces, and then use colons to bind each word or `key` with its definition or `value`. Commas separate each key-value pair."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d4dd1ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "myData = {\"name\": \"Larry\", \"rank\": \"full\", \"years\": 30, \"bikes\": 5, \"motorcycles\": 2, \"teslas\": 1}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74c9eede",
   "metadata": {},
   "outputs": [],
   "source": [
    "myData[\"name\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6ae65c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "myData[\"bikes\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67980281",
   "metadata": {},
   "source": [
    "$\\color{blue}{\\text{Complete the following exercise.}}$\n",
    "\n",
    "  - Use the cell below to build a `dict()` describing a student, with a name, with a student ID, a GPA and a major, make up all the values but use the lables as described here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "42416bb7",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "f6b11684",
   "metadata": {},
   "source": [
    "So a dictionary associates a label with data values. **Perfect!**\n",
    "\n",
    "Time to build our data frame!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c8c4d131",
   "metadata": {},
   "outputs": [],
   "source": [
    "my_tidy_data = pd.DataFrame(      # invoke creation\n",
    "    {                             # start the dictionary with a {\n",
    "        \"RTs\": values_col,        # assign each variable to a label\n",
    "        \"sex\": sexes,\n",
    "        \"strain\": strain\n",
    "    }                             # end the dictionary with a }\n",
    ")                                 # end of creation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6268f1d",
   "metadata": {},
   "source": [
    "Note that the formatting above is just to make the columns we're creating more obvious and human-readable. This will work too:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ea658a54",
   "metadata": {},
   "outputs": [],
   "source": [
    "my_tidy_data = pd.DataFrame({\"RTs\": values_col, \"sex\": sexes, \"strain\": strain})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "671e9a24",
   "metadata": {},
   "source": [
    "It's just not as pretty."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1ff9b28",
   "metadata": {},
   "source": [
    "Let's look at our creation!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "466c0eef",
   "metadata": {},
   "outputs": [],
   "source": [
    "my_tidy_data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f2c021d",
   "metadata": {},
   "source": [
    "Yay! We win!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b07d3de6",
   "metadata": {},
   "source": [
    "**Important point:** Crucially, *the above code doesn't rely on us knowing much about the input data ahead of time*. As long as it's a pandas data frame that contains numerical values, the code will run. It's automatic."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6aedc741",
   "metadata": {},
   "source": [
    "## Look at new data with more observations with same code"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "596bad00",
   "metadata": {},
   "source": [
    "We'll make this code self-contained, so it can be run without running anything above. We'll also add comments, so that future-us can read the code more easily without having to wade through the notebook text above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "277d872c",
   "metadata": {},
   "outputs": [],
   "source": [
    "my_input_data = pd.read_csv('datasets/018DataFile.csv')  # read the data\n",
    "\n",
    "raw_data = my_input_data.to_numpy()                      # convert to numpy array\n",
    "\n",
    "obs, grps = raw_data.shape                               # get the number of rows and columns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3184713d",
   "metadata": {},
   "source": [
    "Check the size of the new data real quick:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ae481c54",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"We have \", obs, \" observations per group and \", grps, \" groups.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da45fddb",
   "metadata": {},
   "source": [
    "And now run the \"meat\" of the code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "49a156d8",
   "metadata": {},
   "outputs": [],
   "source": [
    "new_length = obs*grps                                    # compute total number of observations\n",
    "\n",
    "values_col = np.reshape(raw_data, (new_length, 1), \n",
    "                        order = 'F')                     # reshape the array\n",
    "values_col = np.squeeze(values_col)                      # squeeze to make 1D\n",
    "\n",
    "# construct the inner grouping variable\n",
    "sexes = pd.Series(['male', 'female'])                    # define the levels\n",
    "sexes = sexes.repeat(obs)                                # make one cycle of the levels\n",
    "sexes = pd.concat([sexes]*2, ignore_index=True)          # and repeat the cycle, ditching the indexes\n",
    "\n",
    "# construct the outer grouping variable\n",
    "strain = pd.Series(['wildtype', 'mutant'])               # define the levels\n",
    "strain = strain.repeat(2*obs)                            # make the one cycle\n",
    "strain = strain.reset_index(drop=True)                   # drop the pesky index\n",
    "\n",
    "# construct the data frame\n",
    "my_new_tidy_data = pd.DataFrame(\n",
    "    {\n",
    "        \"RTs\": values_col,                               # make a column named RTs and put the values in\n",
    "        \"sex\": sexes,                                    # ditto for sex\n",
    "        \"strain\": strain                                 # and for genetic strain\n",
    "    }    \n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "248e597e",
   "metadata": {},
   "outputs": [],
   "source": [
    "my_new_tidy_data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26e786fe",
   "metadata": {},
   "source": [
    "**Success!**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bbf6f3ca",
   "metadata": {},
   "source": [
    "## Making the code even more functional"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c51ad194",
   "metadata": {},
   "source": [
    "Now we have a chunk of code that seems handy and re-usable. How could we make it ever more handy?\n",
    "\n",
    "If we make it into a ***function***, then we can run the whole entire thing just by typing one command – no copying, no pasting, fewer ways to make mistakes."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "116d79e2",
   "metadata": {},
   "source": [
    "### Defining a function\n",
    "Since we already have all the code, we can literally just indent it and throw a `def...` in front of it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1bd7aefb",
   "metadata": {},
   "outputs": [],
   "source": [
    "def tidyMyData() :\n",
    "    import pandas as pd\n",
    "    import numpy as np\n",
    "\n",
    "    my_input_data = pd.read_csv('datasets/018DataFile.csv')  # read the data\n",
    "\n",
    "    raw_data = my_input_data.to_numpy()                      # convert to numpy array\n",
    "\n",
    "    obs, grps = raw_data.shape                               # get the number of rows and columns\n",
    "\n",
    "    new_length = obs*grps                                    # compute total number of observations\n",
    "\n",
    "    values_col = np.reshape(raw_data, (new_length, 1), \n",
    "                            order = 'F')                     # reshape the array\n",
    "    values_col = np.squeeze(values_col)                      # squeeze to make 1D\n",
    "\n",
    "    # construct the inner grouping variable\n",
    "    sexes = pd.Series(['male', 'female'])                    # define the levels\n",
    "    sexes = sexes.repeat(obs)                                # make one cycle of the levels\n",
    "    sexes = pd.concat([sexes]*2, ignore_index=True)     # and repeat the cycle, ditching the indexes\n",
    "\n",
    "    # construct the outer grouping variable\n",
    "    strain = pd.Series(['wildtype', 'mutant'])               # define the levels\n",
    "    strain = strain.repeat(2*obs)                            # make the one cycle\n",
    "    strain = strain.reset_index(drop=True)                   # drop the pesky index\n",
    "\n",
    "    # construct the data frame\n",
    "    my_new_tidy_data = pd.DataFrame(\n",
    "        {\n",
    "            \"RTs\": values_col,                               # make a column named RTs and put the values in\n",
    "            \"sex\": sexes,                                    # ditto for sex\n",
    "            \"strain\": strain                                 # and for genetic strain\n",
    "        }    \n",
    "    )\n",
    "    \n",
    "    return my_new_tidy_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7acf5fdd",
   "metadata": {},
   "outputs": [],
   "source": [
    "datFromFun = tidyMyData()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "056d4f2e",
   "metadata": {},
   "outputs": [],
   "source": [
    "datFromFun"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21c107a0",
   "metadata": {},
   "source": [
    "### Defining a function with an argument\n",
    "A common (very common) scenario in data analysis is wanting to run the same code – like the code we just wrote – on different files. So one really nice addition to this function would be to add the ability for the user to specify a filename to tell the function which data file to read.\n",
    "\n",
    "This is actually fairly straightforward. All we have to do as add an **argument** to our function, and then replace the hardcoded filename in the function with the **variable** created by the function argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7dec064a",
   "metadata": {},
   "outputs": [],
   "source": [
    "def tidyMyData(filename) :\n",
    "    import pandas as pd\n",
    "    import numpy as np\n",
    "\n",
    "    my_input_data = pd.read_csv(filename)  # read the data\n",
    "\n",
    "    raw_data = my_input_data.to_numpy()                      # convert to numpy array\n",
    "\n",
    "    obs, grps = raw_data.shape                               # get the number of rows and columns\n",
    "\n",
    "    new_length = obs*grps                                    # compute total number of observations\n",
    "\n",
    "    values_col = np.reshape(raw_data, (new_length, 1), \n",
    "                            order = 'F')                     # reshape the array\n",
    "    values_col = np.squeeze(values_col)                      # squeeze to make 1D\n",
    "\n",
    "    # construct the inner grouping variable\n",
    "    sexes = pd.Series(['male', 'female'])                    # define the levels\n",
    "    sexes = sexes.repeat(obs)                                # make one cycle of the levels\n",
    "    sexes = pd.concat([sexes]*2, ignore_index=True)     # and repeat the cycle, ditching the indexes\n",
    "\n",
    "    # construct the outer grouping variable\n",
    "    strain = pd.Series(['wildtype', 'mutant'])               # define the levels\n",
    "    strain = strain.repeat(2*obs)                            # make the one cycle\n",
    "    strain = strain.reset_index(drop=True)                   # drop the pesky index\n",
    "\n",
    "    # construct the data frame\n",
    "    my_new_tidy_data = pd.DataFrame(\n",
    "        {\n",
    "            \"RTs\": values_col,                               # make a column named RTs and put the values in\n",
    "            \"sex\": sexes,                                    # ditto for sex\n",
    "            \"strain\": strain                                 # and for genetic strain\n",
    "        }    \n",
    "    )\n",
    "    \n",
    "    return my_new_tidy_data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6dae764b",
   "metadata": {},
   "source": [
    "Now we can call the function and specify whatever data files exist. Let's try it with \"datasets/018DataFile2.csv\"!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b60521d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "newDataFromFun = tidyMyData(\"datasets/018DataFile2.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c1febb1",
   "metadata": {},
   "outputs": [],
   "source": [
    "newDataFromFun"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "940c4d21",
   "metadata": {},
   "source": [
    "### Adding help\n",
    "It's always a good idea to **heavily comment your code!** \n",
    "\n",
    "When writing fuctions, it's also a good idea to add a documentation string, called a `docstring`, to your function. This way people can get help on your function with the `help()` function. Like `help(tidyMyData)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11a23e91",
   "metadata": {},
   "outputs": [],
   "source": [
    "def tidyMyData(filename) :\n",
    "    '''\n",
    "    tidyMyData() Takes one-column-per-cell rat reaction time data as input.\n",
    "    Returns tidy one-column-per-variable data.\n",
    "    User specifies a filename string.\n",
    "    '''\n",
    "    \n",
    "    import pandas as pd\n",
    "    import numpy as np\n",
    "\n",
    "    my_input_data = pd.read_csv(filename)  # read the data\n",
    "\n",
    "    raw_data = my_input_data.to_numpy()                      # convert to numpy array\n",
    "\n",
    "    obs, grps = raw_data.shape                               # get the number of rows and columns\n",
    "\n",
    "    new_length = obs*grps                                    # compute total number of observations\n",
    "\n",
    "    values_col = np.reshape(raw_data, (new_length, 1), \n",
    "                            order = 'F')                     # reshape the array\n",
    "    values_col = np.squeeze(values_col)                      # squeeze to make 1D\n",
    "\n",
    "    # construct the inner grouping variable\n",
    "    sexes = pd.Series(['male', 'female'])                    # define the levels\n",
    "    sexes = sexes.repeat(obs)                                # make one cycle of the levels\n",
    "    sexes = pd.concat([sexes]*2, ignore_index=True)     # and repeat the cycle, ditching the indexes\n",
    "\n",
    "    # construct the outer grouping variable\n",
    "    strain = pd.Series(['wildtype', 'mutant'])               # define the levels\n",
    "    strain = strain.repeat(2*obs)                            # make the one cycle\n",
    "    strain = strain.reset_index(drop=True)                   # drop the pesky index\n",
    "\n",
    "    # construct the data frame\n",
    "    my_new_tidy_data = pd.DataFrame(\n",
    "        {\n",
    "            \"RTs\": values_col,                               # make a column named RTs and put the values in\n",
    "            \"sex\": sexes,                                    # ditto for sex\n",
    "            \"strain\": strain                                 # and for genetic strain\n",
    "        }    \n",
    "    )\n",
    "    \n",
    "    return my_new_tidy_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37d81f55",
   "metadata": {},
   "outputs": [],
   "source": [
    "help(tidyMyData)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d746e2f2",
   "metadata": {},
   "source": [
    "$\\color{blue}{\\text{Complete the following exercise.}}$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2d3fc95",
   "metadata": {},
   "source": [
    "- Use the cell below to show how you would modify the previous function so as to make it even more flexible. Let the user specify the output column headers to be whatever they want.\n",
    "\n",
    "More specifically how would you allos passing in the three labels, `sex`, `RTs` and `strain`, instead of having them 'hard coded' inside the code. This means that instead of using labels such as `sex`, `RTs` and `strain`, we will want to pass paramters for each one of the labels and use the parameters in the function. For example, instead of `sex`, `RTs` and `strain` we will want to pass others say, `s`, `ReactionTime` or `type` or any three combinations of lables, always three but that can change everytime we call the function.\n",
    "\n",
    "You would do this with arguments (obviously). But you could do it with multiple arguments, so users would call it like:\n",
    "\n",
    "`tidyMyData(\"datasets/018DataFile2.csv\", \"Times\", \"Gender\", \"Genotype\")`\n",
    "\n",
    "or you could do it with one additional arguments, so the user would call it by either:\n",
    "\n",
    "`tidyMyData(\"datasets/018DataFile2.csv\", [\"Times\", \"Gender\", \"Genotype\"])`\n",
    "\n",
    "or\n",
    "\n",
    "`colNames = [\"Times\", \"Gender\", \"Genotype\"]`\n",
    "\n",
    "`tidyMyData(\"datasets/018DataFile2.csv\", colNames)`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e9cd8dc",
   "metadata": {},
   "source": [
    "Pro tip: The function would probably be most handy if there were *default* values for the column names, so that user could just type something like\n",
    "\n",
    "`myTidyData = tidyMyData(\"datasets/018DataFile2.csv\")`\n",
    "\n",
    "if they didn't want to specify custom column headers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "375832c3",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}