12_python_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data science in Python\n",
    "\n",
    "- Course GitHub repo: https://github.com/pycam/python-data-science\n",
    "- Python website: https://www.python.org/ \n",
    "\n",
    "## Session 1.2: Using existing python modules to explore data in files\n",
    "\n",
    "- [Importing module `statistics`](#Importing-module-statistics)\n",
    "  - [Exercise 1.2.1](#Exercise-1.2.1)\n",
    "- [Python file and directory manipulations](#Python-file-and-directory-manipulations)\n",
    "  - [Exercise 1.2.2](#Exercise-1.2.2)\n",
    "- [Using the `csv` module](#Using-the-csv-module)\n",
    "  - [Exercise 1.2.3](#Exercise-1.2.3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Mind map\n",
    "\n",
    "<img src=\"img/mind_maps/mind_maps.002.jpeg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing module `statistics`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Like other laguages, Python has the ability to import external modules (or libraries) into the current program. These modules may be part of the standard library that is automatically included with the Python installation, they may be extra libraries which you install separately or they may be other Python programs you have written yourself. Whatever the source of the module, they are imported into a program via an **`import`** command.\n",
    "\n",
    "For example, if we wish to access the `mean()` and `median()` functions in Python, we can use the **`import`** keyword to get [the module named `statistics`](https://docs.python.org/3/library/statistics.html) and access its contents with the dot notation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import statistics\n",
    "statistics.mean([1, 2, 3, 4, 4])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also we can use the `as` keyword to give the module a different name in our code, which can be useful for brevity and avoiding name conflicts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import statistics as stats\n",
    "stats.mean([1, 2, 3, 4, 4])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively we can import the separate components using the `from … import` keyword combination:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from statistics import mean, median\n",
    "\n",
    "mean([1, 2, 3, 4, 4])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Listing module contents\n",
    "\n",
    "Using the [function `dir()`](https://docs.python.org/3/library/functions.html?highlight=dir#dir) and passing the module name:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import statistics\n",
    "dir(statistics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Getting help directly from Jupyter notebook\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "statistics?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 1.2.1\n",
    "\n",
    "- Calculate the average GDP per capita per country in Europe in 1962, its median and standard deviation using `data/gapminder.csv` data; and compare these figures with those from Americas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Python file and directory manipulations\n",
    "\n",
    "These two modules `os.path` and `os` implements some useful functions on pathnames, and for accessing the filesystem. To read or write files, we use `open()`. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [`os.path` — Common pathname manipulations](https://docs.python.org/3/library/os.path.html)\n",
    "\n",
    "- `join(*paths)` : joins the paths together into one long path\n",
    "- `exists(path)` : returns whether path exists\n",
    "- `isfile(path)` : returns whether path is a “regular” file (as opposed to a directory)\n",
    "- `isdir(path)` : returns whether path is a directory\n",
    "- `dirname(path)` : returns directory containing the path\n",
    "- `basename(path)` : returns the path minus the dirname(path) in front\n",
    "- `split(path)` : returns (dirname(path), basename(path))\n",
    "\n",
    "### [`os` — Miscellaneous operating system interfaces](https://docs.python.org/3/library/os.html)\n",
    "\n",
    "- `listdir(path)` : returns a list of files/directories in the directory path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Building the path to your file from a list of directory and filename makes your script able to run on any platforms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os.path\n",
    "data_filepath = os.path.join(\"data\", \"gapminder.csv\")\n",
    "# data/mydata.txt - Unix\n",
    "# data\\mydata.txt - Windows\n",
    "print(data_filepath)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Checking if a file exists before opening it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.path.exists(data_filepath)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Checking if it is a file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.path.isfile(data_filepath)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "or a directory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.path.isdir(data_filepath)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Extracting the directory of the file path:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_dirname = os.path.dirname(data_filepath)\n",
    "print(data_dirname)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Checking if it is a directory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.path.isdir(data_dirname)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Extracting the file name from the file path:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_filename = os.path.basename(data_filepath)\n",
    "print(data_filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Getting the directory and the file name from the file path using `os.path.split()` which returns two variables its directory and file name:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_dirname, data_filename = os.path.split(data_filepath)\n",
    "print(data_dirname, data_filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Listing the content of a directory using `os.listdir()` is equivalent to `ls` in the shell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "print(os.listdir(data_dirname))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 1.2.2\n",
    "\n",
    "- Print the name of each `.csv` file in the `data` folder.\n",
    "- Print the first 3 lines from each `.csv` file in the `data` folder.\n",
    "\n",
    "You may wish to use the [`enumerate`](https://docs.python.org/3/library/functions.html?highlight=enumerate#enumerate) function, along with a `break` statement to avoid printing every line in the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i, letter in enumerate(['A', 'B', 'C']):\n",
    "    print(i, letter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using the `csv` module\n",
    "\n",
    "The so-called CSV (Comma Separated Values) format is the most common import and export format for spreadsheets and databases. The `csv` module implements methods to read and write tabular data in CSV format.\n",
    "\n",
    "The csv module’s `reader()` and `writer()` methods read and write CSV files. You can also read and write data into dictionary form using the `DictReader()` and `DictWriter()` methods.\n",
    "\n",
    "For more information about this built-in Python library go to [CSV File Reading and Writing documentation](https://docs.python.org/3/library/csv.html).\n",
    "\n",
    "Let's now read our `data/genes.txt` tab separated file using the `csv` module into a dictionary based on the column headers using `csv.DictReader()`.\n",
    "\n",
    "|gene |\tchrom |\tstart |\tend |\n",
    "|-- | -- | -- | -- | \n",
    "|BRCA2 |\t13 |\t32889611 |\t32973805 |\n",
    "|TNFAIP3 |\t6 |\t138188351 |\t138204449 |\n",
    "|TCF7 |\t5 |\t133450402 |\t133487556 |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, import the `csv` module:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read the data and store each dictionary into a list. Note that `DictReader()` returns an [ordered dictionary](https://docs.python.org/3/library/collections.html#ordereddict-objects).\n",
    "\n",
    "Ordered dictionaries are like regular dictionaries but they remember the order that items were inserted. When iterating over an ordered dictionary, the items are returned in the order their keys were first added."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = []\n",
    "with open(\"data/genes.txt\") as f:\n",
    "    reader = csv.DictReader(f, delimiter = \"\\t\")\n",
    "    for row in reader:  \n",
    "        data.append(row)\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for d in data:\n",
    "    print(d['chrom'], d['gene'], d['start'], d['end'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "data is a list of ordered dictionary representing each row of the data file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# accessing first dictionary from the list\n",
    "print(data[0])\n",
    "\n",
    "# printing its keys\n",
    "print(data[0].keys())\n",
    "\n",
    "# its values\n",
    "print(data[0].values())\n",
    "\n",
    "# the value associated with the key 'gene'\n",
    "print(data[0]['gene'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# looping over the list to print each gene\n",
    "for d in data:\n",
    "    print(d['gene'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculating the length of each gene and adding its value into the dictionary\n",
    "for d in data:\n",
    "    d['len'] = int(d['end']) - int(d['start']) + 1\n",
    "    print(d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The main advantage of using the `DictReader()` method and the `csv` module is to write code that is easier to read and more flexible. Using the name of the column instead if its index make it more meaningful when reading code, and using this method of reading comma or tab separated files, give you the flexibility to add columns and changed their orders without having to modify your code.\n",
    "\n",
    "Let's have a look now at the file `data/genes_withstrand.txt` and spot the differences with `data/genes.txt`. Even though columns `chrom` and `gene` have been swapped and column `strand` added, the code written previously is still working."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_withstrand = []\n",
    "with open(\"data/genes_withstrand.txt\") as f:\n",
    "    reader = csv.DictReader(f, delimiter = \"\\t\")\n",
    "    for row in reader:\n",
    "        print(row)\n",
    "        data_withstrand.append(row)\n",
    "\n",
    "for d in data_withstrand:\n",
    "    print(d['chrom'], d['gene'], d['start'], d['end'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write a delimited file using the csv module from a list of dictionaries \n",
    "with open(\"gene_lengths.txt\", \"w\") as f:\n",
    "    writer = csv.DictWriter(f, data[0].keys(), delimiter='\\t')\n",
    "    writer.writeheader() # write header\n",
    "\n",
    "    for d in data:\n",
    "        writer.writerow(d) # write row\n",
    "\n",
    "# Open the output file and print out its content\n",
    "with open(\"gene_lengths.txt\") as f:\n",
    "    for line in f:\n",
    "        print(line.strip())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting help from the official Python documentation\n",
    "\n",
    "The most useful information is online on https://www.python.org/ website and should  be used as a reference guide.\n",
    "\n",
    "- [Python3 documentation](https://docs.python.org/3/) is the starting page with links to tutorials and libraries' documentation for Python 3\n",
    "    - [The Python Tutorial](https://docs.python.org/3/tutorial/index.html)\n",
    "        - [Modules](https://docs.python.org/3/tutorial/modules.html)\n",
    "        - [Brief Tour of the Standard Library: Mathematics](https://docs.python.org/3/tutorial/stdlib.html#mathematics)\n",
    "    - [The Python Standard Library Reference](https://docs.python.org/3/library/index.html) is the reference documentation of all libraries included in Python like:\n",
    "        - [`statistics` - Mathematical statistics functions](https://docs.python.org/3/library/statistics.html)\n",
    "        - [`os.path` — Common pathname manipulations](https://docs.python.org/3/library/os.path.html)\n",
    "        - [`os` — Miscellaneous operating system interfaces](https://docs.python.org/3/library/os.html)\n",
    "        - [`csv` — CSV File Reading and Writing](https://docs.python.org/3/library/csv.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 1.2.3\n",
    "\n",
    "- Change the script you wrote for [Exercise 1.2.1](#Exercise-1.2.1) to make use of the `csv` module to calculate the average GDP per capita per country in Europe in 1962, its median and standard deviation using `data/gapminder.csv` data; and compare these figures with those from Americas."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next session\n",
    "\n",
    "Go to our next notebook: [Session 1.3: Creating functions and modules to write reusable code](13_python_data.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}