07_BestPractice_Validation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JamesFergusson/Introduction-to-Research-Computing/blob/master/07_BestPractice_Validation.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Validating you code\n",
    "\n",
    "Once you have written some initial code from your prototype with good documentation and a nice modular structure (and of course carefully tracked by git) you will need to check that it is correct.  There are a few things you can do to make this much easier:\n",
    "\n",
    "- Debug\n",
    "- Error trapping\n",
    "- Unit testing\n",
    "- Continuous Integration\n",
    "\n",
    "For the first Python has a good inbuilt debugger that will help you, `pdg`\n",
    "\n",
    "## Debugging\n",
    "If you run code that has raised an exception in jupyter or ipython you can use the magic command `%debug` to launch an interactive debugger where you can examine the code line by line.  Here are the most useful commands\n",
    "\n",
    "| Command | Desciption |\n",
    "|---|---|\n",
    "| u | Up |\n",
    "| d | Down|\n",
    "| p | Print |\n",
    "| q | Quit |\n",
    "\n",
    "This is best seen in an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def bottom_func(x):\n",
    "    y = x**2\n",
    "    return str(x) + \" squared equals \" + str(y)\n",
    "\n",
    "def top_func(z):\n",
    "    result = bottom_func(z)\n",
    "    return result\n",
    "        \n",
    "top_func('7')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%debug"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This can be turned on automatically with `%pdb on` so whenever an exception is raised the debugger is launched automatically"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also run the code interactivly line by line using `%run -d` which can be more useful if your code is just wrong rather than breaking.  If you launch code with this then you can step through it with the following commands:\n",
    "\n",
    "| Command | Desciption |\n",
    "|---|---|\n",
    "| n | Next line |\n",
    "| s | Step into function|\n",
    "| c | Continue to run normally |\n",
    "| q | Quit |\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run -d Examples/UnitTesting/simple.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also run the debugger from the command line with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    ">> python3 -m pdb myscript.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Error trapping\n",
    "\n",
    "It is a good idea to write you code so that when errors occours the code reports this to you.  Python is pretty good a providing reaonable error messages but it's good practice to make you own.  One simeple place to check things is whether the input makes sense. So for fibonacci we could add: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fibonacci(num):\n",
    "    \n",
    "    if not isinstance(num,int):\n",
    "        print(\"non-integer input given to function fibonacci; num=\"+str(num))\n",
    "        return 0\n",
    "    \n",
    "    a=0\n",
    "    b=1\n",
    "    for i in range(num):\n",
    "        a,b = b,a+b\n",
    "    return a\n",
    "\n",
    "fibonacci(3.4)\n",
    "fibonacci(3)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should also check error codes that come back from routines and report on them, for instance reading a file. Better examples are checking inputs are in valid ranges for functions you've created or other things which won't cause python to crash but will mean you code gives the wrong answer, for example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def harmonic_mean(list):\n",
    "    x = 0e0\n",
    "    for item in list:\n",
    "        if item==0:\n",
    "            print(\"Harmonic mean does not exist for data containing zeros.  Returned 'None'\")\n",
    "            return None\n",
    "        \n",
    "        x+= 1e0 / item\n",
    "    x = len(list)/x\n",
    "    \n",
    "    return x\n",
    "\n",
    "list1 = [1,2,3,4,5,6,7,8,9]\n",
    "list2 = [1,2,3,4,5,6,7,8,9,0]\n",
    "print(harmonic_mean(list1))\n",
    "print(harmonic_mean(list2)) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Unit testing\n",
    "\n",
    "The best way to debug your code is to catch them before they happen which you can do with unit testing.  The ideas is that you would set up a bunch of tests for the code then everytime you do a commit to master or after doing major edits you run them to check you haven't broken anything.  For the code calculating the integral in the previous notebook you may want to to create tests that check you polynomials are OK (by checking orthonormality) or that the final integral with Gauss-Legendre quadriture is correct for large l (ie set X = $\\delta_{l,200}$ and see if you get the correct answer, 0.000018285996687338485).  \n",
    "\n",
    "Having set up these tests you can then automate them to create a test package that runs, say, before a push to a central repository.  This is standard practise in commercial development enviroments and if you can include them in interview test questions this will put you above the majority of applicants.\n",
    "\n",
    "It's a good idea to get into the habit of adding them for functions, ideally before you write it.  Then you can use them to check your code does what you thought it should.  You will end up spending lots of time checking your code when you are trying to fix bugs so setting up the tests in advance can save alot of time. Luckily in python basic ones are easy to do, you can just add it to the docstring"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import doctest\n",
    "\n",
    "def function(x):\n",
    "    \"\"\"\n",
    "    Calculate x + 2\n",
    "    >>> function(5)\n",
    "    7\n",
    "    \"\"\"\n",
    "    return x+3\n",
    "\n",
    "doctest.testmod()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is fine for super simple tests but isn't much use once you write functions that process data rather than just a number.  There are alot of packages available but `pytest` is the standard (which is not to say it's the best).  This runs from the terminal and looks for any functions with the name `test_somefunction` or `somefunction_test` then runs them.  These functions should contain some code to run then tests to apply to the outputs using the command `assert` which accepts any booleian argument.  If our function was:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def addtwo(x):\n",
    "    \"\"\"\n",
    "        Add 2 to x\n",
    "    \"\"\"\n",
    "    return x+1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then our test could look like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_addtwo():\n",
    "    \"\"\"\n",
    "        Test addtwo\n",
    "    \"\"\"\n",
    "    assert( addtwo(3)==5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are in the files simple.py in the directory `Examples/UnitTesting`. We can test them from the command line using"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "pytest Examples/UnitTesting/simple.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The tests can also be held in seperate files like test_simple.py:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "pytest Examples/UnitTesting/test_simple1.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One thing to remember is that floating point arithmitic is not exact so the test of add02 in test_simple2.py fails"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "pytest Examples/UnitTesting/test_simple2.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To fix this pytest had a function called `approx` which by default allows a relative tolerance of 1e-6 which is mostly fine and it works on most data objects:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pytest import approx\n",
    "import numpy as np\n",
    "print(0.1 + 0.2 == approx(0.3))\n",
    "print((0.1 + 0.2, 0.2+0.4) == approx((0.3,0.6)))\n",
    "print({'a': 0.1 + 0.2, 'b': 0.2 + 0.4} == approx({'a': 0.3, 'b': 0.6}))\n",
    "print(np.array([0.1, 0.2]) + np.array([0.2, 0.4]) == approx(np.array([0.3, 0.6])))\n",
    "print(np.array([0.1, 0.2]) + np.array([0.2, 0.1]) == approx(0.3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However if you are testing things near zero relative tolerances are useless.  Luckly `approx` also allows you to change the tolerances and make them relative or absolute.  If you specify both, it is true if either are satisfied."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pytest import approx\n",
    "print(1.0001 == approx(1))\n",
    "print(1.0001 == approx(1, rel=1e-3))\n",
    "print(1.0001 == approx(1, abs=1e-3))\n",
    "print(1.0001 == approx(1, rel=1e-5, abs=1e-3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also specify a specific failure message with `fail` ie:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pytest import fail\n",
    "def test_something():\n",
    "    x = somefunc()\n",
    "    if x in badthings:\n",
    "        fail('A bad thing came back from somefunc()')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally note that if we specify no arguments `pytest` looks for files names `test_fimename` or `filename_test` and runs them.  In general is is best practice to put your source code and you tests in different directories as it keeps them seperate and safe.  You should have one test file for each module.  Then running pytest in the test directory will check all your code or you can run on individual modules if you want."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "pytest Examples/UnitTesting/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Continuous Integration\n",
    "\n",
    "Now the above is all useful but the real trick is to automate it.  Continuous Integration is a development practice where developers integrate code into a shared repository frequently, preferably several times a day. Each integration can then be verified by an automated build and automated tests.  This stops people from introducing errors which are not found until much later as the code must pass all tests before it can be accepted into the main repository.\n",
    "\n",
    "As this is a standard development practice there are many tools for this. Common options are: `Jenkins`, `Circle CI`, `TeamCity`, or `Bamboo`.  You can run it localy on your machine or via the cloud.  Most will create multiple virtual machines so you can test on multiple versions of python simultaneously to ensure code stability and most have free options.  Setting these up can be a bit tricky so one of the easiest options is to use the CI tools built into the popular GIT hosting sites. `github`, `gitlab`, and `bitbucket` all have the tools built in and template scripts you can use.\n",
    "\n",
    "We will show you how to set up a simple CI routine using the tools in GitHub to get you started.  We will use the following code for our example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%file CI_Test/basic_maths.py\n",
    "\"\"\"\n",
    "basic math library.\n",
    "\"\"\"\n",
    "\n",
    "\n",
    "def add(a, b):\n",
    "    \"\"\"\n",
    "    add a and b\n",
    "    \"\"\"\n",
    "    return a+b\n",
    "\n",
    "\n",
    "def minus(a, b):\n",
    "    \"\"\"\n",
    "    subtract b from a\n",
    "    \"\"\"\n",
    "    return a-b\n",
    "\n",
    "\n",
    "def multiply(a, b):\n",
    "    \"\"\"\n",
    "    multiply a and b\n",
    "    \"\"\"\n",
    "    return a*b\n",
    "\n",
    "\n",
    "def divide(a, b):\n",
    "    \"\"\"\n",
    "    divide a by b\n",
    "    \"\"\"\n",
    "    return a/b\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%file CI_Test/test_basic_maths.py\n",
    "\"\"\"\n",
    "test basic_maths.py.\n",
    "\"\"\"\n",
    "\n",
    "import basic_maths as bm\n",
    "\n",
    "\n",
    "def test_add():\n",
    "    \"\"\"\n",
    "    Test add\n",
    "    \"\"\"\n",
    "    assert(bm.add(6, 3) == 9)\n",
    "\n",
    "\n",
    "def test_minus():\n",
    "    \"\"\"\n",
    "    Test minus\n",
    "    \"\"\"\n",
    "    assert(bm.minus(6, 3) == 3)\n",
    "\n",
    "\n",
    "def test_multiply():\n",
    "    \"\"\"\n",
    "    Test multiply\n",
    "    \"\"\"\n",
    "    assert(bm.multiply(6, 3) == 18)\n",
    "\n",
    "\n",
    "def test_divide():\n",
    "    \"\"\"\n",
    "    Test divide\n",
    "    \"\"\"\n",
    "    assert(bm.divide(6, 3) == 2)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1st. - We will copy the code in these two files into a folder, create a git repository and commit them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "git init\n",
    "git add basic_maths.py test_basic_maths.py\n",
    "git commit -m \"Initial commit\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2nd. We upload it to GitHub\n",
    "\n",
    "- login to github\n",
    "- click 'repositories'\n",
    "- click 'new'\n",
    "- follow instructions for \"push an existing repository from the command line\"\n",
    "\n",
    "3rd. We click `actions` and select `Python Package` (or `Python Package using Anaconda` / `Python Application`). then `start commit` then `commit new file`.  This creates the following file in the new folder .github/workflows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This workflow will install Python dependencies, run tests and lint with a variety of Python versions\n",
    "# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions\n",
    "\n",
    "name: Python package\n",
    "\n",
    "on:\n",
    "  push:\n",
    "    branches: [ main ]\n",
    "  pull_request:\n",
    "    branches: [ main ]\n",
    "\n",
    "jobs:\n",
    "  build:\n",
    "\n",
    "    runs-on: ubuntu-latest\n",
    "    strategy:\n",
    "      matrix:\n",
    "        python-version: [3.5, 3.6, 3.7, 3.8]\n",
    "\n",
    "    steps:\n",
    "    - uses: actions/checkout@v2\n",
    "    - name: Set up Python ${{ matrix.python-version }}\n",
    "      uses: actions/setup-python@v2\n",
    "      with:\n",
    "        python-version: ${{ matrix.python-version }}\n",
    "    - name: Install dependencies\n",
    "      run: |\n",
    "        python -m pip install --upgrade pip\n",
    "        pip install flake8 pytest\n",
    "        if [ -f requirements.txt ]; then pip install -r requirements.txt; fi\n",
    "    - name: Lint with flake8\n",
    "      run: |\n",
    "        # stop the build if there are Python syntax errors or undefined names\n",
    "        flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics\n",
    "        # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide\n",
    "        flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics\n",
    "    - name: Test with pytest\n",
    "      run: |\n",
    "        pytest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have set up the test and it will run on any push or pull request to the branch main.  To see this we can first do a git pull to update the local repository then edit the comment at the top of `basic_maths.py` the `add`/`commit`/`push`.  Now if we navigate to GitHub and click on actions we can see the tests running.  At the moment there is nothing to stop us pushing rubbish that fails the test and it being added to main.  To stop this we need to protect it.\n",
    "\n",
    "Go to `settings` and click `branches` and click `add rule` then for `branch name pattern` write \"main\" then select `Require status checks to pass before merging` and select the 4 tests then select `Include administrators` and click `Create`.\n",
    "\n",
    "Now go back and edit the comment at the top of `basic_maths.py` again and try to push.  You should now get the message:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "remote: error: GH006: Protected branch update failed for refs/heads/main.\n",
    "remote: error: 4 of 4 required status checks are expected. At least 1 approving review is required by reviewers with write access.\n",
    "To https://github.com/JamesFergusson/CI_Test.git\n",
    " ! [remote rejected] main -> main (protected branch hook declined)\n",
    "error: failed to push some refs to 'https://github.com/JamesFergusson/CI_Test.git'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To edit the code we now have to use branches which we then merge with the main in GitHub. so create a branch and switch to it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "git branch \"test\"\n",
    "git checkout test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now add the following funtion to both basic_functions.py and test_basic_functions.py:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def exponentiate(a, b):\n",
    "    \"\"\"\n",
    "    calculate a to the power of b\n",
    "    \"\"\"\n",
    "    return a^b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_exponentiate():\n",
    "    \"\"\"\n",
    "    Test exponentiate\n",
    "    \"\"\"\n",
    "    assert(bm.exponentiate(6, 3) == 216)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now `add`/`commit` then `git push --set-upstream origin test`.  You should ge the message:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "remote: Resolving deltas: 100% (2/2), completed with 2 local objects.\n",
    "remote: \n",
    "remote: Create a pull request for 'test' on GitHub by visiting:\n",
    "remote:      https://github.com/JamesFergusson/CI_Test2/pull/new/test\n",
    "remote: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "now go back to GitHub and click `Pull Requests` then `new pull request` and under `compare` select `test`.  This should say the tests have failed so we can't merge into main.  Go back and fix the function to:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def exponentiate(a, b):\n",
    "    \"\"\"\n",
    "    calculate a to the power of b\n",
    "    \"\"\"\n",
    "    return a**b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then `add`/`commit`/`push`.  Now go back to the `Pull Request` which should pass and now you can merge `test` with `main`.  Now your `main` branch is safe and can't be commited to.  without all testing passing.  This is a fairly basic CI setup which works fine for very small teams but there are a huge number of options.  The documentation is mostly OK once you have the basics so you can teach yourself but if you want to create CI for a medium size team (3+) it might be best to ask for some computer officer support.  That said, the tools for managing it are advancing rapidly so it's worth exploring what is available regularly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise: Game of life\n",
    "\n",
    "Write a small module that runs Conway's game of life, https://en.wikipedia.org/wiki/Conway%27s_Game_of_Life.  You start with a 2D grid (whose size is to be specified at runtime) where the cells are either 'alive' or 'dead'.  Then the rules for stepping in time are:\n",
    "\n",
    "- Overpopulation. Live cells with more than 3 neighbours die\n",
    "- Underpopulation. Live cells with less than 2 neighbours die\n",
    "- Reproduction. Dead cells with exactly 3 neighbours become live\n",
    "\n",
    "Boundaries are periodic\n",
    "\n",
    "The goal is to:\n",
    "1. Prototype \n",
    "    - It should take a starting set of cells or generate a random board\n",
    "    - Have a \"step\" function which advanced the board one timestep\n",
    "    - Have a \"run\" function which evolves the game n steps and plots/animates the result\n",
    "2. Design unit tests (The \"Oscillators\" from the wikipedia page are good for test cases)\n",
    "3. THEN! Write the code, try to use debug and include error traps.  You can use your unit tests to validate as you go.. See if you can use CI for the project.\n",
    "\n",
    "You can create all the logic yourself but the following functions will make it much easier: 2D convolution from https://docs.scipy.org/doc/scipy-0.15.1/reference/signal.html, which will count your neighbours, and then https://docs.scipy.org/doc/numpy/reference/generated/numpy.where.html for implementing the rules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}