Lesson_AF_07_command_line_applications.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# writing command line applications\n",
    "\n",
    "## using *plumbum*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Command line applications are a generalization of the idea of script: the execution of the program is conditioned on some parameters provided by the user.\n",
    "\n",
    "This is typically done using **options**, **parameters** and **flags**, and are the typical interface used by bash programs.\n",
    "\n",
    "In this lesson we will (quickly) overview how to implement one of these program, leveraging a 3rd party library called **plumbum**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Everything we are doing today with **plumbum** can be done in a reasonable way in basic python, but plumbum will provide us a more simple and consistent way of doing it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1, 6, 7)\n"
     ]
    }
   ],
   "source": [
    "import plumbum\n",
    "# this script uses plumbum version 1.6.7\n",
    "print(plumbum.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Why applications?\n",
    "\n",
    "functionalities implemented as CLI applications share a lot of the advantages with libraries\n",
    "\n",
    "1. Commands are repeatable\n",
    "2. Commands can be shared\n",
    "3. Commands ca be automated (and this scales well)\n",
    "4. Commands should be readable\n",
    "5. Commands can be composed together"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Preface - utilities\n",
    "\n",
    "Plumbum provides us some simple utilities to quickly improve the experience of the user when using the application we are writing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### colored text output\n",
    "\n",
    "```python\n",
    "from plumbum import colors\n",
    "# this will not work in jupyter notebooks\n",
    "print(colors.green & colors.bold | \"This is green and bold.\")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### image visualization in the terminal\n",
    "\n",
    "```python\n",
    "from PIL import Image as imreader\n",
    "from plumbum.cli.image import Image as imdisplay\n",
    "im = imreader.open(\"fractal_wrongness.png\")\n",
    "# does not work with jupyter\n",
    "imdisplay().show_pil(im)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### progress bars"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 196,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                                                                                                                    \r"
     ]
    }
   ],
   "source": [
    "# progress bars\n",
    "from plumbum.cli.terminal import Progress\n",
    "import time\n",
    "\n",
    "for i in Progress.range(10):\n",
    "    time.sleep(0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### configurations files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting .myapp_rc\n"
     ]
    }
   ],
   "source": [
    "%%file .myapp_rc\n",
    "[DEFAULT]\n",
    "three=3\n",
    "\n",
    "[OTHER]\n",
    "a=3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3 2 3\n"
     ]
    }
   ],
   "source": [
    "from plumbum import cli\n",
    "\n",
    "with cli.ConfigINI('.myapp_rc') as conf:\n",
    "    one = conf['three']\n",
    "    two = conf.get('one', default='2')\n",
    "    three = conf.get('OTHER.a')\n",
    "    \n",
    "    # changing the configuration file\n",
    "    conf['OTHER.b'] = 4\n",
    "print(one, two, three)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[DEFAULT]\r\n",
      "three = 3\r\n",
      "one = 2\r\n",
      "\r\n",
      "[OTHER]\r\n",
      "a = 3\r\n",
      "b = 4\r\n",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!cat .myapp_rc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### reading user input"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "favorite color?\n",
      "(1) red\n",
      "(2) green\n",
      "(3) blue\n",
      "Choice [3]: "
     ]
    },
    {
     "data": {
      "text/plain": [
       "'blue'"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import plumbum.cli.terminal as terminal\n",
    "# see also ask and prompt\n",
    "terminal.choose(\"favorite color?\", ['red', 'green', 'blue'], default='blue')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### the local computer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Code dump.ipynb', 'Conda_Environment_instructions.txt', 'directory_structure.png', 'directory_structure.svg']\n"
     ]
    }
   ],
   "source": [
    "from plumbum import local\n",
    "ls = local[\"ls\"]\n",
    "print(ls().splitlines()[:4])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "the `local.cmd` dynamically loads programs from the environment.\n",
    "\n",
    "There is actually no such package, everything is created on the fly!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from plumbum.cmd import grep, cat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Programs can be combined in pipes in the same way as one could do in the bash shell.\n",
    "\n",
    "these pipes, in the same way as the individual programs, are not run until called"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(LocalCommand(/bin/ls), BoundCommand(LocalCommand(/bin/grep), ['.png']))"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ls|grep['.png']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "directory_structure.png\n",
      "file_inode_permissions.png\n",
      "fractal_wrongness.png\n",
      "OS_structure.png\n",
      "__temp_ipython__.png\n",
      "users_and_groups.png\n",
      "\n"
     ]
    }
   ],
   "source": [
    "pipe = ls|grep['.png']\n",
    "print(pipe())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### background and foreground execution\n",
    "```python\n",
    "from plumbum import FG, BG\n",
    "ls[\"-l\"] & FG\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### input-output redirection\n",
    "```python\n",
    "import sys\n",
    "((grep[\"world\"] < sys.stdin) > \"tmp.txt\")()\n",
    "cat(\"tmp.txt\")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'foo\\n'"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe = (cat << \"hello world\\nfoo\\nbar\\spam\" | grep[\"oo\"])\n",
    "pipe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 170,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/enrico/didattica/corso_programmazione_1819/programmingCourseDIFA/myapp\n"
     ]
    }
   ],
   "source": [
    "# temporarely change the working directory\n",
    "with local.cwd(local.cwd / \"myapp\"):\n",
    "    print(local.cwd)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### remote connection to other machines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from plumbum import SshMachine\n",
    "rem = SshMachine(\"hostname\", user = \"john\", keyfile = \"/path/to/idrsa\")\n",
    "rem.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# connect to a remote machine \n",
    "with SshMachine(\"hostname\", user = \"john\", keyfile = \"/path/to/idrsa\") as rem:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# temporarely change the working directory in the remote server\n",
    "with rem.cwd(rem.cwd / \"Desktop\"):\n",
    "    print(rem.cwd)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# port tunneling\n",
    "with rem.tunnel(6666, 8888):\n",
    "    pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# local grep of a remote ls\n",
    "from plumbum.cmd import grep\n",
    "r_ls = rem[\"ls\"]\n",
    "pipe = r_ls | grep[\"b\"]\n",
    "pipe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Command line programs\n",
    "\n",
    "In python any script can be at the same time a library and a command line program.\n",
    "\n",
    "All the code in the general section of the script is executed everytime the script is imported as a library, but one can set a part of the code to be run only when the script is called as a command line program.\n",
    "\n",
    "this is achieved by testing the `__name__` parameter of the program (a special variable that the interpreter set at the start), that when executed as a program is set to the string `\"__main__\"`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 183,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing my_first_app.py\n"
     ]
    }
   ],
   "source": [
    "%%file my_first_app.py\n",
    "\n",
    "def mysum(a, b):\n",
    "    return a+b\n",
    "\n",
    "print(\"this will be executed everytime\")\n",
    "\n",
    "if __name__=='__main__':\n",
    "    print(\"this will be executed only on the command line\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "this will be executed everytime\r\n"
     ]
    }
   ],
   "source": [
    "# this execute a single python command and quit\n",
    "!python3 -c \"import my_first_app\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "this will be executed everytime\r\n",
      "this will be executed only on the command line\r\n"
     ]
    }
   ],
   "source": [
    "!python3 ./my_first_app.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "This method could be used directly to implement very simple applications, by reading the arguments using `sys.argv`, and then parsing them by hand, but it's not very comfortable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 189,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting my_first_app.py\n"
     ]
    }
   ],
   "source": [
    "%%file my_first_app.py\n",
    "if __name__=='__main__':\n",
    "    import sys\n",
    "    print(sys.argv)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 190,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['./my_first_app.py', 'arg1', 'arg2']\r\n"
     ]
    }
   ],
   "source": [
    "!python3 ./my_first_app.py arg1 arg2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Python comes with a module that provide automatic arguments and options parsing, and works reasonably well, called `argparse`, that can be used in situations where external libraries can't be installed.\n",
    "\n",
    "the official tutorial can be found on the [python documentation](https://docs.python.org/dev/howto/argparse.html)\n",
    "\n",
    "It works, but it creates a code that is usually hard to read and manage when the options get more complicated.\n",
    "\n",
    "it is also missing all the support classes and functions that plumbum provides and that make easier to create high quality CLI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting my_first_app.py\n"
     ]
    }
   ],
   "source": [
    "%%file my_first_app.py\n",
    "if __name__=='__main__':\n",
    "    import argparse\n",
    "    parser = argparse.ArgumentParser()\n",
    "    parser.add_argument(\"echo\", help=\"echo the string you use here\")\n",
    "    \n",
    "    args = parser.parse_args()\n",
    "    \n",
    "    print(\"the received string is: {}\".format(args.echo))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 195,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "usage: my_first_app.py [-h] echo\r\n",
      "\r\n",
      "positional arguments:\r\n",
      "  echo        echo the string you use here\r\n",
      "\r\n",
      "optional arguments:\r\n",
      "  -h, --help  show this help message and exit\r\n"
     ]
    }
   ],
   "source": [
    "!python3 ./my_first_app.py --help"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 194,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the received string is: to_be_said\r\n"
     ]
    }
   ],
   "source": [
    "!python3 ./my_first_app.py to_be_said"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Command line classes\n",
    "\n",
    "command line applications can be imagined as structured in a similar way to a function:\n",
    "\n",
    "There are a certain number of ordinal arguments, and a certain number of named options.\n",
    "\n",
    "The arguments are the main focus of the program, while the option determines exactly how the execution is performed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The design approach of plumbum is that a program is a class.\n",
    "\n",
    "Options and flags connect to methods that configure the **self** attributes\n",
    "\n",
    "Parameters are given to a **main** function that is the method that performs the program execution.\n",
    "\n",
    "The **main** function have access to the results of the configuration as the configured **self** instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 176,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting my_app.py\n"
     ]
    }
   ],
   "source": [
    "%%file my_app.py\n",
    "from plumbum import cli\n",
    "\n",
    "class MyApp(cli.Application):\n",
    "    \"\"\"returns the square of a number\n",
    "    \"\"\"\n",
    "    PROGNAME = \"MyGloriousApp\"\n",
    "    VERSION = \"0.1\"\n",
    "    \n",
    "    def main(self, value: float):\n",
    "        #value = float(value)\n",
    "        print(\"result: {}\".format(value**2))\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    MyApp()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 177,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MyGloriousApp 0.1\r\n",
      "\r\n",
      "returns the square of a number\r\n",
      "\r\n",
      "Usage:\r\n",
      "    MyGloriousApp [SWITCHES] value\r\n",
      "\r\n",
      "Meta-switches:\r\n",
      "    -h, --help         Prints this help message and quits\r\n",
      "    --help-all         Prints help messages of all sub-commands and quits\r\n",
      "    -v, --version      Prints the program's version and quits\r\n",
      "\r\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!python3 my_app.py --help"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 180,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "result: 9.0\r\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "# type decoration is used to enforce arguments type and convert them.\n",
    "!python3 my_app.py 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 178,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Error: Argument of value expected to be <class 'float'>, not 'a':\r\n",
      "    ValueError(\"could not convert string to float: 'a'\",)\r\n",
      "------\r\n",
      "MyGloriousApp 0.1\r\n",
      "\r\n",
      "returns the square of a number\r\n",
      "\r\n",
      "Usage:\r\n",
      "    MyGloriousApp [SWITCHES] value\r\n",
      "\r\n",
      "Meta-switches:\r\n",
      "    -h, --help         Prints this help message and quits\r\n",
      "    --help-all         Prints help messages of all sub-commands and quits\r\n",
      "    -v, --version      Prints the program's version and quits\r\n",
      "\r\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "# type decoration is used to enforce arguments type and convert them.\n",
    "!python3 my_app.py a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting my_app.py\n"
     ]
    }
   ],
   "source": [
    "%%file my_app.py\n",
    "from plumbum import cli\n",
    "\n",
    "class MyApp(cli.Application):\n",
    "    PROGNAME = \"MyGloriousApp\"\n",
    "    VERSION = \"0.1\"\n",
    "    \n",
    "    verbose = cli.Flag([\"v\", \"verbose\"], help = \"If given, I will be very talkative\")\n",
    "\n",
    "    def main(self, filename):\n",
    "        print(\"I will now read {0}\".format(filename))\n",
    "        if self.verbose:\n",
    "            print(\"Yadda \" * 200)\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    MyApp()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Flags and switches can be:\n",
    "\n",
    "* compulsory\n",
    "* dependent on each other\n",
    "* mutually exclusive\n",
    "* repeatable\n",
    "* gouped together in the help description\n",
    "* bound to have specific values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MyGloriousApp 0.1\r\n",
      "\r\n",
      "Usage:\r\n",
      "    MyGloriousApp [SWITCHES] filename\r\n",
      "\r\n",
      "Meta-switches:\r\n",
      "    -h, --help         Prints this help message and quits\r\n",
      "    --help-all         Prints help messages of all sub-commands and quits\r\n",
      "    --version          Prints the program's version and quits\r\n",
      "\r\n",
      "Switches:\r\n",
      "    -v, --verbose      If given, I will be very talkative\r\n",
      "\r\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!python3 my_app.py --help"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I will now read foo\r\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!python3 my_app.py foo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I will now read foo\r\n",
      "Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda \r\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!python3 my_app.py foo --verbose"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I will now read foo\r\n",
      "Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda Yadda \r\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!python3 my_app.py foo -v"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting my_app.py\n"
     ]
    }
   ],
   "source": [
    "%%file my_app.py\n",
    "from plumbum import cli\n",
    "\n",
    "class MyApp(cli.Application):\n",
    "    PROGNAME = \"MyGloriousApp\"\n",
    "    VERSION = \"0.1\"\n",
    "    _log_file = None\n",
    "    \n",
    "    @cli.switch(\"--log-to-file\", str)\n",
    "    def log_to_file(self, filename):\n",
    "        \"\"\"Sets the file into which logs will be emitted\"\"\"\n",
    "        self._log_file = filename\n",
    "\n",
    "    def main(self, filename):\n",
    "        print(\"I will now read {0}\".format(filename))\n",
    "        if self._log_file is not None:\n",
    "            with open(self._log_file, 'w') as outfile:\n",
    "                print(\"I will now read {0}\".format(filename), file=outfile)\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    MyApp()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "!rm -f temp_log.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I will now read foo\r\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!python3 my_app.py foo --log-to-file temp_log.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I will now read foo\r\n"
     ]
    }
   ],
   "source": [
    "!cat temp_log.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### sub commands\n",
    "\n",
    "sub-commands are a way to group together several related program under a single umbrella.\n",
    "\n",
    "a classical example is **git**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting my_app.py\n"
     ]
    }
   ],
   "source": [
    "%%file my_app.py\n",
    "from plumbum import cli\n",
    "\n",
    "class MyApp(cli.Application):\n",
    "    \"\"\"returns the square of a number\n",
    "    \"\"\"\n",
    "    PROGNAME = \"MyGloriousApp\"\n",
    "    VERSION = \"0.1\"\n",
    "    \n",
    "    def main(self):\n",
    "        if not self.nested_command:           # will be ``None`` if no sub-command follows\n",
    "            print(\"No command given\")\n",
    "            return 1   # error exit code\n",
    "\n",
    "@MyApp.subcommand(\"pow\")\n",
    "class MyAppPow(cli.Application):\n",
    "    \"calculate the square power instead\"\n",
    "    def main(self, value, power=2):\n",
    "        value = float(value)\n",
    "        power = float(power)\n",
    "        print(\"result: {}\".format(value**power))    \n",
    "    \n",
    "@MyApp.subcommand(\"sqrt\")\n",
    "class MyAppSqrt(cli.Application):\n",
    "    \"calculate the square root instead\"\n",
    "    def main(self, value):\n",
    "        value = float(value)\n",
    "        print(\"result: {}\".format(value**0.5))\n",
    "        \n",
    "if __name__ == \"__main__\":\n",
    "    MyApp()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MyGloriousApp 0.1\r\n",
      "\r\n",
      "returns the square of a number\r\n",
      "\r\n",
      "Usage:\r\n",
      "    MyGloriousApp [SWITCHES] [SUBCOMMAND [SWITCHES]] \r\n",
      "\r\n",
      "Meta-switches:\r\n",
      "    -h, --help         Prints this help message and quits\r\n",
      "    --help-all         Prints help messages of all sub-commands and quits\r\n",
      "    -v, --version      Prints the program's version and quits\r\n",
      "\r\n",
      "Sub-commands:\r\n",
      "    pow                calculate the square power instead; see 'MyGloriousApp\r\n",
      "                       pow --help' for more info\r\n",
      "    sqrt               calculate the square root instead; see 'MyGloriousApp\r\n",
      "                       sqrt --help' for more info\r\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!python3 my_app.py --help"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MyGloriousApp sqrt 0.1\r\n",
      "\r\n",
      "calculate the square root instead\r\n",
      "\r\n",
      "Usage:\r\n",
      "    MyGloriousApp sqrt [SWITCHES] value\r\n",
      "\r\n",
      "Meta-switches:\r\n",
      "    -h, --help         Prints this help message and quits\r\n",
      "    --help-all         Prints help messages of all sub-commands and quits\r\n",
      "    -v, --version      Prints the program's version and quits\r\n",
      "\r\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!python3 my_app.py sqrt --help"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MyGloriousApp pow 0.1\r\n",
      "\r\n",
      "calculate the square power instead\r\n",
      "\r\n",
      "Usage:\r\n",
      "    MyGloriousApp pow [SWITCHES] value [power=2]\r\n",
      "\r\n",
      "Meta-switches:\r\n",
      "    -h, --help         Prints this help message and quits\r\n",
      "    --help-all         Prints help messages of all sub-commands and quits\r\n",
      "    -v, --version      Prints the program's version and quits\r\n",
      "\r\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!python3 my_app.py pow --help"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "result: 3.0\r\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!python3 my_app.py sqrt 9"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "result: 9.0\r\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!python3 my_app.py pow 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "result: 1.7320508075688772\r\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!python3 my_app.py pow 3 0.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## system wide installation\n",
    "\n",
    "to perform a system wide installation we will use the `setuptools` package, that allows us to interface our program with the pip installation mechanism."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### preparing the field\n",
    "\n",
    "The first step is to create a proper folder structure for our program.\n",
    "In this case I'm creating a CLI program called **myapp**.\n",
    "\n",
    "* **myapp** (project folder)\n",
    "    * `setupy.py`\n",
    "    * **myapp** (program folder)\n",
    "        * `__init__.py`\n",
    "        * `__main__.py`\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "the outside folder is the project folder, that will contain everything concerning my app: the app itself, its documentation, the installation procedure, and so on.\n",
    "\n",
    "Then there is an inside folder, usually with the same name of the app, that will contain the proper files to be executed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Inside this folder there should be a file `__init__.py`.\n",
    "This file can be empty, but is used by python to recognize that this directory is a module that can be imported or executed.\n",
    "\n",
    "My application is going to be in the `__main__.py` file. No special reason for the name, just more consistant"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "!touch ./myapp/myapp/__init__.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting ./myapp/myapp/__main__.py\n"
     ]
    }
   ],
   "source": [
    "%%file ./myapp/myapp/__main__.py\n",
    "from plumbum import cli\n",
    "\n",
    "class MyApp(cli.Application):\n",
    "    \"\"\"returns the square of a number\n",
    "    \"\"\"\n",
    "    PROGNAME = \"MyGloriousApp\"\n",
    "    VERSION = \"0.1\"\n",
    "    \n",
    "    def main(self):\n",
    "        if not self.nested_command:           # will be ``None`` if no sub-command follows\n",
    "            print(\"No command given\")\n",
    "            return 1   # error exit code\n",
    "\n",
    "@MyApp.subcommand(\"pow\")\n",
    "class MyAppPow(cli.Application):\n",
    "    \"calculate the square power instead\"\n",
    "    def main(self, value, power=2):\n",
    "        value = float(value)\n",
    "        power = float(power)\n",
    "        print(\"result: {}\".format(value**power))    \n",
    "    \n",
    "@MyApp.subcommand(\"sqrt\")\n",
    "class MyAppSqrt(cli.Application):\n",
    "    \"calculate the square root instead\"\n",
    "    def main(self, value):\n",
    "        value = float(value)\n",
    "        print(\"result: {}\".format(value**0.5))\n",
    "        \n",
    "if __name__ == \"__main__\":\n",
    "    MyApp.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The file necessary for setuptools to work properly is the `setup.py`, where we will use the `setup` function of the `setuptools` module.\n",
    "\n",
    "This function configure many things for us, but the most important thing is the `entry_point` parameter.\n",
    "This creates the link to the scripts that we will actually execute.\n",
    "Inside this parameter we specify that two entry points are `console_scripts` that should execute the script and the function provided."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting ./myapp/setup.py\n"
     ]
    }
   ],
   "source": [
    "%%file ./myapp/setup.py\n",
    "from setuptools import setup\n",
    "\n",
    "setup(\n",
    "    name = 'myapp',\n",
    "    version = '0.1.0',\n",
    "    packages = ['myapp'],\n",
    "    install_requires=[ 'plumbum', ],\n",
    "    entry_points = {\n",
    "        'console_scripts': [\n",
    "            'myapp = myapp.__main__:MyApp',\n",
    "            'myappsqrt = myapp.__main__:MyAppSqrt',\n",
    "        ]\n",
    "    })"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In this case we decided to let the user use the standard program (choosing the sub command to use) or to provide a shortcut to the subcommand itself.\n",
    "\n",
    "In this case the plumbum structure comes very useful: we can call directly the subcommand class to execute our subcommand.\n",
    "\n",
    "the name we put before the `=` sign is going to be the command name that we will use to call the program from the command line"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Last special mention goes to the `install_requires`: this list allows to specify all the dependencies of the package.\n",
    "If they are not installed, they will automatically downloaded and installed by pip.\n",
    "In our case the only dependency that we have is for the `plumbum` package"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### the installation step\n",
    "\n",
    "from the folder containing the project folder, we can call the pip installation program\n",
    "\n",
    "by installing it with the `--editable` (or `-e` for short) the program will not be copied in the system, but rather linked in it.\n",
    "This means that if you edit the program, the modifications will appear live in the systems without need to reinstall"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Obtaining file:///home/enrico/didattica/corso_programmazione_1819/programmingCourseDIFA/myapp\n",
      "Requirement already satisfied: plumbum in /home/enrico/miniconda3/lib/python3.6/site-packages (from myapp==0.1.0) (1.6.7)\n",
      "Installing collected packages: myapp\n",
      "  Running setup.py develop for myapp\n",
      "Successfully installed myapp\n"
     ]
    }
   ],
   "source": [
    "!pip3 install --editable myapp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MyGloriousApp 0.1\r\n",
      "\r\n",
      "returns the square of a number\r\n",
      "\r\n",
      "Usage:\r\n",
      "    MyGloriousApp [SWITCHES] [SUBCOMMAND [SWITCHES]] \r\n",
      "\r\n",
      "Meta-switches:\r\n",
      "    -h, --help         Prints this help message and quits\r\n",
      "    --help-all         Prints help messages of all sub-commands and quits\r\n",
      "    -v, --version      Prints the program's version and quits\r\n",
      "\r\n",
      "Sub-commands:\r\n",
      "    pow                calculate the square power instead; see 'MyGloriousApp\r\n",
      "                       pow --help' for more info\r\n",
      "    sqrt               calculate the square root instead; see 'MyGloriousApp\r\n",
      "                       sqrt --help' for more info\r\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!myapp --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can call the base program or the specific subcommand, as we specified"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "result: 3.0\r\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!myapp sqrt 9 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "result: 3.0\r\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!myappsqrt 9"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Lastly, we can uninstall the program from any location"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Uninstalling myapp-0.1.0:\n",
      "  Successfully uninstalled myapp-0.1.0\n"
     ]
    }
   ],
   "source": [
    "!pip3 uninstall -y myapp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "If one needs to shop non-code files (such as images, configuration files, etc...)\n",
    "check:\n",
    "\n",
    "https://python-packaging.readthedocs.io/en/latest/non-code-files.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "given that now **myapp** is a system program, I can load it using `plumbum` as any other application!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "result: 3.0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "myapp_hook = local['myapp']\n",
    "sqrt_hook = myapp_hook['sqrt']\n",
    "print(sqrt_hook(9))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## CLI application good practices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Errors management\n",
    "\n",
    "\n",
    "#### clean up after yourself, especially on errors\n",
    "If something goes wrong, try to avoid leaving the system in an invalid state, and clean unnecessary files\n",
    "\n",
    "#### allow if possible to recover partial results\n",
    "This is especially important in long executions: take a hint from the snakemake approach, and try to have partial results stored as temporary files to avoid unnecessary repetition when possible"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### try to be robust to bad data and configuration\n",
    "it is better to do less things, but making sure that those things are well covered and the program will not trow a fit for some input data in an awkward format.\n",
    "\n",
    "Utilities should handle arbitrarily long lines, huge files, and so on.\n",
    "It is perhaps tolerable for a utility to fail on a data set larger than it can hold in memory, but some utilities don't do this.\n",
    "for instance, sort, by using temporary files, can generally sort data sets much larger than it can hold in memory. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Exit with different error codes\n",
    "this is the values that is returned by the **main**\n",
    "0 means success, any other number means failure\n",
    "the best solution is to code each error to a specific return value (and document it)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### User interaction\n",
    "\n",
    "\n",
    "#### give feedback for task longer than a second or two (such as progress bars)\n",
    "Users might worry that something has gone wrong and just kill the process if they have to wait without any prompt that the program is doing something.\n",
    "Even better, try to use a progress bar with a time estimate (plumbum has you covered)\n",
    "\n",
    "#### let the user control the level of verbosity of the output\n",
    "include a `--verbose`, a `--terse` and a `--silent` option to allow the user to control the level of feedback they wants"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Colour code your output\n",
    "\n",
    "Ideally, your script should:\n",
    "* output white/default (it’s the foreground process), \n",
    "* child processes should output grey (usually not needed unless things screw up), \n",
    "* success should be denoted with green, \n",
    "* failure, red, and \n",
    "* warnings in yellow."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Have both short ie one letter (Eg `-h`) and long forms (eg `--help`) of the command's switches\n",
    "long format helps readability and long term maintanability, and are good for scripting and manuals\n",
    "\n",
    "short forms are useful for manual input, especially of commonly used options (don't waste the user time)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Ask for confirmation for irreversible action\n",
    "such as deleting data, changing informations without backup, etc...\n",
    "\n",
    "there should be an option to make this automatic, but the user should be protected from involuntary changes to the system.\n",
    "\n",
    "Even better, while asking for confirmation, also suggest which is the option to turn it off"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Documentation\n",
    "\n",
    "#### consider adding a `help` sub command\n",
    "alongside the `--help` flag, consider including an help command that contains information about various topics\n",
    "\n",
    "#### provide some use cases in the documentation\n",
    "Especially if the case uses are not trivial, with many interlocked features interacting with each other.\n",
    "\n",
    "And, is possible, put them nearby the start of the documentation, rather than at the end."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### In case of mistakes try suggesting common correction techniques\n",
    "Git again does this perfectly: when something goes wrong, it also suggest reliable methods to correct the bad situation, without covering this information in \"output noise\"\n",
    "\n",
    "#### make common commands memorable\n",
    "Don't use weird names, inside jokes, abstract references, words referring to implementation details (unless relevant).\n",
    "\n",
    "keep it simple and easy to remember."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Configuration\n",
    "\n",
    "\n",
    "#### Work independently of current working directory\n",
    "use only absolute paths (/path/to/something) and paths relative to the script (demonstrated below)\n",
    "\n",
    "#### consider using configuration files\n",
    "This allows a greater deal of automation and configurability, reducing the load on the user to write all the options by hand.\n",
    "\n",
    "on the other hand, a configuration file should never be a necessity, and everything should be used as an option"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### all the options that can have a default value, should have one\n",
    "Ideally, the program should be executable without specifying any option.\n",
    "\n",
    "compulsory options are probably candidates for sub-commands"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Interface\n",
    "\n",
    "#### try to be consistent with other GNU tools : \n",
    "For example see a list of options from other GNU tools and if possible try to align to them\n",
    "\n",
    "http://www.catb.org/esr/writings/taoup/html/ch10s05.html\n",
    "\n",
    "https://www.gnu.org/prep/standards/standards.html\n",
    "\n",
    "#### play well with pipes and allow input and output to files\n",
    "To play well with pipes it should read and write to stdout by default, but the user should be free to redirect to file if necessary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### If a command has a side effect provide a dry-run/whatif/no_post option.\n",
    "don't let the user jump blindly without checking what would happen.\n",
    "\n",
    "This also allows the user to find possible mistakes faster and with less risks\n",
    "\n",
    "Snakemake perfors this very well."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### everything should be doable in a script, without human interaction\n",
    "To allow the program to be fully used in script and automated, it should always be possible to execute it without any input from the user.\n",
    "\n",
    "When possible, this should be the preferred behavior\n",
    "\n",
    "#### try to make it composable\n",
    "programs that play nicely with other can be used to implement even more complicated pipelines and programs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### conside adding switches to both turn on and off behaviors\n",
    "If a switch is in the long-form (for example --foo), which turns “on” some behavior (that is off by default), there should also be another switch that turns “off” the behavior.\n",
    "\n",
    "This allow the user to be more explicit and change-proof the script for future change of default behavior"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### consider using a long informative name for you app\n",
    "\n",
    "the GNU convention is to use 2 or 3 letters names, but if one can avoid it, the deafult program name should be something either explicit or memorable.\n",
    "\n",
    "The user has always the option to alias it to something shorter."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### if no output is necessary, avoid output unless required\n",
    "that's what the `--verbose` option is for.\n",
    "\n",
    "if the program performs an action and doesn't have to return a result in case of success (such as moving a file, changing directory, etc...), the user should see an output only in case of errors or if the verbose option is selected.\n",
    "\n",
    "This makes the program more script friendly"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Exercise\n",
    "\n",
    "Write a personal knowledge database app.\n",
    "\n",
    "this app should:\n",
    "* add new notes\n",
    "* show notes\n",
    "* search notes based on words contained in them\n",
    "\n",
    "## advanced\n",
    "\n",
    "* set tags, also in previously created notes\n",
    "* filter by tags (including logic of tags to use or not)\n",
    "* filter by date"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "https://codeburst.io/13-tips-tricks-for-writing-shell-scripts-with-awesome-ux-19a525ae05ae\n",
    "\n",
    "https://eng.localytics.com/exploring-cli-best-practices/\n",
    "\n",
    "https://amir.rachum.com/blog/2017/07/28/python-entry-points/"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}