programming basics/Unix.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a9e782f1",
   "metadata": {},
   "source": [
    "# Introduction to the Unix Shell\n",
    "This example is sourced from [software carpentry](https://swcarpentry.github.io/shell-novice/) and is here to provide a simple set of instructions explaining how to interface with a Unix enabled command line. </br>\n",
    "See a [cheat sheet](./Unix_Commands.txt) for quick instructions on different commands."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ca2e987",
   "metadata": {},
   "source": [
    "### Background: Novice\n",
    "Humans and computers commonly interact in many different ways, such as through a keyboard and mouse, touch screen interfaces, or using speech recognition systems. The most widely used way to interact with personal computers is called a **graphical user interface** (GUI). With a GUI, we give instructions by clicking a mouse and using menu-driven interactions.\n",
    "\n",
    "While the visual aid of a GUI makes it intuitive to learn, this way of delivering instructions to a computer scales very poorly. Imagine the following task: for a literature search, you have to copy the third line of one thousand text files in one thousand different directories and paste it into a single file. Using a GUI, you would not only be clicking at your desk for several hours, but you could potentially also commit an error in the process of completing this repetitive task. This is where we take advantage of the Unix shell. The Unix shell is both a **command-line interface** (CLI) and a scripting language, allowing such repetitive tasks to be done automatically and fast. With the proper commands, the shell can repeat tasks with or without some modification as many times as we want. Using the shell, the task in the literature example can be accomplished in seconds."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09495e10",
   "metadata": {},
   "source": [
    "### The Shell: Novice\n",
    "The shell is a program where users can type commands. With the shell, it’s possible to invoke complicated programs like climate modeling software or simple commands that create an empty directory with only one line of code. The most popular Unix shell is **Bash** (the Bourne Again SHell — so-called because it’s derived from a shell written by Stephen Bourne). Bash is the default shell on most modern implementations of Unix and in most packages that provide Unix-like tools for Windows.\n",
    "\n",
    "Using the shell will take some effort and some time to learn. While a GUI presents you with choices to select, CLI choices are not automatically presented to you, so you must learn a few commands like new vocabulary in a language you’re studying. However, unlike a spoken language, a small number of “words” (i.e. commands) gets you a long way, and we’ll cover those essential few today.\n",
    "\n",
    "The grammar of a shell allows you to combine existing tools into powerful pipelines and handle large volumes of data automatically. Sequences of commands can be written into a script, improving the reproducibility of workflows.\n",
    "\n",
    "In addition, the command line is often the easiest way to interact with remote machines and supercomputers. Familiarity with the shell is near essential to run a variety of specialized tools and resources including high-performance computing systems. As clusters and cloud computing systems become more popular for scientific data crunching, being able to interact with the shell is becoming a necessary skill. We can build on the command-line skills covered here to tackle a wide range of scientific questions and computational challenges.\n",
    "\n",
    "Let’s get started.\n",
    "\n",
    "When the shell is first opened, you are presented with a **prompt**, indicating that the shell is waiting for input."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8c729ab",
   "metadata": {},
   "source": [
    "### Using Comands: Novice\n",
    "The shell typically uses **\\\\$** as the prompt, but may use a different symbol. In the examples for this lesson, we’ll show the prompt as **\\\\$**. Most importantly: when typing commands, either from these lessons or from other sources, do not type the prompt, only the commands that follow it. Also note that after you type a command, you have to press the Enter key to execute it.\n",
    "\n",
    "The prompt is followed by a text cursor, a character that indicates the position where your typing will appear. The cursor is usually a flashing or solid block, but it can also be an underscore or a pipe. You may have seen it in a text editor program, for example.\n",
    "\n",
    "So let’s try our first command, ls which is short for listing. This command will list the contents of the current directory:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "feb85a3f",
   "metadata": {},
   "source": [
    "```bash\n",
    "$ ls\n",
    "```\n",
    "Output\n",
    "```bash\n",
    "Desktop     Downloads   Movies      Pictures\n",
    "Documents   Library     Music       Public\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b57d4746",
   "metadata": {},
   "source": [
    "### Navigating Directories: Novice\n",
    "Commands Covered:\n",
    "* pwd\n",
    "* ls\n",
    "* relative and absolute paths / ~ . ..\n",
    "* cd </br> </br>\n",
    "The part of the operating system responsible for managing files and directories is called the **file system**. It organizes our data into files, which hold information, and directories (also called ‘folders’), which hold files or other directories.\n",
    "\n",
    "Several commands are frequently used to create, inspect, rename, and delete files and directories. To start exploring them, we’ll go to our open shell window.\n",
    "\n",
    "First, let’s find out where we are by running a command called pwd (which stands for ‘print working directory’). Directories are like places — at any time while we are using the shell, we are in exactly one place called our **current working directory**. Commands mostly read and write files in the current working directory, i.e. ‘here’, so knowing where you are before running a command is important. pwd shows you where you are:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "623327fd",
   "metadata": {},
   "source": [
    "```bash\n",
    "$ pwd\n",
    "```\n",
    "Output\n",
    "```bash\n",
    "/Users/calcraven\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a15087d2",
   "metadata": {},
   "source": [
    "#### Home Directory Variation\n",
    "The home directory path will look different on different operating systems. On Linux, it may look like /home/nelle, and on Windows, it will be similar to C:\\Documents and Settings\\nelle or C:\\Users\\nelle. (Note that it may look slightly different for different versions of Windows.) In future examples, we’ve used Mac output as the default - Linux and Windows output may differ slightly but should be generally similar.\n",
    "\n",
    "We will also assume that your pwd command returns your user’s home directory. If pwd returns something different, you may need to navigate there using cd or some commands in this lesson will not work as written."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f59d0bed",
   "metadata": {},
   "source": [
    "To understand what a ‘home directory’ is, let’s have a look at how the file system as a whole is organized. For the sake of this example, we’ll be illustrating the filesystem on our scientist Nelle’s computer. After this illustration, you’ll be learning commands to explore your own filesystem, which will be constructed in a similar way, but not be exactly identical.\n",
    "\n",
    "A filesystem might look like this:\n",
    "\n",
    "<img src=\"images/filesystem.png\" width=200 height=400 />\n",
    "\n",
    "At the top is the root directory that holds everything else. We refer to it using a slash character, `/`, on its own; this character is the leading slash in /Users/calcraven.\n",
    "\n",
    "Inside that directory are several other directories: `bin` (which is where some built-in programs are stored), `data` (for miscellaneous data files), Users (where users’ personal directories are located), `tmp` (for temporary files that don’t need to be stored long-term), and so on.\n",
    "\n",
    "We know that our current working directory `/Users/calcraven` is stored inside `/Users` because `/Users` is the first part of its name. Similarly, we know that `/Users` is stored inside the root directory `/` because its name begins with `/`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7618c45",
   "metadata": {},
   "source": [
    "#### Use \"ls\" and tab-completion to look down file paths\n",
    "The command `ls` is the first command we used. Let's pass some path information to this command to have it list files somewhere else in our filesystem.\n",
    "```bash\n",
    "$ ls .\n",
    "```\n",
    "Output\n",
    "```bash\n",
    "Desktop     Downloads   Movies      Pictures\n",
    "Documents   Library     Music       Public\n",
    "```\n",
    "The `.` argument passed to this command indicates to look exactly where we currently are. That is why the results are the same as the pure `ls` command. However, you can have other symbols that will direct functions along these paths. They are broken up into `absolute path` directories and `relative path` directories. \n",
    "\n",
    "Absolute paths will take you to the same path no matter where you are in your filesystem. These are: </br> </br>\n",
    "`~`: Your home directory. This is typically the directory that you start at when opening a new terminal. </br>\n",
    "`/`: Your root directory. This is the very top level directory in you filesystem hierarch. </br>\n",
    "Try running these commands to see the output.\n",
    "```bash\n",
    "$ ls ~\n",
    "```\n",
    "```bash\n",
    "$ ls /\n",
    "```\n",
    "\n",
    "Relative paths will take you somewhere relative to your current position in the filesystem These are: </br> </br>\n",
    "`.`: Your current directory. </br>\n",
    "`..`: The parent directly. This is akin to looking one step backwards or upwards in your filesystem. </br>\n",
    "Try running these commands to see the output.\n",
    "```bash\n",
    "$ ls ..\n",
    "```\n",
    "```bash\n",
    "$ ls ./Documents\n",
    "```\n",
    "\n",
    "Because the directly `Documents` is in your current path, this will go from your current directory down to the `Documents` directory, and then run `ls`. This will tell you what files might be found in this directory. While typing out the path in the `ls` command, you can click `tab` to autocomplete as you start typing out a directory name. For example, if you type out: \n",
    "```bash\n",
    "$ ls ./Docume\n",
    "```\n",
    "and then hit tab, the command line should autcomplete since `Documents` is the only remaining match in the current working directory to `Docume`. \n",
    "```bash\n",
    "$ ls ./Documents\n",
    "```\n",
    "If you double click tab, the command line may show all remaining options if multiple options exist. For example:\n",
    "```bash\n",
    "$ ls ./Do\n",
    "```\n",
    "Output after double clicking tab\n",
    "```bash\n",
    "Documents Downloads\n",
    "```\n",
    "**Note:** that double clicking tab this may not be configured and may depend on the CLI you are using\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b1ac71e",
   "metadata": {},
   "source": [
    "#### Using the \"cd\" command to move directories\n",
    "Up to this point, we have stayed in the `~` directory. In order to actually move from one directory to another, we can use the `cd` command, which stands for **change directory**. Complete the following task:\n",
    "1. Move to the home directory\n",
    "2. See the directories in the home directory\n",
    "3. Move into one of those directories labeled `Documents`\n",
    "4. List out the files and directories in the current directory\n",
    "5. Move backwards one directory\n",
    "6. List out current directories and compare to step 2.\n",
    "7. Move into `Documents` again\n",
    "8. List out the files and directories in the current directory, and compare to step 4.\n",
    "9. Move back to `Home`\n",
    "10. List out current directories\n",
    "\n",
    "This can be achieved by following these commands:\n",
    "```bash\n",
    "$ cd ~\n",
    "$ ls\n",
    "$ cd Documents\n",
    "$ ls .\n",
    "$ cd ..\n",
    "$ ls .\n",
    "$ cd ./Documents\n",
    "$ ls .\n",
    "$ cd ~\n",
    "$ ls .\n",
    "```\n",
    "As you can see here, we just moved back and forth from the `Home` directory and the `Documents` directory repeatedly. In order to further your understanding, try to move into one of the directories within `Documents`, and see what is found there. </br> </br>\n",
    "\n",
    "To learn more about moving around the filesystem, see this [software carpentry tutorial](https://swcarpentry.github.io/shell-novice/02-filedir/index.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e13ad876",
   "metadata": {},
   "source": [
    "### Working with Files and Directories: Beginner\n",
    "Commands Covered:\n",
    "* mkdir\n",
    "* touch\n",
    "* mv\n",
    "* cp\n",
    "* rm </br> </br>\n",
    "Now that we know how to move around what is currently there, how do we start to make our own directories and files? The `mkdir` function allows you to **make a directory**. You can pass it the name of your newly created directory. Remember that it will create the directory wherever you are currently located. Use the `pwd` command to see the path to where you currently are.\n",
    "```bash\n",
    "$ mkdir MY_NEW_DIR\n",
    "$ cd MY_NEW_DIR\n",
    "$ ls .\n",
    "$ pwd\n",
    "```\n",
    "\n",
    "This set of codes should make the new directory titled \"MY_NEW_DIR\", and move you inside. You should see that the Output from the `ls` command is nothing. </br> </br>\n",
    "\n",
    "#### Create a file and move it around\n",
    "\n",
    "Now lets create a file inside our newly created directory, \"MY_NEW_DIRECTORY\". This can be done using the `touch` command, which will just create an empty file with a specified name, similar to the `mkdir` command. Let's give the newly created file the `.txt` **extension** to remind ourselfs that this is a file with text inside. The letters following the `.` don't actually affect the contents, they just tell you what sort of information is inside $i.e.$ a `.docx` file is a Microsoft Word File, and a `.py` is a file with Python code.\n",
    "```bash\n",
    "$ touch file.txt\n",
    "$ ls .\n",
    "```\n",
    "Now you should see that the file `file.txt` is present in the current directory! If we decide that it needs to be located elsewhere in your hierarchy, we can **move** it using hte `mv` command!\n",
    "```bash\n",
    "$ mv file.txt ..\n",
    "$ ls .\n",
    "$ ls ..\n",
    "```\n",
    "If you look at the results of the two `ls` commands, you'll piece together that the file is now located one directory above \"MY_NEW_DIR\". Let's try to move it back.\n",
    "```bash\n",
    "$ ls .\n",
    "$ mv ../file.txt ./file.txt\n",
    "$ ls .\n",
    "$ ls ..\n",
    "$ mv ./file.txt ./file2.txt\n",
    "$ ls .\n",
    "```\n",
    "Wait a second... the result of the last `ls` shows that we have a new file called `file2.txt` and we don't have `file.txt` anymore. What happened? Well the `mv` command can move the location of the file as well as the name of the file. You can almost think of the name of the file as the location on your hard drive that the file is stored! In that sense, the `mv` command acts as a **rename** command, since you are moving it from one name to another! </br> </br>\n",
    "\n",
    "#### Copy and remove a file\n",
    "If you have a file, you can create a **copy** using the `cp` command. This is nice if you want to test something to make edits and want a backup copy that hasn't been changed. Always work with copies in case you accidentally misuse a command!\n",
    "```bash\n",
    "$ cp file2.txt file3.txt\n",
    "$ ls .\n",
    "```\n",
    "You should now see two files, one named `file2.txt` and the other `file3.txt`. If you want to **delete** one of these files, use the `rm` command, which stands for **remove**.\n",
    "```bash \n",
    "$ rm file2.txt\n",
    "$ ls .\n",
    "```\n",
    "Now you should see that file2.txt doesn't exist anymore, and file3.txt is the only one left.\n",
    "**Note**: Please be careful with the `rm` command. It can seem obvious, but accidentally deleting the wrong files can be a major headache. Use the command with the interactive deletion flag, `rm -i file.txt` in order to get a second confirmation about the files you want to delete."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "271a7a50",
   "metadata": {},
   "source": [
    "### Text editing: Beginner\n",
    "Commands Covered:\n",
    "* cat\n",
    "* head\n",
    "* vim/nano </br> </br>\n",
    "Now that we've been able to do some things will files, how to we figure out what info is stored in them? The `cat` tool and the `head` tool are two useful tools for this. They stand for **concatenate** and **header** respectively. The `cat` command will print out the text in the entire file, whereas the `head` command will print just the first 10 lines. This is useful if the file is too large and printing will take awhile.\n",
    "```bash\n",
    "$ cat file3.txt\n",
    "$ head file3.txt\n",
    "```\n",
    "Huh, this doesn't return anything. Well that's because the original file we made with the `touch` command didn't put any information into the file. In order to add text to the file, we need a text editor. [Vim](https://www.vim.org/), [Emacs](https://www.gnu.org/software/emacs/), and [Nano](https://www.nano-editor.org/) are text editors that usually come compiled with any Unix CLI. There are many other options and are usually up to personal preference. For this example, we will use Nano.\n",
    "```bash \n",
    "$ nano my_script.sh\n",
    "```\n",
    "And type these lines into the window that pops up.\n",
    "```\n",
    "MY_PATH = .\n",
    "cd ~\n",
    "pwd\n",
    "ls .\n",
    "cd $MY_PATH\n",
    "```\n",
    "To save and close, use `Ctrl-O` and `Ctrl-X`. You should now be back to the CLI. To validate that your changes were saved run the following lines of code:\n",
    "```bash\n",
    "$ ls .\n",
    "$ cat my_script.sh\n",
    "$ head my_script.sh\n",
    "```\n",
    "You should see that the file you made with Nano exists in the current path, and the results from what you actually put into that file. We'll use that file in the next section of this lesson. </br> </br>\n",
    "\n",
    "For more information on text editing. Try out [Vimtutor](https://linux.die.net/man/1/vimtutor). This will walk you through some first time commands to get you more comfortable using `vim`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "264e0841",
   "metadata": {},
   "source": [
    "### Scripts: Beginner\n",
    "Commands Covered:\n",
    "* bash </br> </br>\n",
    "Writing scripts is a good way to store commands for repeated use or to remind yourself in the future what commands need to be run for a project. A shell script typically is notified with a **shell** `.sh` extension. Not only will writing shell scripts make your work faster — you won’t have to retype the same commands over and over again — it will also make it more accurate (fewer chances for typos) and more reproducible. If you come back to your work later (or if someone else finds your work and wants to build on it) you will be able to reproduce the same results simply by running your script, rather than having to remember or retype a long list of commands. A shell script can have any number of UNIX commands inside of it, listed in order of which they should be executed. An simple example may look like:\n",
    "\n",
    "my_script.sh\n",
    "```\n",
    "MY_PATH = .\n",
    "cd ~\n",
    "pwd\n",
    "ls .\n",
    "cd $MY_PATH\n",
    "```\n",
    "\n",
    "This script will print the path of the home directory and its contents, and then move back to the current directory. In order to execute this script, use the `bash` command.\n",
    "```bash\n",
    "$ bash my_script.sh\n",
    "```\n",
    "\n",
    "Scripting starts to show the real power of the Unix CLI and as a result can get very complicated very quickly. See this [software carpentry tutorial](https://swcarpentry.github.io/shell-novice/06-script/index.html) for more examples of complex scripting."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fda95294",
   "metadata": {},
   "source": [
    "### Pipes and Filters: Intermediate"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b79ddc83",
   "metadata": {},
   "source": [
    "Commands can be chained together to add functionality, or the results of a command can be pushed to something like an log file to have records of the output. **Pipes** `|`, `<`, `>` and **filters** `sort` can be used to coordinate multiple commands together. See this [software carpentry tutorial](https://swcarpentry.github.io/shell-novice/04-pipefilter/index.html) for more information."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5785dd31",
   "metadata": {},
   "source": [
    "### Loops: Intermediate"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99a53185",
   "metadata": {},
   "source": [
    "Looping is a common tactic used to repeat a command a prescribed number of times. The `for` or `while` commands allow you to do this. See [this software carpentry tutorial](https://swcarpentry.github.io/shell-novice/05-loop/index.html) for more information."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72b0cc43",
   "metadata": {},
   "source": [
    "### Finding Things: Intermediate"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a6f2f43",
   "metadata": {},
   "source": [
    "Sometimes, you want to look for a specific file, or a line of text in various files. You can use the `grep` command or the `find` command in combination for this functionality. This [software carpentry tutorial](https://swcarpentry.github.io/shell-novice/07-find/index.html) will teach you more about using these commands"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9be0c895",
   "metadata": {},
   "source": [
    "### Exercise 1: Advanced\n",
    "Write a shell script that will create a function to print out the number of files in your current working directory. Improve your function to use wildcards to count only files with the extension \".txt\". Create a dummy directory with multiple \".txt\" files in it and test your function on that directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a3194af2",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}