Skip to content

Commit

Permalink
Added some bash scripting stuff
Browse files Browse the repository at this point in the history
  • Loading branch information
@JamesFergusson
JamesFergusson committed Oct 16, 2019
1 parent 25c1040 commit ed3e570
Show file tree
Hide file tree
Showing 2 changed files with 238 additions and 17 deletions.
215 changes: 198 additions & 17 deletions 2_LinuxTerminal.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -217,8 +217,23 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"This allows tab completion so `cd m<\\tab>` would match all directories with `m*` and complete as much as is unique which is pretty helpful.\n",
"\n",
"This allows tab completion so `cd m<\\tab>` would match all directories with `m*` and complete as much as is unique which is pretty helpful. When changing directory to something like \"My Documents\" we need to treat the space as literal otherwise it thinks you've asked to change into two directories which doesn't make sense. For this we can either use:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"cd \"My Documents\"\n",
"cd My\\ Documents"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next we might want to create or destroy files. `touch` will create a blank file and `rm` (<b>r</b>e<b>m</b>ove) to delete it (`touch` also lets you edit the file's time-stamp which is helpful if you need to prove you were busy coding when the diamonds went missing). For directories use `mkdir` and `rmdir`"
]
},
Expand Down Expand Up @@ -248,7 +263,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"These all accept wild cards so `rm *.out` removes all files ending in `.out`. `rm` will also remove empty directories and with `-r` option (<b>r</b>ecursive) will delete the directory and everything in it. You can also disable confirmation with `-f` (<b>f</b>orce). Be VERY careful with this. `rm -rf *` will remove all files and directories from this directory up without confirmation. This cannot be reversed.\n",
"These all accept wild cards so `rm *.out` removes all files ending in `.out`. `rm` will also remove empty directories and with `-r` option (<b>r</b>ecursive) will delete the directory and everything in it. You can also disable confirmation with `-f` (<b>f</b>orce). Be \\*VERY\\* careful with this. `rm -rf *` will remove all files and directories from this directory up without confirmation. This cannot be reversed.\n",
"\n",
"We can make copies with `cp` (<b>c</b>o<b>p</b>y)"
]
Expand Down Expand Up @@ -285,7 +300,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"`mv` is best for changing the file's directory or renaming files. To rename multiple files there is the command `rename` (on linux at least, it's not on all distributions which is a shame as it's handy) which has the form\n",
"`mv` is best for changing the file's directory or renaming files. To rename multiple files there sometimes is the command `rename` (it's not on all distributions, which is a shame as it's handy. Check with `man rename`) which has the form\n",
"\n",
"`rename 'old string' 'new string' 'pattern to match files'`\n",
"\n",
Expand Down Expand Up @@ -360,7 +375,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"`cat` is ony good for small files as it reads all of them at once but it can also con<b>cat</b>enate files (join them together). `less` and `more` both just do a page at a time and have a lot of options with `-n` for line numbers being the most useful. `tail` lets you read from the bottom of a file instead of the top and `tail -f` (<b>f</b>ollow) is useful for tracking output to files your code to without locking them (which can cause code to crash.)\n",
"`cat` is ony good for small files as it reads all of them at once but it can also con<b>cat</b>enate files (join them together). `less` and `more` both just do a page at a time and have a lot of options with `-n` for line numbers being the most useful. `tail` lets you read from the bottom of a file instead of the top and `tail -f` (<b>f</b>ollow) is useful for tracking output to files your code is writing to without locking them (which can cause code to crash.)\n",
"\n",
"### Exercise\n",
"\n",
Expand All @@ -387,7 +402,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Will list our directory and write it to a file called `output.txt`, which is will create if it does not exist (so `> somefile.txt` works the same as `touch somefile.txt`)."
"Will list our directory and write it to a file called `output.txt`, which is will create it if it does not exist (so `> somefile.txt` works the same as `touch somefile.txt`)."
]
},
{
Expand Down Expand Up @@ -438,7 +453,7 @@
"\n",
"### Exercise\n",
"\n",
"5. Go into the directory `pycamb` then write a single line command that finds all `.py` files in all directories and count how any lines contain `if`\n",
"5. Go into the directory `pycamb` then write a single line command that finds all `.py` files and count how any lines contain `if`\n",
"6. Guess the meaning of the following:"
]
},
Expand Down Expand Up @@ -508,7 +523,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"This has many options with some of the more useful being `-o` whish sorts it by any column but `mem`, `user`, `time` and `cpu` (default) are useful and `-user me123` to display only one user. These options can be on the command line or while in the top window itself\n",
"This has many options with some of the more useful being `-o` which sorts it by any column but `mem`, `user`, `time` and `cpu` (default) are useful and `-user me123` to display only one user. These options can be on the command line or while in the top window itself\n",
"\n",
"When you start processes you can do so in the background with `jobscript &` then check them with the command `jobs`. Then `bg` sends the job to the background and `fg` sends it to the foreground."
]
Expand Down Expand Up @@ -604,7 +619,7 @@
"source": [
"This just makes your terminal window act like one on the remote machine. The `-X` allows 'X forwarding' so can transfer GUI interfaces between the computers. To edit files remotely you have to use the standard editors on the machine you have `ssh`ed to, usually `vim` or `emacs`. Both are powerful and have dedicated fans but the learning curve is very steep. You should google tutorials if you want to learn more, there are loads for both of them. \n",
"\n",
"You can copy the files betwwen the machines with `scp` (<b>s</b>ecure <b>c</b>o<b>p</b>y) or `sftp` (<b>s</b>ecure <b>f</b>ile <b>t</b>ransfer <b>p</b>rotocol). First `scp` works just like `cp`"
"You can copy the files between the machines with `scp` (<b>s</b>ecure <b>c</b>o<b>p</b>y) or `sftp` (<b>s</b>ecure <b>f</b>ile <b>t</b>ransfer <b>p</b>rotocol). First `scp` works just like `cp`"
]
},
{
Expand Down Expand Up @@ -662,7 +677,7 @@
"- personal ones in `~/.bash_profile` (login sessions) or `~/.bashrc` (other sessions)\n",
"- edit as you see fit, but be careful: mistakes can lead to inability to log in!\n",
"\n",
"*Environment variables* control how `bash` behaves; most important ones are `PATH` list of colon-separated directory names to look, in order, for commands typed in the prompt"
"*Environment variables* control how `bash` behaves; most important one is `PATH` list of colon-separated directory names to look, in order, for commands typed in the prompt"
]
},
{
Expand All @@ -674,6 +689,22 @@
"echo ${PATH}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To see all enviroment variables use `env`"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"env"
]
},
{
"cell_type": "markdown",
"metadata": {},
Expand Down Expand Up @@ -707,15 +738,156 @@
"source": [
"This sets up the shortcut using the `alias` command so when I type `cosmos2` it executes the command in red which launches an interactive job in the debuging queue. Then I load a bunch of modules I need to run my code on the machines.\n",
"\n",
"You should also note that all the commands here can be executed as a script (just a file with a list of commands try to give it the extension `.sh` as in `somthing.sh`). First you need to start the file with:\n",
"## Scripting\n",
"\n",
"You should also note that all the commands here can be executed as a script (just a file with a list of commands. It is often given the extension `.sh` as in `somthing.sh`. `sh` is an earlier version of `bash` so this is actually a bit misleading. It's not a requirement so you can do whatever you want but it's nice to have something that tells you what it does in the name). \n",
"\n",
"`#!/bin/sh`\n",
"First you need to start the file with:\n",
"\n",
"This isn't strictly nessecary but makes sure your computer knows that the commands are in the `bash` language (what the terminal is using for its `kernel` if any of the above worked. `kernel` just means 'thing which interprets the commands you type') and after this you can just list commands that you want. Then you have to change the file permissions to allow execution ie: `chmod 744 something.sh`. Then you can run it on the command line with\n",
"`#!/bin/bash`\n",
"\n",
"`somthing.sh`\n",
"This isn't strictly nessecary but makes sure your computer knows to interprets the following commands using the `bash` language (what the terminal is using for its `kernel` if any of the above worked. `kernel` just means 'thing which interprets the commands you type') and after this you can just list commands that you want. This can fail if `bash` isn't in the `/bin` directory. Instead you might need:\n",
"\n",
"`#!/usr/local/bin/bash` Another place bash sometimes lives\n",
"`#!/usr/bin/env bash` Works is `bash` is in your path\n",
"\n",
"You can in fact put any kernel here for any other scripting language, like for example:\n",
"\n",
"`#!/bin/python`\n",
"\n",
"Back to bash, once you have this line you can write commands just like you were entering them into the terminal. So an example might be:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#!/bin/bash\n",
"\n",
"cd ~code/output\n",
"rm -f *.log *.out"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You would then save this into a file `cleanup.sh`. Then you have to change the file permissions to allow execution ie: `chmod 744 cleanup.sh`. Then you can run it on the command line with\n",
"\n",
"`./cleanup.sh`\n",
"or\n",
"`bash cleanup.sh`\n",
"\n",
"This can be helpful for sets of commands you often run together.\n",
"\n",
"BASH is actually a fairly powerful programming language on its own. You have variables:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"var1=1\n",
"echo \"var1 = $var1\"\n",
"\n",
"This can be helpful for sets of commands you often run together. I good example would be cleaning up `.log` and `.out` files that are older than a month in certain directories"
"str1=\"Hello\"\n",
"echo \"str1 = $str1\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note the double quotes, single quotes will result in `echo` not expanding the variable by printing the name instead. Also you can't have any spaces in assignment so `var1 = 1` fails.\n",
"\n",
"It also has loops:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"for i in {1..10}; # {1..10} expands to \"1 2 3 4 5 6 7 8 9 10\"\n",
"do \n",
" echo \"List form: The iteration number is $i\"\n",
"done\n",
"\n",
"for (( i = 0; i < 10; i++ )) #C style loop\n",
"do\n",
" echo \"C style form: The iteration number is $i\"\n",
"done\n",
"\n",
"i=0\n",
"while [ $i -lt 5 ] #Executes until false\n",
"do\n",
" echo \"while: i is currently $i\"\n",
" i=$[$i+1] #Not the lack of spaces around the brackets. This makes it a not a test expression\n",
"done\n",
"\n",
"i=5\n",
"until [[ $i -eq 10 ]]; #Executes until true\n",
"do\n",
" echo \"until: i is currently $i\"\n",
" i=$((i+1))\n",
"done"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"and conditional expressions"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"if [ \"$num\" -eq 1 ]; then\n",
" echo \"the number is 1\"\n",
"elif [ \"$num\" -gt 2 ]; then\n",
" echo \"the number is greater than 2\"\n",
"else\n",
" echo \"The number was not 1 and is not more than 2.\"\n",
"fi"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"here `-lt`, `-gt`, `-eq` means \"less than\", \"greater than\" and \"equal to\" respectively. We don't really have time to go over `bash` properly and anyway it would be best to do so after learning python to get you used to coding first as `bash` isn't always all that clear. I will just mention one thing which might be useful which is `bash` has `&&` meaning \"and\" and`||` meaning \"or\" which help for error trapping. For the first example `cleanup.sh` we should really have done this:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#!/bin/bash\n",
"\n",
"cd ~code/output || exit\n",
"rm -f *.log *.out"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now the script will exit if it can't change into the directory. Before, if this line failed it would then have just deleted stuff whever you were which is a bit dangerous.\n",
"\n",
"All of the above code is in the file `test1` in the directory `Bash` if you want to test it out.\n",
"\n",
"NOTE: editing `bash` scripts on windows adds different new line characters and the code will fail. This can be fixed with things like the programme `dos2unix`."
]
},
{
Expand All @@ -726,7 +898,16 @@
"\n",
"This should be enough to get you started but there is plenty more. Here are the notes on the university information service course on the unix terminal:\n",
"https://help.uis.cam.ac.uk/service/help-support/training/downloads/course-files/programming-student-files/unix-cli/unix-cli-files/working-copy-unixcli.pdf.\n",
"`bash` is in fact an entire scripting language of its own which can produce surprisingly complicated programmes and even has some limited ability to parallelise. Here is a link to a free 100 page book for people who want to know more: https://books.goalkicker.com/BashBook/. Otherwise, as always in coding, google is your friend."
"\n",
"For `bash` scripting here is a link to a free 100 page book which is worth a skim for people who want to know more: https://books.goalkicker.com/BashBook/. \n",
"\n",
"Or consult the manual:\n",
"https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html\n",
"\n",
"Finally here is a list of all commands available in `bash`\n",
"https://courses.cs.washington.edu/courses/cse391/17sp/bash.html\n",
"\n",
"Otherwise, as always in coding, google is your friend."
]
},
{
Expand All @@ -753,7 +934,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.0"
"version": "3.7.1"
}
},
"nbformat": 4,
Expand Down
40 changes: 40 additions & 0 deletions Bash/test1
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
#!/usr/bin/env bash

var1=1
echo "var1 = $var1"

str1="Hello"
echo "str1 = $str1"

for i in {1..10}; # {1..10} expands to "1 2 3 4 5 6 7 8 9 10"
do
echo "List form: The iteration number is $i"
done

for (( i = 0; i < 10; i++ )) #C style loop
do
echo "C style form: The iteration number is $i"
done

i=0
while [ $i -lt 5 ] #Executes until false
do
echo "while: i is currently $i"
i=$[$i+1] #Not the lack of spaces around the brackets. This makes it a not a test expression
done

i=5
until [[ $i -eq 10 ]]; #Executes until true
do
echo "until: i is currently $i"
i=$((i+1))
done

num=5
if [ "$num" -eq 1 ]; then
echo "the number is 1"
elif [ "$num" -gt 2 ]; then
echo "the number is greater than 2"
else
echo "The number was not 1 and is not more than 2."
fi

0 comments on commit ed3e570

Please sign in to comment.