-
Notifications
You must be signed in to change notification settings - Fork 11
/
hw-fuzzing-tabbed.html
156 lines (151 loc) · 18.4 KB
/
hw-fuzzing-tabbed.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
<!DOCTYPE html>
<html xmlns='http://www.w3.org/1999/xhtml' lang='' xml:lang=''>
<head>
<meta charset='utf-8'></meta>
<meta name='generator' content='pandoc'></meta>
<meta name='viewport' content='width=device-width, initial-scale=1.0, user-scalable=yes'></meta>
<title>ICS: Programming HW: Fuzzing</title>
<style>
code{white-space: pre-wrap;}
span.smallcaps{font-variant: small-caps;}
span.underline{text-decoration: underline;}
div.column{display: inline-block; vertical-align: top; width: 50%;}
div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;}
ul.task-list{list-style: none;}
.display.math{display: block; text-align: center; margin: 0.5rem auto;}
</style>
<link rel='stylesheet' href='../markdown.css'></link>
<script>
function openTab(evt, tabName) {
// Declare all variables
var i, tabcontent, tablinks;
// Get all elements with class="tabcontent" and hide them
tabcontent = document.getElementsByClassName("tabcontent");
for (i = 0; i < tabcontent.length; i++) {
tabcontent[i].style.display = "none";
}
// Get all elements with class="tablinks" and remove the class "active"
tablinks = document.getElementsByClassName("tablinks");
for (i = 0; i < tablinks.length; i++) {
tablinks[i].className = tablinks[i].className.replace(" active", "");
}
// Show the current tab, and add an "active" class to the button that opened the tab
document.getElementById(tabName).style.display = "block";
evt.currentTarget.className += " active";
}
</script>
</head>
<body>
<h1 id='ics-programming-hw-fuzzing'>ICS: Programming HW: Fuzzing</h1>
<p><a href='index.html'>Go up to the ICS HW page</a> (<a href='index.md'>md</a>) | <a href='hw-fuzzing.html'>view one-page version</a></p><div class='tab'>
<button class='tablinks' onclick="openTab(event,'toverview')" id='defaultOpen'>Overview</button>
<button class='tablinks' onclick="openTab(event,'tchangelog')">Changelog</button>
<button class='tablinks' onclick="openTab(event,'tsetup')">Setup</button>
<button class='tablinks' onclick="openTab(event,'ttask-1-basic-fuzzing')">Basic Fuzzing</button>
<button class='tablinks' onclick="openTab(event,'ttask-2-advanced-fuzzing')">Advanced Fuzzing</button>
<button class='tablinks' onclick="openTab(event,'ttesting')">Testing</button>
<button class='tablinks' onclick="openTab(event,'tsubmission')">Submission</button>
</div>
<div id='toverview' class='tabcontent'><h3 id='overview'>Overview</h3>
<p>In this assignment, you will be creating a <em>network fuzzer</em>, which is a program that will try out many different URL variants to see if any exist on a web server. Given a base URL, such as <code>http://server.com/FUZZ</code>, and a word list containing many items, it will replace the string <code>FUZZ</code> with each of the words in the word list, and see if that URL exists.</p>
<p>Credit: this homework is based heavily on <a href='https://github.com/kernelmethod/xfuzz/'>Will Shand’s xfuzz assignment</a>.</p>
<h4 id='background'>Background</h4>
<ul>
<li>You can read about what fuzzing is on the <a href='https://en.wikipedia.org/wiki/Fuzzing'>Fuzzing Wikipedia page</a>. You should know what fuzzing is before you proceed.</li>
<li>You should be familiar with the <a href='../slides/networks.html'>networking slide set</a>.</li>
<li>You should be familiar with the <a href='https://developer.mozilla.org/en-US/docs/Web/HTTP/Status'>HTTP status codes</a> – whenever a web page is requested, one of these codes is provided. For example, if the page is valid, it provides code 200 (OK) along with the page data. If the page is not found, then it returns 404 (not found).</li>
<li>You should have a recent version of Python, and you will need to be able to install packages, via pip</li>
</ul>
</div><div id='tchangelog' class='tabcontent'><h3 id='changelog'>Changelog</h3>
<p>Any changes to this page will be put here for easy reference. Typo fixes and minor clarifications are not listed here. So far there aren’t any significant changes to report.</p>
</div><div id='tsetup' class='tabcontent'><h3 id='setup'>Setup</h3>
<p>This assignment must be done in a recent version of Python. The only additional package you will need is <code>uvicorn</code> – to install it, enter <code>pip install uvicorn</code> (or <code>pip3 install uvicorn</code>).</p>
<p>You will need a number of files from this repository to work on this assignment:</p>
<ul>
<li><a href='fuzzing/fuzzer.py.html'>fuzzer.py</a> (<a href='fuzzing/fuzzer.py'>src</a>): this is where you will write the code for this assignment, and the only file that will be submitted.</li>
<li><a href='fuzzing/args.py.html'>args.py</a> (<a href='fuzzing/args.py'>src</a>) is supporting code for fuzzer.py that does the command-line argument parsing. You should not modify this file!</li>
<li><a href='fuzzing/common.txt'>common.txt</a>: the word list that we will be using for testing. The format is one word per line, with no spaces in the words. If a space is in the URL, then <code>%20</code> (the URL representation for a space) will be used instead; you do NOT have to replace <code>%20</code> with a space in your code.</li>
<li><a href='fuzzing/common-reduced.txt'>common-reduced.txt</a>: a smaller word list, described in detail below (in the ‘Testing’ section)</li>
<li><a href='fuzzing/server.py.html'>server.py</a> (<a href='fuzzing/server.py'>src</a>) code to run a mini web server on your machine for testing.</li>
</ul>
<h4 id='calling-the-code'>Calling the code</h4>
<p>The provided code in args.py already handles the command line argument parsing. Run <code>python3 fuzzer.py -h</code> to see the various options. You are going to write the <code>fuzz()</code> function in fuzzer.py – the parameter passed into that function contains the parse command line arguments – you can print it to the screen during development to see how they are stored (but be sure to remove that print statement – and any others beyond what is required – before submission).</p>
<h4 id='writing-the-code'>Writing the code</h4>
<p>You have to write the <code>fuzz()</code> function in <a href='fuzzing/fuzzer.py.html'>fuzzer.py</a> (<a href='fuzzing/fuzzer.py'>src</a>) – remove the <code>pass</code> line that is there once you insert your own code. The intent is for you to use the <code>urllib.request</code> library; documentation on that can be found <a href='https://docs.python.org/3/library/urllib.request.html'>here</a>.</p>
<p>For the basic fuzzing, you will likely need the <code>urllib.request.urlopen()</code> function and the <code>status</code> field of the object it returns.</p>
<p>For advanced fuzzing, you will want to create a <code>urllib.request.Request</code> object so that you can set more parameters.</p>
</div><div id='ttask-1-basic-fuzzing' class='tabcontent'><h3 id='task-1-basic-fuzzing'>Task 1: Basic fuzzing</h3>
<p>The first task is to perform basic testing. Given a URL, such as <code>http://server.com/FUZZ</code>, you will replace <code>FUZZ</code> with each word in the word list file, and then try that URL to see if you get a response other than 404 (not found). You will do this by modifying the <code>fuzz()</code> function in the <a href='fuzzing/fuzzer.py.html'>fuzzer.py</a> (<a href='fuzzing/fuzzer.py'>src</a>) skeleton code. Try printing out the parameter to the function to see how the command-line arguments are passed into the function. You can see in the skeleton code how one might load a URL from a string.</p>
<p>When you are ready to test this, read the ‘Testing’ section, below, for easy ways to test it locally on your computer.</p>
<h4 id='output'>Output</h4>
<p>The output for this assignment is very specific. Once you have completed the above task, any URL that does <em>NOT</em> return a 404 should display the output in the following EXACT format. (Note that you will modify what URLs to print later).</p>
<p>For example, if you were to call the fuzzer as: <code>python3 fuzzer.py -u http://ffuf.me/cd/basic/FUZZ -w common.txt</code>, then the output would be exactly:</p>
<pre><code>200 http://ffuf.me/cd/basic/class
200 http://ffuf.me/cd/basic/development.log</code></pre>
<p>Note that the output is the status number, a single space, and the URL. The order you print it out the lines does not matter, only the content on each line.</p>
<p>We are going to use this particular call the visible test when you submit your assignment (although we will use a smaller word list to save time, but it will contain both <code>class</code> and <code>development.log</code>). Other hidden tests will be used to grade your assignment.</p>
<p><strong>DO NOT HAVE ANY OTHER OUTPUT!</strong> We are going to test it by doing a file comparison, so if you have any other output it will report as not the same, and you will fail that test. Again, the order of the lines in your output does not matter.</p>
</div><div id='ttask-2-advanced-fuzzing' class='tabcontent'><h3 id='task-2-advanced-fuzzing'>Task 2: Advanced fuzzing</h3>
<p>There are a number of command-line parameters that the fuzzer.py file will accept. You have to implement usage of the others. You can find them all via <code>python3 fuzzer.py -h</code>. The remaining ones to implement are as follows. Note that these are already parsed for you; you just have to handle when those values are in the <code>args</code> parameter to the <code>fuzz()</code> function.</p>
<ul>
<li><code>-e EXTENSIONS</code> or <code>--extension EXTENSIONS</code>: One or more extensions to append (e.g. php, html, etc.). Multiple extensions may be provided. So if <code>-e php -e html</code> is provided, and the wordlist contains <code>hello</code> and <code>world</code>, then you should be replacing <code>FUZZ</code> with six different values: <code>hello</code>, <code>hello.php</code>, <code>hello.html</code>, <code>world</code>, <code>world.php</code>, and <code>world.html</code>.
<ul>
<li>The value to the <code>-e</code> parameter assumes that it will be prefixed with a period before being added to each word in the word list. So <code>-e html</code> means you will add <code>.html</code> to each word in the word list. However, note that the command line parameter inserts that period for you.</li>
<li>Note that adding any number of extensions still means you try the base word as well. So adding <code>-e html</code> means your program will try <em>both</em> <code>alert</code> and <code>alert.html</code>.</li>
</ul></li>
<li><code>-X METHOD</code> or <code>--method METHOD</code>: HTTP method to use (GET, POST, or PUT) (default: GET)
<ul>
<li>Your code should allow both upper-case and lower-case values (“get” and “GET”)</li>
<li>You can also use the <a href='https://docs.python.org/3/library/urllib.request.html#urllib.request.Request'>urllib.request.Request</a> class to set the method; you just pass that object into <a href='https://docs.python.org/3/library/urllib.request.html#urllib.request.urlopen'>urllib.request.urlopen()</a>. To check the method is received correctly, update <code>server.py</code> to print out the <code>scope</code> variable.</li>
</ul></li>
<li><code>-H HEADERS</code> or <code>--header HEADERS</code>: One or more HTTP headers to add to requests, in the form “HeaderName:HeaderValue” (e.g. “Content-Type:application/json” or “Host:FUZZ.example.com”). This may be specified one or more times.
<ul>
<li>You can again use <a href='https://docs.python.org/3/library/urllib.request.html#urllib.request.Request'>urllib.request.Request</a> to set the headers; to check the header is received correctly, print out the <code>scope</code> variable in <code>server.py</code>. As before, you then past the Request object into <code>urllib.request.urlopen</code>.</li>
<li>Example usage: adding <code>-H "MyHeader:foobarbaz"</code> will cause each request sent to the URL to include that header; note that there is no space after the colon</li>
<li>Note that, in <code>fuzz()</code>, the headers are received as a string with a colon (<code>:</code>) separating the key and value. You have to <code>split()</code> that, as what needs to be passed to the <code>urllib.request.Request</code> constructor is a dictionary of key-value pairs.</li>
</ul></li>
<li><code>-d DATA</code> or <code>--data DATA</code>: Data to send in the body of the HTTP request.
<ul>
<li>To see how to read it from uvicorn, look <a href='https://www.uvicorn.org/'>here</a> – specifically, look at the <code>read_body()</code> function, which is called (in the <code>app()</code> function in server.py) as <code>body = await read_body(receive)</code>. You can cut-and-paste that function right into server.py if you want to use it.</li>
<li>You can also use the <code>urllib.request.Request</code> class to set the data; to check the data is received correctly, print out the <code>scope</code> variable in <code>server.py</code>. As before, you then past the Request object into <code>urllib.request.urlopen</code>.</li>
<li>Warning: the data passed in to the <code>Request</code> object must be <code>bytes</code>, not a string or <code>None</code>.</li>
</ul></li>
<li><code>-mc MATCH_CODES</code>: Match <a href='https://en.wikipedia.org/wiki/List_of_HTTP_status_codes'>HTTP response codes</a>. May be specified multiple times. If left unspecified, defaults to the following response codes: [200, 301, 302, 401, 403]. Previously you printed out any URLs that did not return 404 (not found). That should now be modified to print out the URLs that return one of the response codes in this list (which is parsed for you and passed into the <code>fuzz()</code> function in the <code>args</code> parameter).
<ul>
<li>Specifying just one response code via <code>-mc</code> will replace the default list with just that one. So <code>-mc 200</code> will not check for any of the defaults other than 200. Note that the command line argument parsing does this for you.</li>
<li><strong>NOTE:</strong> this can be specified <em>multiple</em> times, at which point you would then have to check for <em>multiple</em> match codes.</li>
</ul></li>
</ul>
<p>When we test your code, we will never give it invalid parameters, and there will always be a <code>-u</code> parameter and a <code>-w</code> parameter. So you do not have to error-check the parameters.</p>
</div><div id='ttesting' class='tabcontent'><h3 id='testing'>Testing</h3>
<p>First test it locally, then test it remotely. Both are described below.</p>
<h4 id='local-testing'>Local testing</h4>
<p>Initially, as you are developing your code, you will want to test it out locally. One of the Python packages you installed is uvicorn, which is a small Python-based web server. To run it, you first need to save the <a href='fuzzing/server.py.html'>server.py</a> (<a href='fuzzing/server.py'>src</a>) file. This file has a <code>urls</code> list that is the set of URLs that it will respond with a 200 (OK) to; everything else it responds with a 404 (not found).</p>
<p>You are welcome to, and will eventually have to, change the elements in that list during testing – just note that each USLS part has to start with a forward slash. If you are running uvicorn, and you edit the server.py file, it should detect this and reload the contents so that you don’t have to restart it.</p>
<p>You will run your uvicorn server as follows:</p>
<pre><code>uvicorn server:app --reload --port 5000</code></pre>
<p>On some systems (Windows without WSL, in particular), you have to put <code>python -m</code> at the start of that command.</p>
<p>This runs the server on port 5000 on your computer. To test against this, you will run your fuzzer as follows:</p>
<pre><code>python3 fuzzer.py -u http://localhost:5000/FUZZ -w common.txt</code></pre>
<p>Note: it may be that your computer needs to use <code>http://127.0.0.1:5000/FUZZ</code> instead.</p>
<p>Also note: sometimes Macs have issues with port 5000. If that is the case, then try any other port (5001, 5002, etc.).</p>
<p>Following the output specifications described above, the output should be exactly as follows. Note that the order of the lines does not matter, as long as the format of each line is exact.</p>
<pre><code>200 http://localhost:5000/.gitignore
200 http://localhost:5000/employers
200 http://localhost:5000/~admin</code></pre>
<h4 id='uvicorn-server.py-edits'>Uvicorn server.py edits</h4>
<p>The <a href='fuzzing/server.py.html'>server.py</a> (<a href='fuzzing/server.py'>src</a>) file, which is what configures the uvicorn server, will need to be modified for you to test the additional command line flags. For example, you will may to try reporting 200 (OK) on <code>alert.html</code>, to test that your <code>-e html</code> works. Likewise, you will want to test out different server status values other than 200 to ensure that your <code>-mc</code> flag works.</p>
<h4 id='remote-testing'>Remote testing</h4>
<p>As a remote test, you can try it with the URL of http://ffuf.me/cd/basic:</p>
<pre><code>python3 fuzzer.py -u http://ffuf.me/cd/basic/FUZZ -mc 200 -w common.txt </code></pre>
<p>The output would be:</p>
<pre><code>200 http://ffuf.me/cd/basic/class
200 http://ffuf.me/cd/basic/development.log</code></pre>
<p>Please do <strong><em>NOT</em></strong> try it out against any other machines, as ITS will get very cranky at you for doing so.</p>
<h4 id='word-lists'>Word lists</h4>
<p>As mentioned above, the word list we are using is the <a href='fuzzing/common.txt'>common.txt</a> file that we provide. You can also use word lists in the <a href='https://github.com/danielmiessler/SecLists/'>SecLists repository</a> repository; you will probably want to use a word list in the Discovery/Web-Content/ directory. You do not need any word lists beyond the <a href='fuzzing/common.txt'>common.txt</a> file for this assignment – the additional ones are just if you are interested.</p>
<p>There is also a <a href='fuzzing/common-reduced.txt'>common-reduced.txt</a> file that contains only 100 words. It contains the five URLs that are expected to be found in the tests herein (<code>.gitignore</code>, <code>employers</code>, and <code>~admin</code> for the uvicorn testing, and <code>class</code> and <code>development.log</code> for the fuff.me testing), as well as 95 other randomly chosen words from common.txt. This file is useful for testing, as it will take much less time to print the results.</p>
</div><div id='tsubmission' class='tabcontent'><h3 id='submission'>Submission</h3>
<p>You should only submit your <code>fuzzer.py</code> file to Gradescope. The only visible test will be the <code>python3 fuzzer.py -u http://ffuf.me/cd/basic/FUZZ -w common-reduced.txt</code>.</p>
</div><script>document.getElementById('defaultOpen').click();</script></body>
</html>