Master's thesis

Generating Synthetic Web Traffic

---

Author: Peter Koprda [email protected]

Supervisor: Ing. Radek Hranický, Ph.D. (UIFS FIT VUT) [email protected]

Requirements

• Python 3
• pip package installer

Setup

Create virtual environment

Install virtual environment using pip:

pip install virtualenv

Create the new virtual environment for the project:

python3 -m venv crawler-env

Activate virtual environment:

source crawler-env/bin/activate

Install libraries

Set up environment variables and install dependency libraries:

source setup.sh

Configuration

The application can be configured using two YAML files stored in config directory:

• config.yaml - contains settings for session headers, cookies, user agents, web crawler behavior, proxy settings, XSS scanner, SQL injection scanner, and logging
• user-agents.yaml - specifies the user agents to be used during the crawling process

Run the app

To run the application, use the run.py script with the following options:

python3 run.py [-h] -u TARGET [-c METHOD] [-l VALUE] [-a TYPE]

Options:

• -h, --help - show the help message and exit
• -u TARGET, --url TARGET - specify the target URL for the crawl and the attack
• -c METHOD, --crawl METHOD - specify the type of crawling to be used (bfs for Breadth-First Search, dfs for Depth-First-Search)
• -l VALUE, --level VALUE - specify the depth level of crawling
• -a TYPE, --attack TYPE - specify the type of attack (xss for Cross-Site Scripting, injection for SQL injection)

Example:

python3 run.py -u reddit.com -c bfs -l 3 -a xss

This command will run the BFS crawler with a maximum depth level of 3 and perform an XSS vulnerability attack on the founded webpages.