Labyrinth

Have you ever crawled a site? You probably have.

Now try to crawl this site. It should take some time :)

Installing

Use rvm to use Ruby 2.1.0
Run gem install bundler
Run bundler Possibly from the rvm install dir
Run rake:setup
Run rake:migrate
Run rackup

TODO

~~Add logging for requested urls~~
Use something not SQLite for logging This is real!
Cache the wordlist to give the disc some peace