== Medusa: a ruby crawler framework

Medusa is a ruby framework to crawl and collect useful information about the pages it visits. It is versatile, allowing you to write your own specialized tasks quickly and easily.

• Choose the links to follow on each page with focus_crawl()
• Multi-threaded design for high performance
• Tracks 301 HTTP redirects
• Allows exclusion of URLs based on regular expressions
• HTTPS support
• Records response time for each page
• Obey robots.txt
• In-memory or persistent storage of pages during crawl using Moneta adapters.
• Inherits OpenURI behavior (redirects, automatic charset and encoding detection, proxy configuration options).