Code

crawl delay support#4

Code

Open

johnjansen wants to merge 2 commits intorinzi:masterrinzi/robotstxt:masterfrom quid:masterCopy head branch name to clipboard

+110-16

Commits

Commits on Mar 4, 2013

• Hacked in support for crawl delay, removed some duplication in parser, added web mock … quick and dirty

  

  John Jansen

  committed
  f5a1e75

• added a gemspec

  

  John Jansen

  committed
  8bb60dd