crawlerdetect is a Go version of PHP class @CrawlerDetect.
It helps to detect bots/crawlers/spiders via the user agent and other HTTP-headers. Currently able to detect 1,000's of bots/spiders/crawlers.
go get github.com/x-way/crawlerdetect
import "fmt"
import "github.com/x-way/crawlerdetect"
func main() {
uastring := "curl/7.54.0"
if crawlerdetect.IsCrawler(uastring) {
fmt.Println("Found a crawler")
}
}
The patterns and testcases are synced from the PHP repo. If you find a bot/spider/crawler user agent that crawlerdetect fails to detect, please submit a pull request with the regex pattern and a testcase to the upstream PHP repo.