HCF3263827 / scaleable-crawler-with-docker-cluster Public

forked from arianpasquali/scaleable-crawler-with-docker-cluster

Notifications You must be signed in to change notification settings
Fork 0
Star 0

a scaleable and efficient crawelr with docker cluster , crawl million pages in 2 hours with a single machine

0 stars 27 forks Branches Tags Activity

Notifications

Name		Name	Last commit message	Last commit date
Latest commit History 31 Commits
test_celery		test_celery
README.md		README.md
docker-compose.yml		docker-compose.yml
dockerfile		dockerfile
requirements.txt		requirements.txt

Repository files navigation

scaleable-crawler-with-docker-cluster

a scaleable and efficient crawelr with docker cluster , crawl million pages in 2 hours with a single machine Check out here for full document

About

a scaleable and efficient crawelr with docker cluster , crawl million pages in 2 hours with a single machine

Report repository

Releases

No releases published

Packages

No packages published

Languages

Python 100.0%