- 中文介绍(Chinese)
Chinese README.md Here
- If you meet some problem when running this demo, you can send email or make an issue to me.
- Email: [email protected]
- QQ: 379978424
- Zookeeper Environment(zookeeper-3.4.10) -> Install Step
- Kafka Environment(kafka-1.0.0) -> Install
- System Environment(Win10 x64)
- Python Environment(Python 3.4.4)
- Environment
- Python3.4.4(Python2 not test. If someone has tested it, please tell me or make an issue.)
- Library
- How to install:
windows: pip install requirements.txt
linux: pip3 install requiremnets.txt
- consumer --- pykafka consumer module(Test reception and receive Scrapy data)
- producer --- pykafka producer modele(Test send something)
- scrapy_kafka --- Scrapy + pykafka spider(Crawl all html a tag from my campus website(main page).)
- I choose some special point about kafka instead of Scrapy.
- kafka need bytes data for transfer, so when pipeline are receive the data. You must use encode method; Encode method`s argument encoding should same as with consumer decode arguments.
- Implement close_spider(self, spider) method in pipeline for shutdown the producer; Otherwise Scrapy will hang on for wating producer close.
- I code a method in pipeline to judge setting
KAFKA_IP_PORT
:
- Single deployment can use list or str.
- Pseudo distributed or fully distributed can use list, or use str(multiple IP and ports are separated by comma). -> e.g: "192.168.1.101:9092,192.168.1.102:9092"
- Download zookeeper.Download link
- Unzip the package, and get into
conf
, copy zoo_sample.cfg
to zoo.cfg
or rename.
- set zookeeper
root path
as ZOOKEEPER_HOME
as system environment variable
- add zookeeper
bin
path in PATH.
- run
zkServer
in cmd.
- Download kafka and unzip.Download link
- Get into
conf
, and edit server.properties
variable log.dirs
to somewhere you make folder for save log.
- Configure system environment variable
KAFKA_HOME
.
- add kafka bin/windows path in
PATH
(if you use linux juet set bin
in PATH
)
- run kafka: kafka-server-start
- Create topic:
kafka-topics --create --topic newtest --partitions 1 --replication-factor 1 --zookeeper localhost:2181
- Create producer:
kafka-console-producer --broker-list localhost:9092 --topic newtest
- At the moment, the window is in the state of waiting for input. Don`t close it, please run consumer.
- Create consumer
kafka-console-consumer --zookeeper localhost:2181 --topic newtest
- When consumer is start success. You can back to producer window and input something. Then you can check consumer window receiver something you input at producer window.
- Other kafka operation, please check the official document.