Crawler

The crawler is a Application that searches https://devblogs.microsoft.com/oldnewthing/ and https://herbsutter.com/ using boost asio.

In addition, it has the following additional functions.

• The request and parsing parts were implemented in parallel.
• Object-oriented design makes it easy to add a web blog crawler and an api crawler.
• When searching, the virtual list is displayed even if the number of items is large.

Please check Project details for more information.

Project Details

For details on project structure and technical design, please see the Project Readme.

Support only 64 bit build now because of memory.

IDE

Microsoft Visual Studio 2017 version 15.9.11

Fast Start

Install Crawler

git clone https://github.com/seh9306/Crawler.git

Submodule Update

git submodule init 
git submodule update

Connect Dll Link

It tells the Crawler.exe the path to the sqlite.dll file.

Demo

Gif

Crawl

Seach

Crawl Complete