I worked with a group to scrape the UCI Information and Computer Science Repository. The main code is in scraper.py. We started with a seed URL and extracted its HTML for important text in specfic headings. This was also the way we extracted the next url. Additionally, we did some text processing such as stemming to make our index stronger. To avoid duplicate documennts, we introduced the use of SimHash.
To show some of the results, here are the following:
