News Aggregator

This is a Web Application developed using GoLang which collects daily news from The Washington Post website using their sitemap url.

In this web app I have demonstrated how we can use Go Routines to speed up the loading speed of the site by reducing the idle time using Concept of Concurrency

Go routines are functions or methods that run concurrently with other functions or methods. Go routines can be thought of as light weight threads. The cost of creating a Go routine is tiny when compared to a thread. On Go we can run hundreds of thousands or millions of go routines.

How to use

Requirements

Go must be installed to run the program.
To install Go follow the official site instructions https://golang.org/doc/install

Steps

1. Clone the repository

       https://github.com/shushantkumar/NewsAggregator-Webapp.git
   

2. Move into the directory

       cd NewsAggregator-Webapp
   

3. To run the version with Concurrency implemented

       go run final_withconc.go
   

   and open http://localhost:8000 in your browser

4. To run the version with Concurrency not implemented

       go run before_conc.go
   

   and open http://localhost:8000 in your browser

Working

• The file before_conc.go access sitemaps without any go routines. The idle time while loading was 11631.0 ms (avg around 11500 ms for other tries).
  
• Since the web app accesses hundreds of sitemaps and it sends request for each sitemap and waits for response it wastes a lot of time increasing the idle time.
  

• To reduce the loading time I used Go Routines.
  
• The file final_withconc.go implements the same function but uses Go Routines for accessing each internal sitemaps, so basically when one request is sent other requests are also sent concurrently without waiting for previous request to get back the response.
  
• After implementing go routines for this web application the idle time went down to 1444.9 ms (avg around 1450 ms for other tries).
  

• The basic concept used here is concurrency. I also used Channels to provide concurrency. It allows goroutines to synchronize without explicit locks or condition variables.
  
• Synchronization needs to be there while using go routines. To implement it I imported "sync". Go provides another built in feature defer and functions under sync - Done(), Add() and Wait().
  

Sitemap link : https://www.washingtonpost.com/news-sitemap-index.xml