Skip to content

Explore the world of Data Engineering through a sophisticated ETL pipeline leveraging Reddit's API, AWS S3, Redshift, dbt transformations, and Airflow orchestration in Docker. Visualize insights on r/dataengineering with a dynamic Google Data Studio report, showcasing the power of integrated data tools in action.

Notifications You must be signed in to change notification settings

Naga-Manohar-Y/Reddit-Pipeline-DE

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

7 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Reddit ETL Pipeline

A data pipeline to extract Reddit data from r/dataengineering.

Output is a Google Data Studio report, providing insight into the Data Engineering official subreddit.

Motivation

Project was based on an interest in Data Engineering and the types of Q&A found on the official subreddit.

It also provided a good opportunity to develop skills and experience in a range of tools. As such, project is more complex than required, utilising dbt, airflow, docker and cloud based storage.

Architecture

  1. Extract data using Reddit API
  2. Load into AWS S3
  3. Copy into AWS Redshift
  4. Transform using dbt
  5. Create PowerBI or Google Data Studio Dashboard
  6. Orchestrate with Airflow in Docker
  7. Create AWS resources with Terraform

Output

  • Final output from Google Data Studio. Link here. Note that Dashboard is reading from a static CSV output from Redshift. Redshift database was deleted so as not to incur cost.

Setup

Follow below steps to setup pipeline. I've tried to explain steps where I can. Feel free to make improvements/changes.

NOTE: This was developed using an M1 Macbook Pro. If you're on Windows or Linux, you may need to amend certain components if issues are encountered.

As AWS offer a free tier, this shouldn't cost you anything unless you amend the pipeline to extract large amounts of data, or keep infrastructure running for 2+ months. However, please check AWS free tier limits, as this may change.

First clone the repository into your home directory and follow the steps.

git clone https://github.com/ABZ-Aaron/Reddit-API-Pipeline.git
cd Reddit-API-Pipeline
  1. Overview
  2. Reddit API Configuration
  3. AWS Account
  4. Infrastructure with Terraform
  5. Configuration Details
  6. Docker & Airflow
  7. dbt
  8. Dashboard
  9. Final Notes & Termination
  10. Improvements

About

Explore the world of Data Engineering through a sophisticated ETL pipeline leveraging Reddit's API, AWS S3, Redshift, dbt transformations, and Airflow orchestration in Docker. Visualize insights on r/dataengineering with a dynamic Google Data Studio report, showcasing the power of integrated data tools in action.

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published