This project demonstrates building a big data pipeline on AWS at scale. The Covid-19 dataset is streamed real-time from an external UPI using NiFi. The complex JSON data is parsed into CSV using NiFi and the result is stored in HDFS. This data is then sent to Kafka for data processing using PySpark. The processed data will then be consumed from Spark and stored in HDFS. A Hive table is created on top of HDFS and finally the cleaned, transformed data is stored in the data lake and deployed.
- Use orchestration other than AirFlow, like Luigi, Prefect, Dagster
- Create a visualization