Streaming Twitter Data with Apache Spark/Pyspark
In Folder "Code" one can find following Notebooks:
- to_Kafka: TOPIC: Sending Twitter Data to Kafka Consumer on a Azure VM
- Reading_Kafka_Stream_data_from_twitter.ipynb: TOPIC: Reading Twitter Data from Kafka Consumer, dooing a basic agg and in the final step sending Data to Postgres.
- sql_check_if_data_is_in_Postgres.ipynb: TOPIC: Checking with sql-Alchemy if the Data arrived in Postgres as well as making Tables and or droping tables.
- Alerts_If_trashold_reached.ipynb: TOPIC: If a trashold is reached (COUNT TWEETS per City) then code sends an email with some message and with Data.