Use AWS Lambda to pull E-Scooter/Bike Location Data, store in Redshift Data Vault. Serve to Google Data Studio Dashboard.

This code takes a list of E-Scooter APIs stored in a JSON file and uses AWS Cloudwatch & SNS to run Python in Lambda to GET them, store them in AWS in a S3 Data Lake and a Redshift Data Warehouse using a Data Vault Data Model. Then it serves the data in a Google Data Studio Dashboard.

Table of Contents

1. About The Project
   • Built With
2. Prerequisites/Instructions
3. Contact

About The Project

• Personal mobility devices are changing transportation in major cities. This data is publically available from various companies via APIs if you spend the time and money to collect it. This project collects and tracks scooter data from companies over time using AWS and implementing Data Vault data modeling.

• I take various E-Scooter Scooter APIs, use AWS to pull and store data (Cloud Watch, SNS, Lambda, S3, Redshift) and serve a dashboard using Google Data Studio.

Built With

• Tech

  • Python
  • Pandas
  • AWS Cloud Watch
  • AWS SNS
  • AWS Lambda
  • AWS S3
  • AWS Redshift
  • GCP Data Studio
  • JSON

• E-Scooter/Bike APIs (7 APIs, 56 Locations)

  • Bird Bird with Battery Status
  • Lime
  • Lime (Rentals)
  • Lyft
  • Razor
  • Spin

Prerequisites

1. Create a cluster in Redshift open to public IPs & add the schemas, tables, views, and stored procedures included in the sql folder. The data model will be a data vault model and store each change in the sattelites.

2. Create a lambda function with the lambda_function.py file. Swap out the "ACCESS_KEY", "SECRET_KEY", "creds", and "my_bucket" names. Also add Data wrangler and psycopg layers. Trigger it with the sns topic you'll make in the next step.

3. Create an sns topic.

4. Create a cloud watch schedule to run every X minutes (you choose X).

5. Create Data Studio dashboard pulling from Redshift query. Schedule to send to self daily.

Contact

Jared Fiacco - [email protected]

Another GCP Project of Mine: Pull and Store and Server Clash of Clans API Data

A GCP Project of Mine: Publish Computer Statistics to Pub/Sub, Use Cloud Functions to Store in BigQuery