Data Processing Pipeline

Diagram of Data Processing Pipeline

Specs of our pipeline

As students, we would like to reduce the cost as much as possible. We figured out through testing that an E2 micro instance works for our scrapper.

• VMs:
  • E2 Micro instance (2 vCPU, 1GB RAM)
  • OS: Container optimized
  • Storage: 10GB persistent
• Cloud function: serverless function set to 216mb of RAM. It gets triggered when there is an upload to the bucket.
• Cloud Storage: 5GB located at us-east1 (South Carolina).

Cost analysis

• VMs: 0.009$/h per instance. We are using four so total of ~25$/h.
• Function: Very generous tier, we don't expect going over it.
• Storage: Free 5GB, we don't expect to go other it either.

Total cost: Around 25$/month if the scrappers run 24/7.