Skip to content

Latest commit

 

History

History
50 lines (43 loc) · 1.24 KB

README.md

File metadata and controls

50 lines (43 loc) · 1.24 KB

infrastructure setup

prerequisite

  • a service account
  • credential key file saved in your local machine

After following the steps below, the following resources will be provisioned in GCP:

  • Firewall Rule
  • 2 Compute Instances
    • kafka-vm
    • airflow-vm
  • Cloud Storage Bucket
  • Dataproc Cluster
    • 1 master node
  • BigQuery Dataset
    • Staging
    • Production

how to setup

  1. setup terraform

  2. configure the terraform variables.tf file

    • specify your project ID in line 3
    • edit other variables as you see fit
  3. run the following commands in your terminal

    • navigate to the terraform directory

      cd ~/musicaly-project/terraform
    • initiatalize terraform and download the required dependencies

      terraform init
    • preview the infrastructure to be provisioned

      terraform plan
    • apply the config and provision the infrastructure

      terraform apply
    • to tear down the resources after completing the project

      terraform destroy