- a service account
- credential key file saved in your local machine
After following the steps below, the following resources will be provisioned in GCP:
- Firewall Rule
- 2 Compute Instances
- kafka-vm
- airflow-vm
- Cloud Storage Bucket
- Dataproc Cluster
- 1 master node
- BigQuery Dataset
- Staging
- Production
-
setup terraform
-
configure the terraform
variables.tf
file- specify your project ID in line 3
- edit other variables as you see fit
-
run the following commands in your terminal
-
navigate to the terraform directory
cd ~/musicaly-project/terraform
-
initiatalize terraform and download the required dependencies
terraform init
-
preview the infrastructure to be provisioned
terraform plan
-
apply the config and provision the infrastructure
terraform apply
-
to tear down the resources after completing the project
terraform destroy
-