Update terraform

iac/README.md

@@ -1,32 +1,69 @@
-# IAC for ETL-Core infrastructure
+# IAC for ETL-Core Infrastructure
 
 ## Overview
-This repository contains the necessary infrastructure as code configure a secure environment on Google Cloud Platform (GCP) and deploy a Solana RPC node and RabbitMQ for the Solana-ETL pipeline.
+This repository contains the necessary infrastructure as code to set up a Solana RPC node and RabbitMQ for an ETL (Extract, Transform, Load) project. It provisions and configures a secure environment on Google Cloud Platform (GCP) for running a Solana node and RabbitMQ for efficient data handling and processing.
+
+## Prerequisites
+- Terraform >= 1.5.7
+- A GCP bucket for storing Terraform states
+- A Google Cloud Platform account
+- Authenticate Terraform to Google Cloud Platform:
+    ```bash
+    gcloud auth application-default login
+    ```
 
 ## Structure
-- `init/`: Contains Terraform scripts for enabling required APIs in the GCP environment.
-- `main/`: Houses the main Terraform configuration for provisioning the GCP environment. This includes setting up firewalls, VPC, and all prerequisites to securely run a Solana node.
-- `scripts/`: Includes shell scripts for configuring disks, downloading, and running the Solana RPC node, as well as setting up RabbitMQ configuration automatically.
+- **`init/`**: Contains Terraform scripts for enabling required APIs in the GCP environment.
+- **`main/`**: Houses the main Terraform configuration for provisioning the GCP environment, including firewalls, VPC, and all prerequisites to securely run a Solana node.
+- **`scripts/`**: Includes shell scripts for configuring disks, downloading and running the Solana RPC node, and setting up RabbitMQ configuration automatically.
 
-## Usage
-1. **API Initialization**:
-   - Navigate to the `init/` directory.
-   - Run the Terraform scripts to enable the necessary APIs in your GCP project adapt Region and Project variable to your needs
+## Deployment Steps
 
-2. **Environment Provisioning**:
-   - Move to the `main/` directory.
-   - Execute the Terraform scripts to provision the GCP environment, including firewalls, VPC, and other required infrastructure components.adapt the `variables.tf` to your needs and run the Terraform scripts to set up the GCP environment.
+### 1. API Initialization
+1. Navigate to the `init/` directory.
+2. Run the Terraform scripts to enable the necessary APIs in your GCP project. Adapt the region and project variables to your needs:
+    ```bash
+    terraform apply -auto-approve
+    ```
+3. Note: Make sure to replace the [bucket name](./init/main.tf) with your own.
 
+### 2. Environment Provisioning
+1. Move to the `main/` directory.
+2. Adapt the [variables.tf](./main/variables.tf) file to your needs and run the Terraform scripts to set up the GCP environment, including firewalls, VPC, and other required infrastructure components:
+    ```bash
+    terraform apply -auto-approve
+    ```
+3. Note: Make sure to replace the [project_id](./main/variables.tf) and [bucket name](./main/provider.tf) with your own.
 
-3. **Solana Node Configuration**:
-   - After provisioning the infrastructure, download and run the scripts located in the `scripts/solana-rpc` directory on the Solana node.
-   - These scripts will handle disk configuration, Solana RPC node setup and will expose RPC port on "8899"
+### 3. App Provisioning
+1. The Terraform code will create:
+   - 1 RabbitMQ VM
+   - 1 Inserter VM
+   - 1 Indexer VM
+   - 1 Solana RPC VM
+2. Specify the version for the Indexer app with the `VERSION` variable defined in the [script](./scripts/indexer/indexer-service.sh#L4).
+3. Specify the version for the Inserter app with the `VERSION` variable defined in the [script](./scripts/inserter/inserter-service.sh#L4).
+4. To connect to a pre-existing BigQuery dataset:
+   - A GCP service account with `roles/bigquery.dataEditor` on the target BigQuery dataset is required.
+   - The service account key needs to be copied to the Indexer VM.
+   - The location of the service account key is defined by the `SERVICE_ENVIRONMENT` variable in the [script](./scripts/indexer/indexer-service.sh#L4).
+5. Note: If the target BigQuery dataset is in the same project, the service account key is not required.
 
-4. **RabbitMQ configuration**
-   - After provisioning the RabbitMQ server, download the scripts/rabbitmq/run-rabbit.sh and run it,
-   - This script automates the installation and configuration of RabbitMQ on a Linux system, including updating packages, installing dependencies, setting up repositories, installing Erlang and RabbitMQ, enabling the management plugin, and configuring RabbitMQ settings.
+### 4. BigQuery Provisioning
+1. The BigQuery dataset is provisioned by the [bq.tf](./main/bq.tf) file with pre-created data tables.
+2. This setup is only used to create a target BigQuery dataset within the same project. Otherwise, the code should be commented out.
 
-## Prerequisites
-- A Google Cloud Platform account.
-- Terraform installed on your local machine.
-- Basic knowledge of GCP, Terraform, and shell scripting.
+### 5. Solana Node Configuration
+1. After provisioning the infrastructure, download and run the scripts located in the `scripts/solana-rpc` directory on the Solana node.
+2. These scripts handle disk configuration, Solana RPC node setup,Nginx configuraiton and will expose the RPC port on `8899` internally which will be proxied through Nginx on port 80.
+3. Execute the `configure-solana-disks.sh` script first and ensure it completes successfully:
+    ```bash
+    chmod +x configure-solana-disks.sh
+    ./configure-solana-disks.sh
+    ```
+4. Execute the `solana-install.sh` script and ensure it completes successfully:
+    ```bash
+    chmod +x solana-install.sh
+    ./solana-install.sh
+    ```
+5. Note: Ensure the scripts are run with root privileges.

iac/init/main.tf

@@ -6,20 +6,24 @@ terraform {
       version = "~> 5.2"
     }
   }
+  backend "gcs" {
+    bucket = <your bucket name>
+    prefix = "solana-bq/init"
+  }
 }
 
 
 variable "region" {
-  default = "us-east"
+  default = "us-central1"
 }
 
 variable "project" {
-    default =
+  default = "project-name"
 }
 
 locals {
-  project_id = ""
-  region     = ""
+  project_id = <your project id>
+  region     = "us-central1"
   env        = "shared"
   default_labels = {
     env        = local.env

iac/main/.gitignore

@@ -0,0 +1,3 @@
+.terraform.tfstate
+.terraform.tfstate.backup
+./terraform/*

iac/main/bq.tf

@@ -0,0 +1,39 @@
+locals {
+  schema_names = [
+    "Accounts",
+    "Block Rewards",
+    "Blocks",
+    "Instructions",
+    "Token Transfers",
+    "Tokens",
+    "Transactions"
+  ]
+
+  schemas_and_datasets = [
+    for name in local.schema_names : {
+      dataset_name = "crypto_solana_mainnet_us"
+      table_name   = name
+      schema_file  = name
+    }
+  ]
+}
+
+resource "google_bigquery_dataset" "solana_dataset" {
+  dataset_id                  = "crypto_solana_mainnet_us"
+  location                    = "US"
+  default_table_expiration_ms = 21600000
+}
+
+
+resource "google_bigquery_table" "solana_tables" {
+  for_each            = { for item in local.schemas_and_datasets : "${item.dataset_name}.${item.table_name}" => item }
+  dataset_id          = each.value.dataset_name
+  table_id            = each.value.table_name
+  deletion_protection = false
+
+  schema = file("${path.module}/schemas/${each.value.schema_file}.json")
+
+  depends_on = [
+    google_bigquery_dataset.solana_dataset
+  ]
+}

iac/main/cloudnat.tf

@@ -0,0 +1,13 @@
+resource "google_compute_router" "solana_bq_router" {
+  name    = "solana-bq-router"
+  network = google_compute_network.solana_etl.self_link
+  region  = var.region
+}
+
+resource "google_compute_router_nat" "solana_bq_nat_config" {
+  name                               = "solana-bq-nat-config"
+  router                             = google_compute_router.solana_bq_router.name
+  region                             = var.region
+  nat_ip_allocate_option             = "AUTO_ONLY"
+  source_subnetwork_ip_ranges_to_nat = "ALL_SUBNETWORKS_ALL_IP_RANGES"
+}