From 429ed989ae49da10b06060f62c1e8eb11f1cb024 Mon Sep 17 00:00:00 2001 From: tornike Date: Wed, 7 Aug 2024 12:41:11 +0400 Subject: [PATCH 1/5] update for e2e pipeline for infra deployment --- iac/README.md | 81 +++++-- iac/init/main.tf | 12 +- iac/main/.!48051!plan | 0 iac/main/.!48137!plan | 0 iac/main/.!49459!plan | 0 iac/main/.gitignore | 3 + iac/main/.terraform.lock.hcl | 73 +++--- iac/main/bq.tf | 39 +++ iac/main/cloudnat.tf | 13 + iac/main/compute-engine.tf | 111 ++++++++- iac/main/firewall.tf | 20 +- iac/main/iam.tf | 19 ++ iac/main/network.tf | 30 ++- iac/main/provider.tf | 8 +- iac/main/schemas/Accounts.json | 229 ++++++++++++++++++ iac/main/schemas/Block Rewards.json | 50 ++++ iac/main/schemas/Blocks.json | 50 ++++ iac/main/schemas/Instructions.json | 93 +++++++ iac/main/schemas/Token Transfers.json | 77 ++++++ iac/main/schemas/Tokens.json | 94 +++++++ iac/main/schemas/Transactions.json | 165 +++++++++++++ iac/main/variables.tf | 12 +- iac/scripts/indexer/indexer-service.sh | 101 ++++++++ iac/scripts/inserter/inserter-service.sh | 82 +++++++ .../{rabbitmq => rabbit-mq}/run-rabbit.sh | 15 +- .../solana-rpc/configure-solana-disks.sh | 20 ++ iac/scripts/solana-rpc/solana-install.sh | 146 ++++++++++- 27 files changed, 1446 insertions(+), 97 deletions(-) delete mode 100644 iac/main/.!48051!plan delete mode 100644 iac/main/.!48137!plan delete mode 100644 iac/main/.!49459!plan create mode 100644 iac/main/.gitignore create mode 100644 iac/main/bq.tf create mode 100644 iac/main/cloudnat.tf create mode 100644 iac/main/schemas/Accounts.json create mode 100644 iac/main/schemas/Block Rewards.json create mode 100644 iac/main/schemas/Blocks.json create mode 100644 iac/main/schemas/Instructions.json create mode 100644 iac/main/schemas/Token Transfers.json create mode 100644 iac/main/schemas/Tokens.json create mode 100644 iac/main/schemas/Transactions.json create mode 100644 iac/scripts/indexer/indexer-service.sh create mode 100644 iac/scripts/inserter/inserter-service.sh rename iac/scripts/{rabbitmq => rabbit-mq}/run-rabbit.sh (87%) diff --git a/iac/README.md b/iac/README.md index f9e31af..a78dc3c 100644 --- a/iac/README.md +++ b/iac/README.md @@ -1,32 +1,69 @@ -# IAC for ETL-Core infrastructure +# IAC for ETL-Core Infrastructure ## Overview -This repository contains the necessary infrastructure as code configure a secure environment on Google Cloud Platform (GCP) and deploy a Solana RPC node and RabbitMQ for the Solana-ETL pipeline. +This repository contains the necessary infrastructure as code to set up a Solana RPC node and RabbitMQ for an ETL (Extract, Transform, Load) project. It provisions and configures a secure environment on Google Cloud Platform (GCP) for running a Solana node and RabbitMQ for efficient data handling and processing. + +## Prerequisites +- Terraform >= 1.5.7 +- A GCP bucket for storing Terraform states +- A Google Cloud Platform account +- Authenticate Terraform to Google Cloud Platform: + ```bash + gcloud auth application-default login + ``` ## Structure -- `init/`: Contains Terraform scripts for enabling required APIs in the GCP environment. -- `main/`: Houses the main Terraform configuration for provisioning the GCP environment. This includes setting up firewalls, VPC, and all prerequisites to securely run a Solana node. -- `scripts/`: Includes shell scripts for configuring disks, downloading, and running the Solana RPC node, as well as setting up RabbitMQ configuration automatically. +- **`init/`**: Contains Terraform scripts for enabling required APIs in the GCP environment. +- **`main/`**: Houses the main Terraform configuration for provisioning the GCP environment, including firewalls, VPC, and all prerequisites to securely run a Solana node. +- **`scripts/`**: Includes shell scripts for configuring disks, downloading and running the Solana RPC node, and setting up RabbitMQ configuration automatically. -## Usage -1. **API Initialization**: - - Navigate to the `init/` directory. - - Run the Terraform scripts to enable the necessary APIs in your GCP project adapt Region and Project variable to your needs +## Deployment Steps -2. **Environment Provisioning**: - - Move to the `main/` directory. - - Execute the Terraform scripts to provision the GCP environment, including firewalls, VPC, and other required infrastructure components.adapt the `variables.tf` to your needs and run the Terraform scripts to set up the GCP environment. +### 1. API Initialization +1. Navigate to the `init/` directory. +2. Run the Terraform scripts to enable the necessary APIs in your GCP project. Adapt the region and project variables to your needs: + ```bash + terraform apply -auto-approve + ``` +3. Note: Make sure to replace the [bucket name](./init/main.tf) with your own. +### 2. Environment Provisioning +1. Move to the `main/` directory. +2. Adapt the [variables.tf](./main/variables.tf) file to your needs and run the Terraform scripts to set up the GCP environment, including firewalls, VPC, and other required infrastructure components: + ```bash + terraform apply -auto-approve + ``` +3. Note: Make sure to replace the [project_id](./main/variables.tf) and [bucket name](./main/provider.tf) with your own. -3. **Solana Node Configuration**: - - After provisioning the infrastructure, download and run the scripts located in the `scripts/solana-rpc` directory on the Solana node. - - These scripts will handle disk configuration, Solana RPC node setup and will expose RPC port on "8899" +### 3. App Provisioning +1. The Terraform code will create: + - 1 RabbitMQ VM + - 1 Inserter VM + - 1 Indexer VM + - 1 Solana RPC VM +2. Specify the version for the Indexer app with the `VERSION` variable defined in the [script](./scripts/indexer/indexer-service.sh#L4). +3. Specify the version for the Inserter app with the `VERSION` variable defined in the [script](./scripts/inserter/inserter-service.sh#L4). +4. To connect to a pre-existing BigQuery dataset: + - A GCP service account with `roles/bigquery.dataEditor` on the target BigQuery dataset is required. + - The service account key needs to be copied to the Indexer VM. + - The location of the service account key is defined by the `SERVICE_ENVIRONMENT` variable in the [script](./scripts/indexer/indexer-service.sh#L4). +5. Note: If the target BigQuery dataset is in the same project, the service account key is not required. -4. **RabbitMQ configuration** - - After provisioning the RabbitMQ server, download the scripts/rabbitmq/run-rabbit.sh and run it, - - This script automates the installation and configuration of RabbitMQ on a Linux system, including updating packages, installing dependencies, setting up repositories, installing Erlang and RabbitMQ, enabling the management plugin, and configuring RabbitMQ settings. +### 4. BigQuery Provisioning +1. The BigQuery dataset is provisioned by the [bq.tf](./main/bq.tf) file with pre-created data tables. +2. This setup is only used to create a target BigQuery dataset within the same project. Otherwise, the code should be commented out. -## Prerequisites -- A Google Cloud Platform account. -- Terraform installed on your local machine. -- Basic knowledge of GCP, Terraform, and shell scripting. +### 5. Solana Node Configuration +1. After provisioning the infrastructure, download and run the scripts located in the `scripts/solana-rpc` directory on the Solana node. +2. These scripts handle disk configuration, Solana RPC node setup, and will expose the RPC port on `8899`. +3. Execute the `configure-solana-disks.sh` script first and ensure it completes successfully: + ```bash + chmod +x configure-solana-disks.sh + ./configure-solana-disks.sh + ``` +4. Execute the `solana-install.sh` script and ensure it completes successfully: + ```bash + chmod +x solana-install.sh + ./solana-install.sh + ``` +5. Note: Ensure the scripts are run with root privileges. diff --git a/iac/init/main.tf b/iac/init/main.tf index efd92be..d7221db 100644 --- a/iac/init/main.tf +++ b/iac/init/main.tf @@ -6,20 +6,24 @@ terraform { version = "~> 5.2" } } + backend "gcs" { + bucket = + prefix = "solana-bq/init" + } } variable "region" { - default = "us-east" + default = "us-central1" } variable "project" { - default = + default = "bcwt-sandbox" } locals { - project_id = "" - region = "" + project_id = + region = "us-central1" env = "shared" default_labels = { env = local.env diff --git a/iac/main/.!48051!plan b/iac/main/.!48051!plan deleted file mode 100644 index e69de29..0000000 diff --git a/iac/main/.!48137!plan b/iac/main/.!48137!plan deleted file mode 100644 index e69de29..0000000 diff --git a/iac/main/.!49459!plan b/iac/main/.!49459!plan deleted file mode 100644 index e69de29..0000000 diff --git a/iac/main/.gitignore b/iac/main/.gitignore new file mode 100644 index 0000000..baf627c --- /dev/null +++ b/iac/main/.gitignore @@ -0,0 +1,3 @@ +.terraform.tfstate +.terraform.tfstate.backup +./terraform/* diff --git a/iac/main/.terraform.lock.hcl b/iac/main/.terraform.lock.hcl index 71326b1..4b3ad30 100644 --- a/iac/main/.terraform.lock.hcl +++ b/iac/main/.terraform.lock.hcl @@ -2,40 +2,59 @@ # Manual edits may be lost in future updates. provider "registry.terraform.io/hashicorp/google" { - version = "5.9.0" - constraints = "~> 5.2" + version = "5.40.0" + constraints = "~> 5.40.0" hashes = [ - "h1:jG9wcaMKIuI8JSf8T+SjAcw5vhEpW/fnFfPjSXMbuEY=", - "zh:19c618c257b2d9e30a0978b1282b1e418748323ae74d9c1ad63a858cb159cd86", - "zh:2c1f18b6714062fe8eab633918b41c622423693f2a4fd747dc516f3578b8e738", - "zh:440b31f85e2d823919639c4d248a058cd90020724a2fa543546e36611ca18df4", - "zh:453edfad0fbd30e6d694f1b38cc9d5f0b8ec356bbce3f2919f1c4622518c46ca", - "zh:47965b68bf9afd2f6a7412792083911d22b6a1a17f0052c9a8329b5ade47bbe9", - "zh:5621990ad07b8cd9af6862f7a66b593b19bbdf20986d7c8cfd8948302810de51", - "zh:74e2380a9acceb552d067697c38b4679e950fc2ba4bf47025d8917910b08df3d", - "zh:a588be4fa16331c406a15e784d419a04e995741ed09eb2e14ec58b53f3ecd8cc", - "zh:a60af7611f69b76ff727ee569b1ebefee82a5e5e1f1809d2df04286ee2c0aa4e", - "zh:c15d781c9a198d343201eb1a4bc17c616ca8cb38bb33739c3e138db06022a171", - "zh:d5c15eeb3be0e01b17ed67ab9b52137480139816edd7e90e93643be57564d2d0", + "h1:3GmqYKo7bbbedWdAUTgEogezyx5SNfMfpbXOKffSxSg=", + "zh:441627704233ee27f67d8fa73466a4e0d4097f0f304ad994088bb7947c0d035d", + "zh:6bc65aaf6730a9d1ca597e47054fc07f9cad55d31a66a327818a3b4ffe656700", + "zh:882fb575f5e3f1e2968764569b373932bba789983075bd57cb464f4d2ac52b80", + "zh:a8b1bc92e0f2be329ac3b57cefd735850b1786f1b861b5db1a00d52bc57df6c3", + "zh:b8d446bdf44ca8ce0589b8c7fa04e3f2295ccf61fc05181da9693e142051e355", + "zh:c0a6ce0a02b03bf41259211030046e7f04706f25116e746279bbbbf5e8f10cdb", + "zh:c653ce4ec1e99b16bfc59e5b44b50a1b7b004bbbba98c8ddeb8e8db9860f77d5", + "zh:d2c841cad79edd48f08f4871327dc3664a0b4ce51a92fcaa85c13884523b1475", + "zh:d644d066989e40b9512d4c2d6d2a1a24075b0b3086dbe781127b8f3fb571be73", + "zh:d95b6ee54e9ee1f8bd22a47814f65222edbcecf5393a6aff598ef081f0f9b172", + "zh:dfc8bd95f1aac132fe136d7e13a963551b3853ce40e34425c290b6f17c5e14aa", "zh:f569b65999264a9416862bca5cd2a6177d94ccb0424f3a4ef424428912b9cb3c", ] } provider "registry.terraform.io/hashicorp/google-beta" { - version = "5.9.0" + version = "5.40.0" hashes = [ - "h1:/bizlmcNQaK/mYn8+WFayo2ZUVeKkWov2QXwfNGrofY=", - "zh:2fcb82487f8c5335646375bd98a44f22df5b01a3290317e138f638cc156da8a8", - "zh:452959d6aa53837b613dfa8d9e9f301cc7dba6c0176e03ac0d50408f1b1c6eb5", - "zh:49d1f65ba5a8fa5462a95c238aec17b8638b09bee28bb848b9f38d2911f7e8b7", - "zh:528821fcbf788721f71f1e78e38b24cf47da1785521637d211f95d01f53519d2", - "zh:54f9d5e2df07a463f23d40d04014eedad49b274280d220a81dc8f0bde5591226", - "zh:5767e527b13f66a7098fc3059786ade0eaa39fb6d6d3199a4976ccf2b3cdf280", - "zh:582d800aa8c5ff345ce9b5494f83d5d5684d553f5b317daf8e832aa30e708e47", - "zh:723ef344fb8e60244a18b9a0d3a38941f89b568aa09d2c5d7f5fb2693982da61", - "zh:b1ad47427a67f8c8a14868cd474ac8973a068b72dee4d1b828749017dc241212", - "zh:c5b980159090c9d8b5e664c79e324c60bc4c1a5f3cfa904999f027949869eef3", + "h1:ZjpwYkJ5jnR175Ox4kolOZdChKKK/0w3z4CY64NG1Hk=", + "zh:00ea69f59764ad54fc5c97a4b46c649fbd77181de3843277b5fe2dc03846a46d", + "zh:1a9d2320d17a9cfa10ce3153be6121d5a8ea9676c42f60153341f42e99be04ae", + "zh:1c293175cff1d5316f3f56e7517ebc6fb1e7d45fbc7c844371f8e0640671a35f", + "zh:1cb294f4f805a220e7177e28800788c5afb4520ccd2d5204f5602dc6c81ce309", + "zh:3f352c88f5c8b2ffe2e301ce0a0c381559a63f8b6c8c17f189dc4a42ae0ed7a5", + "zh:48b72165d5c5edf0e3a5c27e76c132789760e5256e78e5cf2ca1c968791c06e2", + "zh:6274b561d1697bc9901d218ef22e4b9202f4fe7be688cd895c744151f6e416e0", + "zh:71ae311789879248a0ed5daa22971c34111ec93a7e6a44c3905fbabcd37d6c3a", + "zh:a0b695d4b209638953a3692c882f15917ef0329aa454410314ff6e21c5ca41dd", + "zh:a3824dfb6df374f1009bd7a24f5acde5b2c2780362fbc3334bc68b132d2a8525", + "zh:ac200d49c72ba4ec5b8492d816eb564ea57a28448a87ac567c6b9f4c81465d5a", "zh:f569b65999264a9416862bca5cd2a6177d94ccb0424f3a4ef424428912b9cb3c", - "zh:ff5fea7d6c6b259938d1d2f008fd4d31242513e2ef592a68b0b6de7eab494a10", + ] +} + +provider "registry.terraform.io/hashicorp/local" { + version = "2.5.1" + hashes = [ + "h1:/GAVA/xheGQcbOZEq0qxANOg+KVLCA7Wv8qluxhTjhU=", + "zh:0af29ce2b7b5712319bf6424cb58d13b852bf9a777011a545fac99c7fdcdf561", + "zh:126063ea0d79dad1f68fa4e4d556793c0108ce278034f101d1dbbb2463924561", + "zh:196bfb49086f22fd4db46033e01655b0e5e036a5582d250412cc690fa7995de5", + "zh:37c92ec084d059d37d6cffdb683ccf68e3a5f8d2eb69dd73c8e43ad003ef8d24", + "zh:4269f01a98513651ad66763c16b268f4c2da76cc892ccfd54b401fff6cc11667", + "zh:51904350b9c728f963eef0c28f1d43e73d010333133eb7f30999a8fb6a0cc3d8", + "zh:73a66611359b83d0c3fcba2984610273f7954002febb8a57242bbb86d967b635", + "zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3", + "zh:7ae387993a92bcc379063229b3cce8af7eaf082dd9306598fcd42352994d2de0", + "zh:9e0f365f807b088646db6e4a8d4b188129d9ebdbcf2568c8ab33bddd1b82c867", + "zh:b5263acbd8ae51c9cbffa79743fbcadcb7908057c87eb22fd9048268056efbc4", + "zh:dfcd88ac5f13c0d04e24be00b686d069b4879cc4add1b7b1a8ae545783d97520", ] } diff --git a/iac/main/bq.tf b/iac/main/bq.tf new file mode 100644 index 0000000..d9e9264 --- /dev/null +++ b/iac/main/bq.tf @@ -0,0 +1,39 @@ +locals { + schema_names = [ + "Accounts", + "Block Rewards", + "Blocks", + "Instructions", + "Token Transfers", + "Tokens", + "Transactions" + ] + + schemas_and_datasets = [ + for name in local.schema_names : { + dataset_name = "crypto_solana_mainnet_us" + table_name = name + schema_file = name + } + ] +} + +resource "google_bigquery_dataset" "solana_dataset" { + dataset_id = "crypto_solana_mainnet_us" + location = "US" + default_table_expiration_ms = 21600000 +} + + +resource "google_bigquery_table" "solana_tables" { + for_each = { for item in local.schemas_and_datasets : "${item.dataset_name}.${item.table_name}" => item } + dataset_id = each.value.dataset_name + table_id = each.value.table_name + deletion_protection = false + + schema = file("${path.module}/schemas/${each.value.schema_file}.json") + + depends_on = [ + google_bigquery_dataset.solana_dataset + ] +} diff --git a/iac/main/cloudnat.tf b/iac/main/cloudnat.tf new file mode 100644 index 0000000..046fffd --- /dev/null +++ b/iac/main/cloudnat.tf @@ -0,0 +1,13 @@ +resource "google_compute_router" "solana_bq_router" { + name = "solana-bq-router" + network = google_compute_network.solana_etl.self_link + region = var.region +} + +resource "google_compute_router_nat" "solana_bq_nat_config" { + name = "solana-bq-nat-config" + router = google_compute_router.solana_bq_router.name + region = var.region + nat_ip_allocate_option = "AUTO_ONLY" + source_subnetwork_ip_ranges_to_nat = "ALL_SUBNETWORKS_ALL_IP_RANGES" +} diff --git a/iac/main/compute-engine.tf b/iac/main/compute-engine.tf index 446e01f..a1b5130 100644 --- a/iac/main/compute-engine.tf +++ b/iac/main/compute-engine.tf @@ -1,13 +1,14 @@ # RabbitMQ Server Configuration resource "google_compute_instance" "rabbitmq" { - name = "rabbitmq-server" - project = var.project_id - zone = "${var.region}-a" - machine_type = "n2-standard-8" - min_cpu_platform = "Intel Ice Lake" + name = "rabbitmq-server" + project = var.project_id + zone = "${var.region}-a" + # machine_type = "n2-standard-8" + machine_type = var.app_vm_size + # min_cpu_platform = "Intel Ice Lake" boot_disk { - auto_delete = false + auto_delete = true initialize_params { image = "https://www.googleapis.com/compute/beta/projects/ubuntu-os-cloud/global/images/ubuntu-minimal-2204-jammy-v20230715" size = 1200 @@ -19,11 +20,6 @@ resource "google_compute_instance" "rabbitmq" { network = google_compute_network.solana_etl.name subnetwork = google_compute_subnetwork.solana_etl.name - - access_config { - nat_ip = google_compute_address.rabbit_mq_public.address - network_tier = "STANDARD" - } network_ip = google_compute_address.rabbit_mq_internal.address } @@ -43,8 +39,99 @@ resource "google_compute_instance" "rabbitmq" { enable_integrity_monitoring = true enable_vtpm = true } + + metadata_startup_script = file("../scripts/rabbit-mq/run-rabbit.sh") + + tags = ["solana"] } +resource "google_compute_instance" "inserter" { + name = "bq-inserter" + project = var.project_id + zone = "${var.region}-a" + machine_type = var.app_vm_size + + boot_disk { + auto_delete = true + initialize_params { + image = "https://www.googleapis.com/compute/beta/projects/ubuntu-os-cloud/global/images/ubuntu-minimal-2204-jammy-v20230715" + size = 200 + type = "pd-balanced" + } + } + + network_interface { + + network = google_compute_network.solana_etl.name + subnetwork = google_compute_subnetwork.solana_etl.name + network_ip = google_compute_address.inserter_internal.address + + } + + scheduling { + automatic_restart = true + on_host_maintenance = "MIGRATE" + provisioning_model = "STANDARD" + } + + service_account { + email = google_service_account.app_sa.email + scopes = local.scopes + } + + shielded_instance_config { + enable_integrity_monitoring = true + enable_vtpm = true + } + + metadata_startup_script = file("../scripts/inserter/inserter-service.sh") + tags = ["solana"] +} + +resource "google_compute_instance" "indexer" { + name = "bq-indexer" + project = var.project_id + zone = "${var.region}-a" + machine_type = var.app_vm_size + + boot_disk { + auto_delete = true + initialize_params { + image = "https://www.googleapis.com/compute/beta/projects/ubuntu-os-cloud/global/images/ubuntu-minimal-2204-jammy-v20230715" + size = 200 + type = "pd-balanced" + } + } + + network_interface { + + network = google_compute_network.solana_etl.name + subnetwork = google_compute_subnetwork.solana_etl.name + network_ip = google_compute_address.indexer_internal.address + + } + + scheduling { + automatic_restart = true + on_host_maintenance = "MIGRATE" + provisioning_model = "STANDARD" + } + + service_account { + email = google_service_account.app_sa.email + scopes = local.scopes + } + + shielded_instance_config { + enable_integrity_monitoring = true + enable_vtpm = true + } + + metadata_startup_script = file("../scripts/indexer/indexer-service.sh") + tags = ["solana"] +} + + resource "google_compute_instance" "solana_rpc" { name = "solana-rpc" @@ -54,7 +141,7 @@ resource "google_compute_instance" "solana_rpc" { min_cpu_platform = "Intel Ice Lake" boot_disk { - auto_delete = false + auto_delete = true initialize_params { image = "https://www.googleapis.com/compute/beta/projects/debian-cloud/global/images/debian-11-bullseye-v20231010" size = 500 diff --git a/iac/main/firewall.tf b/iac/main/firewall.tf index cf406a2..9f9aba7 100644 --- a/iac/main/firewall.tf +++ b/iac/main/firewall.tf @@ -14,12 +14,12 @@ resource "google_compute_firewall" "allow_ssh_iap" { resource "google_compute_firewall" "solana-rpc" { name = "solana-rpc" allow { - ports = ["8000-10000"] + ports = ["8000-10000", "80"] protocol = "udp" } allow { - ports = ["8000-10000"] + ports = ["8000-10000", "80"] protocol = "tcp" } @@ -30,3 +30,19 @@ resource "google_compute_firewall" "solana-rpc" { target_tags = ["solana"] } +resource "google_compute_firewall" "allow_rabbitmq" { + name = "allow-rabbitmq" + network = google_compute_network.solana_etl.self_link + direction = "INGRESS" + priority = 1000 + source_ranges = ["0.0.0.0/0"] + + allow { + protocol = "tcp" + ports = ["5672"] + } + + target_tags = ["solana"] +} + + diff --git a/iac/main/iam.tf b/iac/main/iam.tf index 7e29e2e..5cd403a 100644 --- a/iac/main/iam.tf +++ b/iac/main/iam.tf @@ -5,6 +5,11 @@ locals { "roles/monitoring.metricWriter" # Add more roles as needed ] + + app_roles = [ + "roles/bigquery.dataEditor" + ] + } resource "google_service_account" "solana_etl" { account_id = "compute-service-account" @@ -19,3 +24,17 @@ resource "google_project_iam_member" "service_account_roles" { role = each.value member = "serviceAccount:${google_service_account.solana_etl.email}" } + +resource "google_service_account" "app_sa" { + account_id = "solana-app-sa" + display_name = "Solana Apps Account" +} + +resource "google_project_iam_member" "apps_sa_roles" { + for_each = toset(local.app_roles) + + project = var.project_id + role = each.value + member = "serviceAccount:${google_service_account.app_sa.email}" +} + diff --git a/iac/main/network.tf b/iac/main/network.tf index 0747ecb..180f4fa 100644 --- a/iac/main/network.tf +++ b/iac/main/network.tf @@ -1,13 +1,13 @@ resource "google_compute_network" "solana_etl" { auto_create_subnetworks = false mtu = 1460 - name = "${var.project_id}-vpc" + name = "${var.project_id}-solala-bq-vpc" project = var.project_id routing_mode = "REGIONAL" } resource "google_compute_subnetwork" "solana_etl" { ip_cidr_range = var.cird_range - name = var.project_id + name = "${var.project_id}-solala-bq-subnet" network = google_compute_network.solana_etl.id private_ip_google_access = true private_ipv6_google_access = "DISABLE_GOOGLE_ACCESS" @@ -24,7 +24,7 @@ resource "google_compute_address" "solana_rpc_public" { region = var.region } resource "google_compute_address" "solana_rpc_internal" { - address = "10.0.0.10" + address = "10.0.0.12" address_type = "INTERNAL" name = "solana-rpc-internal" project = var.project_id @@ -32,18 +32,28 @@ resource "google_compute_address" "solana_rpc_internal" { subnetwork = google_compute_subnetwork.solana_etl.name } -resource "google_compute_address" "rabbit_mq_public" { - address_type = "EXTERNAL" - name = "rabbitmq-public" - network_tier = "STANDARD" +resource "google_compute_address" "rabbit_mq_internal" { + address = "10.0.0.3" + address_type = "INTERNAL" + name = "rabbit-mq-internal" project = var.project_id region = var.region + subnetwork = google_compute_subnetwork.solana_etl.name } -resource "google_compute_address" "rabbit_mq_internal" { - address = "10.0.0.20" +resource "google_compute_address" "inserter_internal" { + address = "10.0.0.4" address_type = "INTERNAL" - name = "rabbit-mq-internal" + name = "inserter-internal" + project = var.project_id + region = var.region + subnetwork = google_compute_subnetwork.solana_etl.name +} + +resource "google_compute_address" "indexer_internal" { + address = "10.0.0.5" + address_type = "INTERNAL" + name = "indexer-internal" project = var.project_id region = var.region subnetwork = google_compute_subnetwork.solana_etl.name diff --git a/iac/main/provider.tf b/iac/main/provider.tf index f036911..9cd359e 100644 --- a/iac/main/provider.tf +++ b/iac/main/provider.tf @@ -3,9 +3,13 @@ terraform { required_providers { google = { source = "hashicorp/google" - version = "~> 5.2" + version = "~> 5.40.0" } } + backend "gcs" { + bucket = + prefix = "solana-bq/bq" + } } provider "google" { @@ -19,5 +23,3 @@ provider "google-beta" { data "google_client_config" "this" {} data "google_project" "this" {} - - diff --git a/iac/main/schemas/Accounts.json b/iac/main/schemas/Accounts.json new file mode 100644 index 0000000..85b218a --- /dev/null +++ b/iac/main/schemas/Accounts.json @@ -0,0 +1,229 @@ +[ + { + "name": "block_slot", + "type": "INTEGER", + "mode": "REQUIRED" + }, + { + "name": "block_timestamp", + "type": "TIMESTAMP", + "mode": "NULLABLE" + }, + { + "name": "block_hash", + "type": "STRING", + "mode": "REQUIRED" + }, + { + "name": "pubkey", + "type": "STRING", + "mode": "REQUIRED" + }, + { + "name": "tx_signature", + "type": "STRING", + "mode": "REQUIRED" + }, + { + "name": "retrieval_timestamp", + "type": "TIMESTAMP", + "mode": "REQUIRED" + }, + { + "name": "executable", + "type": "BOOLEAN", + "mode": "NULLABLE" + }, + { + "name": "lamports", + "type": "NUMERIC", + "mode": "NULLABLE" + }, + { + "name": "owner", + "type": "STRING", + "mode": "NULLABLE" + }, + { + "name": "rent_epoch", + "type": "INTEGER", + "mode": "NULLABLE" + }, + { + "name": "program", + "type": "STRING", + "mode": "NULLABLE" + }, + { + "name": "space", + "type": "INTEGER", + "mode": "NULLABLE" + }, + { + "name": "account_type", + "type": "STRING", + "mode": "NULLABLE" + }, + { + "name": "is_native", + "type": "BOOLEAN", + "mode": "NULLABLE" + }, + { + "name": "mint", + "type": "STRING", + "mode": "NULLABLE" + }, + { + "name": "state", + "type": "STRING", + "mode": "NULLABLE" + }, + { + "name": "token_amount", + "type": "NUMERIC", + "mode": "NULLABLE" + }, + { + "name": "token_amount_decimals", + "type": "INTEGER", + "mode": "NULLABLE" + }, + { + "name": "program_data", + "type": "STRING", + "mode": "NULLABLE" + }, + { + "name": "authorized_voters", + "type": "RECORD", + "mode": "REPEATED", + "fields": [ + { + "name": "authorized_voter", + "type": "STRING", + "mode": "NULLABLE" + }, + { + "name": "epoch", + "type": "INTEGER", + "mode": "NULLABLE" + } + ] + }, + { + "name": "authorized_withdrawer", + "type": "STRING", + "mode": "NULLABLE" + }, + { + "name": "prior_voters", + "type": "RECORD", + "mode": "REPEATED", + "fields": [ + { + "name": "authorized_pubkey", + "type": "STRING", + "mode": "NULLABLE" + }, + { + "name": "epoch_of_last_authorized_switch", + "type": "INTEGER", + "mode": "NULLABLE" + }, + { + "name": "target_epoch", + "type": "INTEGER", + "mode": "NULLABLE" + } + ] + }, + { + "name": "node_pubkey", + "type": "STRING", + "mode": "NULLABLE" + }, + { + "name": "commission", + "type": "INTEGER", + "mode": "NULLABLE" + }, + { + "name": "epoch_credits", + "type": "RECORD", + "mode": "REPEATED", + "fields": [ + { + "name": "credits", + "type": "STRING", + "mode": "NULLABLE" + }, + { + "name": "epoch", + "type": "INTEGER", + "mode": "NULLABLE" + }, + { + "name": "previous_credits", + "type": "STRING", + "mode": "NULLABLE" + } + ] + }, + { + "name": "votes", + "type": "RECORD", + "mode": "REPEATED", + "fields": [ + { + "name": "confirmation_count", + "type": "INTEGER", + "mode": "NULLABLE" + }, + { + "name": "slot", + "type": "INTEGER", + "mode": "NULLABLE" + } + ] + }, + { + "name": "root_slot", + "type": "INTEGER", + "mode": "NULLABLE" + }, + { + "name": "last_timestamp", + "type": "RECORD", + "mode": "REPEATED", + "fields": [ + { + "name": "slot", + "type": "INTEGER", + "mode": "NULLABLE" + }, + { + "name": "timestamp", + "type": "TIMESTAMP", + "mode": "NULLABLE" + } + ] + }, + { + "name": "data", + "type": "RECORD", + "mode": "REPEATED", + "fields": [ + { + "name": "raw", + "type": "STRING", + "mode": "NULLABLE" + }, + { + "name": "encoding", + "type": "STRING", + "mode": "NULLABLE" + } + ] + } +] \ No newline at end of file diff --git a/iac/main/schemas/Block Rewards.json b/iac/main/schemas/Block Rewards.json new file mode 100644 index 0000000..36ea667 --- /dev/null +++ b/iac/main/schemas/Block Rewards.json @@ -0,0 +1,50 @@ +[ + { + "name": "block_slot", + "type": "INTEGER", + "mode": "REQUIRED", + "description": "The block's slot" + }, + { + "name": "block_hash", + "type": "STRING", + "mode": "REQUIRED", + "description": "The block's hash" + }, + { + "name": "block_timestamp", + "type": "TIMESTAMP", + "mode": "NULLABLE", + "description": "The block's timestamp" + }, + { + "name": "commission", + "type": "NUMERIC", + "mode": "NULLABLE", + "description": "The fee that the validator collected from staking delegators" + }, + { + "name": "lamports", + "type": "NUMERIC", + "mode": "NULLABLE", + "description": "The amount of reward SOL, in Lamports" + }, + { + "name": "post_balance", + "type": "NUMERIC", + "mode": "NULLABLE", + "description": "The account balance, in Lamports, after receiving the reward" + }, + { + "name": "pubkey", + "type": "STRING", + "mode": "NULLABLE", + "description": "The public key of the account that received the reward" + }, + { + "name": "reward_type", + "type": "STRING", + "mode": "NULLABLE", + "description": "The type of reward, such as fee, rent, voting, or staking" + } +] diff --git a/iac/main/schemas/Blocks.json b/iac/main/schemas/Blocks.json new file mode 100644 index 0000000..75261d1 --- /dev/null +++ b/iac/main/schemas/Blocks.json @@ -0,0 +1,50 @@ +[ + { + "name": "slot", + "type": "INTEGER", + "mode": "REQUIRED", + "description": "The block's slot" + }, + { + "name": "block_hash", + "type": "STRING", + "mode": "REQUIRED", + "description": "The block's hash" + }, + { + "name": "block_timestamp", + "type": "TIMESTAMP", + "mode": "NULLABLE", + "description": "The block's timestamp" + }, + { + "name": "height", + "type": "INTEGER", + "mode": "NULLABLE", + "description": "The block height" + }, + { + "name": "previous_block_hash", + "type": "STRING", + "mode": "NULLABLE", + "description": "The hash of the previous block" + }, + { + "name": "transaction_count", + "type": "INTEGER", + "mode": "NULLABLE", + "description": "The number of transactions in the block" + }, + { + "name": "leader_reward", + "type": "NUMERIC", + "mode": "NULLABLE", + "description": "The amount of rewarded SOL, in Lamports, of the leading validator" + }, + { + "name": "leader", + "type": "STRING", + "mode": "NULLABLE", + "description": "The leading validator account" + } +] diff --git a/iac/main/schemas/Instructions.json b/iac/main/schemas/Instructions.json new file mode 100644 index 0000000..907a91e --- /dev/null +++ b/iac/main/schemas/Instructions.json @@ -0,0 +1,93 @@ +[ + { + "name": "block_slot", + "type": "INTEGER", + "mode": "REQUIRED", + "description": "The block's slot" + }, + { + "name": "block_timestamp", + "type": "TIMESTAMP", + "mode": "NULLABLE", + "description": "The block's timestamp" + }, + { + "name": "block_hash", + "type": "STRING", + "mode": "NULLABLE", + "description": "The block's hash" + }, + { + "name": "tx_signature", + "type": "STRING", + "mode": "REQUIRED", + "description": "The signature of the transaction that this instruction came from" + }, + { + "name": "index", + "type": "INTEGER", + "mode": "NULLABLE", + "description": "The position of this instruction within all of instructions" + }, + { + "name": "parent_index", + "type": "INTEGER", + "mode": "NULLABLE", + "description": "The position of the parent instruction with all of the parent instructions. Only present for the inner instructions" + }, + { + "name": "accounts", + "type": "STRING", + "mode": "REPEATED", + "description": "The accounts involved in the transaction that this instruction came from" + }, + { + "name": "data", + "type": "STRING", + "mode": "NULLABLE", + "description": "The instruction's raw data. Only present if the data source did not have a parser for the program" + }, + { + "name": "parsed", + "type": "STRING", + "mode": "NULLABLE", + "description": "The instruction's parsed data. Only present if the data source had a parser for the program." + }, + { + "name": "program", + "type": "STRING", + "mode": "NULLABLE", + "description": "The associated program that executes the instruction" + }, + { + "name": "program_id", + "type": "STRING", + "mode": "NULLABLE", + "description": "The account of the program that executes the instruction" + }, + { + "name": "instruction_type", + "type": "STRING", + "mode": "NULLABLE", + "description": "The type of the instruction" + }, + { + "name": "params", + "type": "RECORD", + "mode": "REPEATED", + "fields": [ + { + "name": "key", + "type": "STRING", + "mode": "NULLABLE", + "description": "The identifier of the instruction parameter" + }, + { + "name": "value", + "type": "STRING", + "mode": "NULLABLE", + "description": "The instruction parameter value" + } + ] + } +] diff --git a/iac/main/schemas/Token Transfers.json b/iac/main/schemas/Token Transfers.json new file mode 100644 index 0000000..68efaee --- /dev/null +++ b/iac/main/schemas/Token Transfers.json @@ -0,0 +1,77 @@ +[ + { + "name": "block_slot", + "type": "INTEGER", + "mode": "REQUIRED" + }, + { + "name": "block_timestamp", + "type": "TIMESTAMP", + "mode": "NULLABLE" + }, + { + "name": "block_hash", + "type": "STRING", + "mode": "REQUIRED" + }, + { + "name": "tx_signature", + "type": "STRING", + "mode": "REQUIRED" + }, + { + "name": "source", + "type": "STRING", + "mode": "NULLABLE" + }, + { + "name": "destination", + "type": "STRING", + "mode": "NULLABLE" + }, + { + "name": "authority", + "type": "STRING", + "mode": "NULLABLE" + }, + { + "name": "value", + "type": "NUMERIC", + "mode": "NULLABLE" + }, + { + "name": "decimals", + "type": "NUMERIC", + "mode": "NULLABLE" + }, + { + "name": "mint", + "type": "STRING", + "mode": "NULLABLE" + }, + { + "name": "mint_authority", + "type": "STRING", + "mode": "NULLABLE" + }, + { + "name": "fee", + "type": "NUMERIC", + "mode": "NULLABLE" + }, + { + "name": "fee_decimals", + "type": "NUMERIC", + "mode": "NULLABLE" + }, + { + "name": "memo", + "type": "STRING", + "mode": "NULLABLE" + }, + { + "name": "transfer_type", + "type": "STRING", + "mode": "NULLABLE" + } +] \ No newline at end of file diff --git a/iac/main/schemas/Tokens.json b/iac/main/schemas/Tokens.json new file mode 100644 index 0000000..b9f4cbd --- /dev/null +++ b/iac/main/schemas/Tokens.json @@ -0,0 +1,94 @@ +[ + { + "name": "block_slot", + "type": "INTEGER", + "mode": "NULLABLE" + }, + { + "name": "block_timestamp", + "type": "TIMESTAMP", + "mode": "NULLABLE" + }, + { + "name": "block_hash", + "type": "STRING", + "mode": "NULLABLE" + }, + { + "name": "tx_signature", + "type": "STRING", + "mode": "NULLABLE" + }, + { + "name": "retrieval_timestamp", + "type": "TIMESTAMP", + "mode": "REQUIRED" + }, + { + "name": "is_nft", + "type": "BOOLEAN", + "mode": "NULLABLE" + }, + { + "name": "mint", + "type": "STRING", + "mode": "NULLABLE" + }, + { + "name": "update_authority", + "type": "STRING", + "mode": "NULLABLE" + }, + { + "name": "name", + "type": "STRING", + "mode": "NULLABLE" + }, + { + "name": "symbol", + "type": "STRING", + "mode": "NULLABLE" + }, + { + "name": "uri", + "type": "STRING", + "mode": "NULLABLE" + }, + { + "name": "seller_fee_basis_points", + "type": "NUMERIC", + "mode": "NULLABLE" + }, + { + "name": "creators", + "type": "RECORD", + "mode": "REPEATED", + "fields": [ + { + "name": "address", + "type": "STRING", + "mode": "NULLABLE" + }, + { + "name": "verified", + "type": "BOOLEAN", + "mode": "NULLABLE" + }, + { + "name": "share", + "type": "INTEGER", + "mode": "NULLABLE" + } + ] + }, + { + "name": "primary_sale_happened", + "type": "BOOLEAN", + "mode": "NULLABLE" + }, + { + "name": "is_mutable", + "type": "BOOLEAN", + "mode": "NULLABLE" + } +] diff --git a/iac/main/schemas/Transactions.json b/iac/main/schemas/Transactions.json new file mode 100644 index 0000000..013dfdc --- /dev/null +++ b/iac/main/schemas/Transactions.json @@ -0,0 +1,165 @@ +[ + { + "name": "block_slot", + "type": "INTEGER", + "mode": "NULLABLE" + }, + { + "name": "block_hash", + "type": "STRING", + "mode": "NULLABLE" + }, + { + "name": "block_timestamp", + "type": "TIMESTAMP", + "mode": "NULLABLE" + }, + { + "name": "recent_block_hash", + "type": "STRING", + "mode": "NULLABLE" + }, + { + "name": "signature", + "type": "STRING", + "mode": "NULLABLE" + }, + { + "name": "index", + "type": "INTEGER", + "mode": "NULLABLE" + }, + { + "name": "fee", + "type": "NUMERIC", + "mode": "NULLABLE" + }, + { + "name": "status", + "type": "STRING", + "mode": "NULLABLE" + }, + { + "name": "err", + "type": "STRING", + "mode": "NULLABLE" + }, + { + "name": "compute_units_consumed", + "type": "NUMERIC", + "mode": "NULLABLE" + }, + { + "name": "accounts", + "type": "RECORD", + "mode": "REPEATED", + "fields": [ + { + "name": "pubkey", + "type": "STRING", + "mode": "NULLABLE" + }, + { + "name": "signer", + "type": "BOOLEAN", + "mode": "NULLABLE" + }, + { + "name": "writable", + "type": "BOOLEAN", + "mode": "NULLABLE" + } + ] + }, + { + "name": "log_messages", + "type": "STRING", + "mode": "REPEATED" + }, + { + "name": "balance_changes", + "type": "RECORD", + "mode": "REPEATED", + "fields": [ + { + "name": "account", + "type": "STRING", + "mode": "NULLABLE" + }, + { + "name": "before", + "type": "NUMERIC", + "mode": "NULLABLE" + }, + { + "name": "after", + "type": "NUMERIC", + "mode": "NULLABLE" + } + ] + }, + { + "name": "pre_token_balances", + "type": "RECORD", + "mode": "REPEATED", + "fields": [ + { + "name": "account_index", + "type": "INTEGER", + "mode": "NULLABLE" + }, + { + "name": "mint", + "type": "STRING", + "mode": "NULLABLE" + }, + { + "name": "owner", + "type": "STRING", + "mode": "NULLABLE" + }, + { + "name": "amount", + "type": "BIGNUMERIC", + "mode": "NULLABLE" + }, + { + "name": "decimals", + "type": "INTEGER", + "mode": "NULLABLE" + } + ] + }, + { + "name": "post_token_balances", + "type": "RECORD", + "mode": "REPEATED", + "fields": [ + { + "name": "account_index", + "type": "INTEGER", + "mode": "NULLABLE" + }, + { + "name": "mint", + "type": "STRING", + "mode": "NULLABLE" + }, + { + "name": "owner", + "type": "STRING", + "mode": "NULLABLE" + }, + { + "name": "amount", + "type": "BIGNUMERIC", + "mode": "NULLABLE" + }, + { + "name": "decimals", + "type": "INTEGER", + "mode": "NULLABLE" + } + ] + } +] diff --git a/iac/main/variables.tf b/iac/main/variables.tf index 499d25c..f06070f 100644 --- a/iac/main/variables.tf +++ b/iac/main/variables.tf @@ -5,17 +5,23 @@ locals { "https://www.googleapis.com/auth/monitoring.write", "https://www.googleapis.com/auth/service.management.readonly", "https://www.googleapis.com/auth/servicecontrol", - "https://www.googleapis.com/auth/trace.append" + "https://www.googleapis.com/auth/trace.append", + "https://www.googleapis.com/auth/cloud-platform" ] } variable "project_id" { - default = "PROJECT-408009" + default = } variable "region" { description = "The zone for the instance" - default = "us-east4" + default = "us-central1" } variable "cird_range" { default = "10.0.0.0/24" } + +variable "app_vm_size" { + description = "The VM size for the applications" + default = "n2d-standard-16" +} diff --git a/iac/scripts/indexer/indexer-service.sh b/iac/scripts/indexer/indexer-service.sh new file mode 100644 index 0000000..59ef612 --- /dev/null +++ b/iac/scripts/indexer/indexer-service.sh @@ -0,0 +1,101 @@ +#!/bin/bash + +# Define variables +VERSION="v1.0.1" +SERVICE_NAME="blockchain_etl_indexer" +RELEASE_URL="https://github.com/BCWResearch/solana-etl/releases/download/extractor.$VERSION/$SERVICE_NAME" +INSTALL_DIR="/var/etl_files" +SERVICE_FILE_PATH="/etc/systemd/system/${SERVICE_NAME}.service" +USER_NAME="etl_user" +GROUP_NAME="etl_user" +ENV_FILE_PATH="${INSTALL_DIR}/.env" +SLOTVALUE="282024813" # Define the slot value variable + +# Create the user and group if they do not exist +if ! id "$USER_NAME" &>/dev/null; then + echo "Creating user $USER_NAME..." + sudo useradd -r -s /bin/false "$USER_NAME" +fi + +if ! getent group "$GROUP_NAME" &>/dev/null; then + echo "Creating group $GROUP_NAME..." + sudo groupadd "$GROUP_NAME" +fi + +# Add the user to the group +echo "Adding user $USER_NAME to group $GROUP_NAME..." +sudo usermod -a -G "$GROUP_NAME" "$USER_NAME" + +# Create the installation directory if it does not exist +if [ ! -d "$INSTALL_DIR" ]; then + echo "Creating installation directory $INSTALL_DIR..." + sudo mkdir -p "$INSTALL_DIR" + sudo chown "$USER_NAME:$GROUP_NAME" "$INSTALL_DIR" +fi + +# Download the binary file +echo "Downloading binary version $VERSION..." +sudo curl -L -o "${INSTALL_DIR}/${SERVICE_NAME}" "$RELEASE_URL" + +# Make the binary executable +echo "Setting executable permissions for the binary..." +sudo chmod +x "${INSTALL_DIR}/${SERVICE_NAME}" + +# Define the .env file content +ENV_FILE_CONTENT="RABBITMQ_USER=jb +RABBITMQ_PASSWORD=jb +RABBITMQ_PORT=5672 +RABBITMQ_ADDRESS=10.0.0.3 +NUM_EXTRACTOR_THREADS=20 +ENABLE_METRICS=true +METRICS_PORT=4000 +METRICS_ADDRESS=127.0.0.1 +RPC_METHOD_TIMEOUT=30 +ENDPOINT=http://10.0.0.12/rpc +QUEUE_NAME=\"solana-etl\" +" + +# Create the .env file +echo "Creating .env file at ${ENV_FILE_PATH}..." +echo "$ENV_FILE_CONTENT" | sudo tee $ENV_FILE_PATH > /dev/null + +# Define the service file content +SERVICE_CONTENT="[Unit] +Description=Blockchain-ETL indexer +After=network.target + +[Service] +ExecStart=${INSTALL_DIR}/${SERVICE_NAME} index-range stream $SLOTVALUE +WorkingDirectory=${INSTALL_DIR} +Restart=no +User=${USER_NAME} +Group=${GROUP_NAME} +EnvironmentFile=${ENV_FILE_PATH} +Environment=\"RUST_LOG=warn\" +ExecStop=/bin/kill -2 \$MAINPID +TimeoutStopSec=1800s + +[Install] +WantedBy=multi-user.target +" + +# Create the service file +echo "Creating service file at ${SERVICE_FILE_PATH}..." +echo "$SERVICE_CONTENT" | sudo tee $SERVICE_FILE_PATH > /dev/null + +# Set correct permissions for the service file +sudo chmod 644 $SERVICE_FILE_PATH + +# Reload systemd configuration +echo "Reloading systemd configuration..." +sudo systemctl daemon-reload + +# Enable the service to start on boot +echo "Enabling the service..." +sudo systemctl enable "${SERVICE_NAME}.service" + +# Optionally, start the service immediately +echo "Starting the service..." +sudo systemctl start "${SERVICE_NAME}.service" + +echo "User, group, binary, .env file, and service setup complete." diff --git a/iac/scripts/inserter/inserter-service.sh b/iac/scripts/inserter/inserter-service.sh new file mode 100644 index 0000000..fb326d8 --- /dev/null +++ b/iac/scripts/inserter/inserter-service.sh @@ -0,0 +1,82 @@ +#!/bin/bash + +# Define variables +VERSION="v1.0.0" +SERVICE_FILE_PATH="/etc/systemd/system/solana_etl_inserter.service" +SERVICE_USER="etl_user" +SERVICE_GROUP="etl_user" +SERVICE_WORKING_DIRECTORY="/var/solana_etl_inserter" +SERVICE_EXEC_START="${SERVICE_WORKING_DIRECTORY}/main myFirstConsumer" +SERVICE_ENVIRONMENT="GOOGLE_APPLICATION_CREDENTIALS=${SERVICE_WORKING_DIRECTORY}/solana-bq.json" +BINARY_URL="https://github.com/BCWResearch/solana-etl/releases/download/inserter.${VERSION}/blockchain_etl_inserter" +ENV_FILE_PATH="${SERVICE_WORKING_DIRECTORY}/.env" + +# Check if the script is being run as root +if [ "$EUID" -ne 0 ]; then + echo "Please run as root." + exit 1 +fi + +# Create the etl_user and group if they do not exist +if ! id -u $SERVICE_USER > /dev/null 2>&1; then + groupadd $SERVICE_GROUP + useradd -g $SERVICE_GROUP $SERVICE_USER +fi + +# Create the working directory if it doesn't exist +mkdir -p $SERVICE_WORKING_DIRECTORY + +# Download the binary +curl -L $BINARY_URL -o ${SERVICE_WORKING_DIRECTORY}/blockchain_etl_inserter + +# Rename the binary to main +mv ${SERVICE_WORKING_DIRECTORY}/blockchain_etl_inserter ${SERVICE_WORKING_DIRECTORY}/main + +# Make the binary executable +chmod +x ${SERVICE_WORKING_DIRECTORY}/main + +# Create the .env file +cat < $ENV_FILE_PATH +QUEUE_NAME="solana-etl" +BQ_PROJECT_ID="bcwt-sandbox" +BQ_DATASET_ID="crypto_solana_mainnet_us" +RABBITMQ_USER="jb" +RABBITMQ_PASS="jb" +RABBITMQ_HOST="10.0.0.3" +RABBITMQ_PORT="5672" +EOF + +# Change ownership of the working directory to the service user and group +chown -R $SERVICE_USER:$SERVICE_GROUP $SERVICE_WORKING_DIRECTORY + +# Create the systemd service file +cat < $SERVICE_FILE_PATH +[Unit] +Description=Solana ETL Inserter +After=network.target + +[Service] +ExecStart=$SERVICE_EXEC_START +WorkingDirectory=$SERVICE_WORKING_DIRECTORY +Restart=always +RestartSec=5 +User=$SERVICE_USER +Group=$SERVICE_GROUP +#Environment="$SERVICE_ENVIRONMENT" +EnvironmentFile=$ENV_FILE_PATH +ExecStop=/bin/kill -2 \$MAINPID + +[Install] +WantedBy=multi-user.target +EOF + +# Reload the systemd daemon to apply the new service +systemctl daemon-reload + +# Enable the service to start on boot +systemctl enable solana_etl_inserter.service + +# Start the service +systemctl start solana_etl_inserter.service + +echo "Systemd service for Solana ETL Inserter created and started successfully." \ No newline at end of file diff --git a/iac/scripts/rabbitmq/run-rabbit.sh b/iac/scripts/rabbit-mq/run-rabbit.sh similarity index 87% rename from iac/scripts/rabbitmq/run-rabbit.sh rename to iac/scripts/rabbit-mq/run-rabbit.sh index 143e49c..c717bd0 100644 --- a/iac/scripts/rabbitmq/run-rabbit.sh +++ b/iac/scripts/rabbit-mq/run-rabbit.sh @@ -59,14 +59,19 @@ enable_management_plugin() { # Configure RabbitMQ configure_rabbitmq() { log_message "Configuring RabbitMQ" - read -p "Enter RabbitMQ user: " RABBITMQ_USER - read -sp "Enter RabbitMQ password: " RABBITMQ_PASSWORD - echo - read -p "Enter RabbitMQ vhost: " RABBITMQ_VHOST - read -p "Enter RabbitMQ queue name: " RABBITMQ_QUEUE + # read -p "Enter RabbitMQ user: " RABBITMQ_USER + # read -sp "Enter RabbitMQ password: " RABBITMQ_PASSWORD + # echo + # read -p "Enter RabbitMQ vhost: " RABBITMQ_VHOST + # read -p "Enter RabbitMQ queue name: " RABBITMQ_QUEUE + RABBITMQ_USER="jb" + RABBITMQ_PASSWORD="jb" + RABBITMQ_VHOST="/" + RABBITMQ_QUEUE="solana-etl" rabbitmqctl add_vhost $RABBITMQ_VHOST || handle_error "Failed to add vhost $RABBITMQ_VHOST" rabbitmqctl add_user $RABBITMQ_USER $RABBITMQ_PASSWORD || handle_error "Failed to add user $RABBITMQ_USER" + rabbitmqctl set_user_tags $RABBITMQ_USER administrator || handle_error "Failed to set user tags for $RABBITMQ_USER" rabbitmqctl set_permissions -p $RABBITMQ_VHOST $RABBITMQ_USER "." "." ".*" || handle_error "Failed to set permissions for user $RABBITMQ_USER" rabbitmqadmin declare queue --vhost=$RABBITMQ_VHOST name=$RABBITMQ_QUEUE durable=true || handle_error "Failed to declare queue $RABBITMQ_QUEUE" } diff --git a/iac/scripts/solana-rpc/configure-solana-disks.sh b/iac/scripts/solana-rpc/configure-solana-disks.sh index 117dcb3..b9d15dc 100644 --- a/iac/scripts/solana-rpc/configure-solana-disks.sh +++ b/iac/scripts/solana-rpc/configure-solana-disks.sh @@ -55,6 +55,25 @@ update_fstab() { echo "UUID=$uuid $mount_point ext4 discard,defaults,nofail 0 2" >> /etc/fstab || handle_error $? } +# Create user and password +create_user() { + local username="$1" + local password="$2" + + if [ -z "$username" ] || [ -z "$password" ]; then + log_message "Username and password are required" + exit 1 + fi + + log_message "Creating user '$username'" + useradd -m -s /bin/bash "$username" || handle_error "Failed to create user '$username'" + echo "$username:$password" | chpasswd || handle_error "Failed to set password for user '$username'" + # Add user 'sol' to 'sudo' group + log_message "Adding user 'sol' to 'sudo' group" + usermod -aG sudo $username || handle_error "Failed to add user '$username' to 'sudo' group" +} + + # Function to set permissions and ownership set_permissions() { local directory="$1" @@ -92,6 +111,7 @@ update_fstab /dev/md0 /solana/ledger update_fstab /dev/md1 /solana/rest # Set permissions and ownership +create_user sol sol set_permissions /solana/ledger sol sol set_permissions /solana/rest sol sol diff --git a/iac/scripts/solana-rpc/solana-install.sh b/iac/scripts/solana-rpc/solana-install.sh index 2f21594..a582547 100644 --- a/iac/scripts/solana-rpc/solana-install.sh +++ b/iac/scripts/solana-rpc/solana-install.sh @@ -38,13 +38,50 @@ create_user_sol() { install_solana() { if ! su - sol -c 'solana --version' &>/dev/null; then log_message "Installing Solana for user sol" - su - sol -c 'sh -c "$(curl -sSfL https://release.solana.com/v1.17.9/install)"' || handle_error "Solana installation failed" + su - sol -c 'sh -c "$(curl -sSfL https://release.solana.com/v1.18.18/install)"' || handle_error "Solana installation failed" echo 'export PATH=$PATH:/home/sol/.local/share/solana/install/active_release/bin' >> /home/sol/.bashrc || handle_error "Failed to set PATH for Solana" else log_message "Solana is already installed for user sol" fi } +# Create and configure tmpfs and swap +configure_tmpfs_and_swap() { + log_message "Configuring tmpfs and swap" + + # Create the directory for Solana accounts + mkdir -p /mnt/solana-accounts || handle_error "Failed to create /mnt/solana-accounts directory" + + # Add tmpfs entry to /etc/fstab + if ! grep -q '/mnt/solana-accounts' /etc/fstab; then + echo 'tmpfs /mnt/solana-accounts tmpfs rw,size=400G,user=sol 0 0' >> /etc/fstab || handle_error "Failed to add tmpfs entry to /etc/fstab" + else + log_message "tmpfs entry already exists in /etc/fstab" + fi + + # Create swap file + if ! grep -q '/swapfile' /etc/fstab; then + log_message "Creating swap file" + dd if=/dev/zero of=/swapfile bs=1MiB count=250KiB || handle_error "Failed to create swap file" + chmod 0600 /swapfile || handle_error "Failed to set permissions for swap file" + mkswap /swapfile || handle_error "Failed to format swap file" + echo '/swapfile swap swap defaults 0 0' >> /etc/fstab || handle_error "Failed to add swap file entry to /etc/fstab" + else + log_message "Swap file entry already exists in /etc/fstab" + fi + + # Enable swap + swapon -a || handle_error "Failed to enable swap" + + # Mount tmpfs + mount /mnt/solana-accounts || handle_error "Failed to mount tmpfs" + + # Confirm swap is active and tmpfs is mounted + free -g || handle_error "Failed to confirm swap" + mount | grep tmpfs || handle_error "Failed to confirm tmpfs" +} + +# Create user sol log directory create_sol_log_directory() { local sol_log_dir="/home/sol/log" if [ ! -d "$sol_log_dir" ]; then @@ -56,6 +93,7 @@ create_sol_log_directory() { fi } +# Configure log rotation configure_log_rotation() { local logrotate_conf="/etc/logrotate.d/solana-validator" @@ -87,7 +125,6 @@ create_validator_identity() { fi } - # Create Validator Start Script create_validator_script() { if [ ! -f /home/sol/rpc-start.sh ]; then @@ -100,8 +137,12 @@ set -o errexit # Validator start commands solana-validator \ - --entrypoint entrypoint.mainnet-beta.solana.com:8001\ - --known-validator 7Np41oeYqPefeNQEHSv1UDhYrehxin3NStELsSKCT4K2\ + --entrypoint entrypoint.mainnet-beta.solana.com:8001 \ + --entrypoint entrypoint2.mainnet-beta.solana.com:8001 \ + --entrypoint entrypoint3.mainnet-beta.solana.com:8001 \ + --entrypoint entrypoint4.mainnet-beta.solana.com:8001 \ + --entrypoint entrypoint5.mainnet-beta.solana.com:8001 \ + --known-validator 7Np41oeYqPefeNQEHSv1UDhYrehxin3NStELsSKCT4K2 \ --known-validator GdnSyH3YtwcxFvQrVVJMm1JhTS4QVX7MFsX56uJLUfiZ \ --known-validator DE1bawNcRJB9rVm3buyMVfr8mBEoyyu73NBovf2oXJsJ \ --known-validator CakcnaRDHka2gXyfbEd2d3xsvkJkqsLw2akB3zsN1D2S \ @@ -113,7 +154,7 @@ solana-validator \ --no-voting \ --no-wait-for-vote-to-start-leader \ --accounts /mnt/solana-accounts \ - --ledger /solana/ledger \ + --ledger /solana/ledger \ --limit-ledger-size 100000000 \ --snapshot-interval-slots 5000 \ --maximum-local-snapshot-age 500 \ @@ -122,10 +163,10 @@ solana-validator \ --enable-cpi-and-log-storage \ --full-rpc-api \ --private-rpc \ - --snapshots /solana/rest\ - --accounts-index-path /solana/data/rest\ - --accounts-hash-cache-path /solana/rest/cache\ - --log ~/log/validator.log + --snapshots /solana/rest \ + --accounts-index-path /solana/rest/indexes \ + --accounts-hash-cache-path /solana/rest/cache \ + --log /home/sol/solana-rpc.log EOF chmod +x /home/sol/rpc-start.sh || handle_error "Failed to create validator start script" chown sol:sol /home/sol/rpc-start.sh @@ -134,6 +175,46 @@ EOF fi } +# Optimize sysctl knobs +optimize_sysctl() { + log_message "Optimizing sysctl knobs" + cat > /etc/sysctl.d/21-solana-validator.conf <> /etc/systemd/system.conf || handle_error "Failed to set DefaultLimitNOFILE in /etc/systemd/system.conf" + fi + + # Reload systemd configuration + systemctl daemon-reload || handle_error "Failed to reload systemd daemon" + + # Set session file limits + cat > /etc/security/limits.d/90-solana-nofiles.conf < /etc/nginx/sites-available/rpc.conf < Date: Wed, 7 Aug 2024 12:43:40 +0400 Subject: [PATCH 2/5] Update main.tf --- iac/init/main.tf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/iac/init/main.tf b/iac/init/main.tf index d7221db..5446dce 100644 --- a/iac/init/main.tf +++ b/iac/init/main.tf @@ -18,7 +18,7 @@ variable "region" { } variable "project" { - default = "bcwt-sandbox" + default = "project-name" } locals { From 1cdcbebb747a3f4e0c365cf78b586e8f8efef490 Mon Sep 17 00:00:00 2001 From: Toko Machavariani <109960534+tornikebcw@users.noreply.github.com> Date: Wed, 7 Aug 2024 12:46:44 +0400 Subject: [PATCH 3/5] Delete leftovers --- iac/main/terraform.tfstate | 829 ------------------------------------- 1 file changed, 829 deletions(-) delete mode 100644 iac/main/terraform.tfstate diff --git a/iac/main/terraform.tfstate b/iac/main/terraform.tfstate deleted file mode 100644 index afcdd93..0000000 --- a/iac/main/terraform.tfstate +++ /dev/null @@ -1,829 +0,0 @@ -{ - "version": 4, - "terraform_version": "1.5.7", - "serial": 83, - "lineage": "f8045795-339b-f672-8d7a-d12150904cb3", - "outputs": {}, - "resources": [ - { - "mode": "data", - "type": "google_client_config", - "name": "this", - "provider": "provider[\"registry.terraform.io/hashicorp/google\"]", - "instances": [ - { - "schema_version": 0, - "attributes": { - "access_token": "ya29.a0AfB_byB3R1g4CaW253C6y1zionzpnDyWjPJ86NfzdMlp5DAmxdU4ZyM3RidaADSU8tzDF6ziXj_4XQWpnn_o60YpneaGclHAhK5_1Xde7CL4oDeu5yN_pQXQHrDksk2MaHn806OrNRj_D6M7HJ9XKGvwSG3pinmu1UtyP6AaCgYKAfMSARISFQHGX2MiXuewosVChBmZXU3WVrBeFg0174", - "id": "projects/\"PROJECT-408009\"/regions/\"us-east4\"/zones/\u003cnull\u003e", - "project": "PROJECT-408009", - "region": "us-east4", - "zone": null - }, - "sensitive_attributes": [] - } - ] - }, - { - "mode": "data", - "type": "google_project", - "name": "this", - "provider": "provider[\"registry.terraform.io/hashicorp/google\"]", - "instances": [ - { - "schema_version": 0, - "attributes": { - "auto_create_network": null, - "billing_account": "017876-C35189-870678", - "effective_labels": {}, - "folder_id": "", - "id": "projects/PROJECT-408009", - "labels": {}, - "name": "PROJECT", - "number": "188431643889", - "org_id": "186593810310", - "project_id": "PROJECT-408009", - "skip_delete": null, - "terraform_labels": {} - }, - "sensitive_attributes": [] - } - ] - }, - { - "mode": "managed", - "type": "google_compute_address", - "name": "rabbitmq_internal", - "provider": "provider[\"registry.terraform.io/hashicorp/google\"]", - "instances": [ - { - "schema_version": 0, - "attributes": { - "address": "10.0.0.20", - "address_type": "INTERNAL", - "creation_timestamp": "2023-12-13T02:04:38.075-08:00", - "description": "", - "effective_labels": {}, - "id": "projects/PROJECT-408009/regions/us-east4/addresses/rabbitmq-internal", - "ip_version": "", - "ipv6_endpoint_type": "", - "label_fingerprint": "42WmSpB8rSM=", - "labels": {}, - "name": "rabbitmq-internal", - "network": "", - "network_tier": "PREMIUM", - "prefix_length": 0, - "project": "PROJECT-408009", - "purpose": "GCE_ENDPOINT", - "region": "us-east4", - "self_link": "https://www.googleapis.com/compute/v1/projects/PROJECT-408009/regions/us-east4/addresses/rabbitmq-internal", - "subnetwork": "https://www.googleapis.com/compute/v1/projects/PROJECT-408009/regions/us-east4/subnetworks/PROJECT-408009", - "terraform_labels": {}, - "timeouts": null, - "users": [ - "https://www.googleapis.com/compute/v1/projects/PROJECT-408009/zones/us-east4-a/instances/rabbitmq-server" - ] - }, - "sensitive_attributes": [], - "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjoxMjAwMDAwMDAwMDAwLCJkZWxldGUiOjEyMDAwMDAwMDAwMDAsInVwZGF0ZSI6MTIwMDAwMDAwMDAwMH19", - "dependencies": [ - "google_compute_network.solana_etl", - "google_compute_subnetwork.solana_etl" - ] - } - ] - }, - { - "mode": "managed", - "type": "google_compute_address", - "name": "rabbitmq_public", - "provider": "provider[\"registry.terraform.io/hashicorp/google\"]", - "instances": [ - { - "schema_version": 0, - "attributes": { - "address": "35.212.21.90", - "address_type": "EXTERNAL", - "creation_timestamp": "2023-12-13T02:04:02.367-08:00", - "description": "", - "effective_labels": {}, - "id": "projects/PROJECT-408009/regions/us-east4/addresses/rabbitmq-public", - "ip_version": "", - "ipv6_endpoint_type": "", - "label_fingerprint": "42WmSpB8rSM=", - "labels": {}, - "name": "rabbitmq-public", - "network": "", - "network_tier": "STANDARD", - "prefix_length": 0, - "project": "PROJECT-408009", - "purpose": "", - "region": "us-east4", - "self_link": "https://www.googleapis.com/compute/v1/projects/PROJECT-408009/regions/us-east4/addresses/rabbitmq-public", - "subnetwork": "", - "terraform_labels": {}, - "timeouts": null, - "users": [ - "https://www.googleapis.com/compute/v1/projects/PROJECT-408009/zones/us-east4-a/instances/rabbitmq-server" - ] - }, - "sensitive_attributes": [], - "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjoxMjAwMDAwMDAwMDAwLCJkZWxldGUiOjEyMDAwMDAwMDAwMDAsInVwZGF0ZSI6MTIwMDAwMDAwMDAwMH19" - } - ] - }, - { - "mode": "managed", - "type": "google_compute_address", - "name": "solana_rpc_internal", - "provider": "provider[\"registry.terraform.io/hashicorp/google\"]", - "instances": [ - { - "schema_version": 0, - "attributes": { - "address": "10.0.0.10", - "address_type": "INTERNAL", - "creation_timestamp": "2023-12-13T02:04:38.330-08:00", - "description": "", - "effective_labels": {}, - "id": "projects/PROJECT-408009/regions/us-east4/addresses/solana-rpc-internal", - "ip_version": "", - "ipv6_endpoint_type": "", - "label_fingerprint": "42WmSpB8rSM=", - "labels": {}, - "name": "solana-rpc-internal", - "network": "", - "network_tier": "PREMIUM", - "prefix_length": 0, - "project": "PROJECT-408009", - "purpose": "GCE_ENDPOINT", - "region": "us-east4", - "self_link": "https://www.googleapis.com/compute/v1/projects/PROJECT-408009/regions/us-east4/addresses/solana-rpc-internal", - "subnetwork": "https://www.googleapis.com/compute/v1/projects/PROJECT-408009/regions/us-east4/subnetworks/PROJECT-408009", - "terraform_labels": {}, - "timeouts": null, - "users": [ - "https://www.googleapis.com/compute/v1/projects/PROJECT-408009/zones/us-east4-a/instances/solana-rpc" - ] - }, - "sensitive_attributes": [], - "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjoxMjAwMDAwMDAwMDAwLCJkZWxldGUiOjEyMDAwMDAwMDAwMDAsInVwZGF0ZSI6MTIwMDAwMDAwMDAwMH19", - "dependencies": [ - "google_compute_network.solana_etl", - "google_compute_subnetwork.solana_etl" - ] - } - ] - }, - { - "mode": "managed", - "type": "google_compute_address", - "name": "solana_rpc_public", - "provider": "provider[\"registry.terraform.io/hashicorp/google\"]", - "instances": [ - { - "schema_version": 0, - "attributes": { - "address": "35.212.42.100", - "address_type": "EXTERNAL", - "creation_timestamp": "2023-12-13T02:04:02.241-08:00", - "description": "", - "effective_labels": {}, - "id": "projects/PROJECT-408009/regions/us-east4/addresses/solana-rpc-public", - "ip_version": "", - "ipv6_endpoint_type": "", - "label_fingerprint": "42WmSpB8rSM=", - "labels": {}, - "name": "solana-rpc-public", - "network": "", - "network_tier": "STANDARD", - "prefix_length": 0, - "project": "PROJECT-408009", - "purpose": "", - "region": "us-east4", - "self_link": "https://www.googleapis.com/compute/v1/projects/PROJECT-408009/regions/us-east4/addresses/solana-rpc-public", - "subnetwork": "", - "terraform_labels": {}, - "timeouts": null, - "users": [ - "https://www.googleapis.com/compute/v1/projects/PROJECT-408009/zones/us-east4-a/instances/solana-rpc" - ] - }, - "sensitive_attributes": [], - "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjoxMjAwMDAwMDAwMDAwLCJkZWxldGUiOjEyMDAwMDAwMDAwMDAsInVwZGF0ZSI6MTIwMDAwMDAwMDAwMH19" - } - ] - }, - { - "mode": "managed", - "type": "google_compute_firewall", - "name": "allow_ssh_iap", - "provider": "provider[\"registry.terraform.io/hashicorp/google\"]", - "instances": [ - { - "schema_version": 1, - "attributes": { - "allow": [ - { - "ports": [ - "22" - ], - "protocol": "tcp" - } - ], - "creation_timestamp": "2023-12-13T02:04:25.021-08:00", - "deny": [], - "description": "", - "destination_ranges": [], - "direction": "INGRESS", - "disabled": false, - "enable_logging": null, - "id": "projects/PROJECT-408009/global/firewalls/allow-ssh-iap", - "log_config": [], - "name": "allow-ssh-iap", - "network": "https://www.googleapis.com/compute/v1/projects/PROJECT-408009/global/networks/PROJECT-408009-vpc", - "priority": 1000, - "project": "PROJECT-408009", - "self_link": "https://www.googleapis.com/compute/v1/projects/PROJECT-408009/global/firewalls/allow-ssh-iap", - "source_ranges": [ - "35.235.240.0/20" - ], - "source_service_accounts": [], - "source_tags": [], - "target_service_accounts": [], - "target_tags": [], - "timeouts": null - }, - "sensitive_attributes": [], - "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjoxMjAwMDAwMDAwMDAwLCJkZWxldGUiOjEyMDAwMDAwMDAwMDAsInVwZGF0ZSI6MTIwMDAwMDAwMDAwMH0sInNjaGVtYV92ZXJzaW9uIjoiMSJ9", - "dependencies": [ - "google_compute_network.solana_etl" - ] - } - ] - }, - { - "mode": "managed", - "type": "google_compute_firewall", - "name": "solana-rpc", - "provider": "provider[\"registry.terraform.io/hashicorp/google\"]", - "instances": [ - { - "schema_version": 1, - "attributes": { - "allow": [ - { - "ports": [ - "8000-10000" - ], - "protocol": "tcp" - }, - { - "ports": [ - "8000-10000" - ], - "protocol": "udp" - } - ], - "creation_timestamp": "2023-12-13T02:04:24.926-08:00", - "deny": [], - "description": "", - "destination_ranges": [], - "direction": "INGRESS", - "disabled": false, - "enable_logging": null, - "id": "projects/PROJECT-408009/global/firewalls/solana-rpc", - "log_config": [], - "name": "solana-rpc", - "network": "https://www.googleapis.com/compute/v1/projects/PROJECT-408009/global/networks/PROJECT-408009-vpc", - "priority": 1000, - "project": "PROJECT-408009", - "self_link": "https://www.googleapis.com/compute/v1/projects/PROJECT-408009/global/firewalls/solana-rpc", - "source_ranges": [ - "0.0.0.0/0" - ], - "source_service_accounts": [], - "source_tags": [], - "target_service_accounts": [], - "target_tags": [ - "solana" - ], - "timeouts": null - }, - "sensitive_attributes": [], - "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjoxMjAwMDAwMDAwMDAwLCJkZWxldGUiOjEyMDAwMDAwMDAwMDAsInVwZGF0ZSI6MTIwMDAwMDAwMDAwMH0sInNjaGVtYV92ZXJzaW9uIjoiMSJ9", - "dependencies": [ - "google_compute_network.solana_etl" - ] - } - ] - }, - { - "mode": "managed", - "type": "google_compute_instance", - "name": "rabbitmq", - "provider": "provider[\"registry.terraform.io/hashicorp/google\"]", - "instances": [ - { - "schema_version": 6, - "attributes": { - "advanced_machine_features": [], - "allow_stopping_for_update": null, - "attached_disk": [], - "boot_disk": [ - { - "auto_delete": false, - "device_name": "persistent-disk-0", - "disk_encryption_key_raw": "", - "disk_encryption_key_sha256": "", - "initialize_params": [ - { - "image": "https://www.googleapis.com/compute/v1/projects/ubuntu-os-cloud/global/images/ubuntu-minimal-2204-jammy-v20230715", - "labels": {}, - "resource_manager_tags": {}, - "size": 1200, - "type": "pd-balanced" - } - ], - "kms_key_self_link": "", - "mode": "READ_WRITE", - "source": "https://www.googleapis.com/compute/v1/projects/PROJECT-408009/zones/us-east4-a/disks/rabbitmq-server" - } - ], - "can_ip_forward": false, - "confidential_instance_config": [], - "cpu_platform": "Intel Ice Lake", - "current_status": "RUNNING", - "deletion_protection": false, - "description": "", - "desired_status": null, - "effective_labels": {}, - "enable_display": false, - "guest_accelerator": [], - "hostname": "", - "id": "projects/PROJECT-408009/zones/us-east4-a/instances/rabbitmq-server", - "instance_id": "5213235666427853588", - "label_fingerprint": "42WmSpB8rSM=", - "labels": {}, - "machine_type": "n2-standard-8", - "metadata": {}, - "metadata_fingerprint": "-PUCDzbBBL0=", - "metadata_startup_script": null, - "min_cpu_platform": "Intel Ice Lake", - "name": "rabbitmq-server", - "network_interface": [ - { - "access_config": [ - { - "nat_ip": "35.212.21.90", - "network_tier": "STANDARD", - "public_ptr_domain_name": "" - } - ], - "alias_ip_range": [], - "internal_ipv6_prefix_length": 0, - "ipv6_access_config": [], - "ipv6_access_type": "", - "ipv6_address": "", - "name": "nic0", - "network": "https://www.googleapis.com/compute/v1/projects/PROJECT-408009/global/networks/PROJECT-408009-vpc", - "network_ip": "10.0.0.20", - "nic_type": "", - "queue_count": 0, - "stack_type": "IPV4_ONLY", - "subnetwork": "https://www.googleapis.com/compute/v1/projects/PROJECT-408009/regions/us-east4/subnetworks/PROJECT-408009", - "subnetwork_project": "PROJECT-408009" - } - ], - "network_performance_config": [], - "params": [], - "project": "PROJECT-408009", - "reservation_affinity": [], - "resource_policies": [], - "scheduling": [ - { - "automatic_restart": true, - "instance_termination_action": "", - "local_ssd_recovery_timeout": [], - "min_node_cpus": 0, - "node_affinities": [], - "on_host_maintenance": "MIGRATE", - "preemptible": false, - "provisioning_model": "STANDARD" - } - ], - "scratch_disk": [], - "self_link": "https://www.googleapis.com/compute/v1/projects/PROJECT-408009/zones/us-east4-a/instances/rabbitmq-server", - "service_account": [ - { - "email": "compute-service-account@PROJECT-408009.iam.gserviceaccount.com", - "scopes": [ - "https://www.googleapis.com/auth/devstorage.read_only", - "https://www.googleapis.com/auth/logging.write", - "https://www.googleapis.com/auth/monitoring.write", - "https://www.googleapis.com/auth/service.management.readonly", - "https://www.googleapis.com/auth/servicecontrol", - "https://www.googleapis.com/auth/trace.append" - ] - } - ], - "shielded_instance_config": [ - { - "enable_integrity_monitoring": true, - "enable_secure_boot": false, - "enable_vtpm": true - } - ], - "tags": [], - "tags_fingerprint": "42WmSpB8rSM=", - "terraform_labels": {}, - "timeouts": null, - "zone": "us-east4-a" - }, - "sensitive_attributes": [], - "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjoxMjAwMDAwMDAwMDAwLCJkZWxldGUiOjEyMDAwMDAwMDAwMDAsInVwZGF0ZSI6MTIwMDAwMDAwMDAwMH0sInNjaGVtYV92ZXJzaW9uIjoiNiJ9", - "dependencies": [ - "google_compute_address.rabbitmq_internal", - "google_compute_address.rabbitmq_public", - "google_compute_network.solana_etl", - "google_compute_subnetwork.solana_etl", - "google_service_account.solana_etl" - ] - } - ] - }, - { - "mode": "managed", - "type": "google_compute_instance", - "name": "solana_rpc", - "provider": "provider[\"registry.terraform.io/hashicorp/google\"]", - "instances": [ - { - "schema_version": 6, - "attributes": { - "advanced_machine_features": [], - "allow_stopping_for_update": null, - "attached_disk": [], - "boot_disk": [ - { - "auto_delete": false, - "device_name": "persistent-disk-0", - "disk_encryption_key_raw": "", - "disk_encryption_key_sha256": "", - "initialize_params": [ - { - "image": "https://www.googleapis.com/compute/v1/projects/debian-cloud/global/images/debian-11-bullseye-v20231010", - "labels": {}, - "resource_manager_tags": {}, - "size": 500, - "type": "pd-ssd" - } - ], - "kms_key_self_link": "", - "mode": "READ_WRITE", - "source": "https://www.googleapis.com/compute/v1/projects/PROJECT-408009/zones/us-east4-a/disks/solana-rpc" - } - ], - "can_ip_forward": false, - "confidential_instance_config": [], - "cpu_platform": "Intel Ice Lake", - "current_status": "RUNNING", - "deletion_protection": false, - "description": "", - "desired_status": null, - "effective_labels": {}, - "enable_display": false, - "guest_accelerator": [], - "hostname": "", - "id": "projects/PROJECT-408009/zones/us-east4-a/instances/solana-rpc", - "instance_id": "3645019703657906007", - "label_fingerprint": "42WmSpB8rSM=", - "labels": {}, - "machine_type": "n2-highmem-64", - "metadata": {}, - "metadata_fingerprint": "-PUCDzbBBL0=", - "metadata_startup_script": null, - "min_cpu_platform": "Intel Ice Lake", - "name": "solana-rpc", - "network_interface": [ - { - "access_config": [ - { - "nat_ip": "35.212.42.100", - "network_tier": "STANDARD", - "public_ptr_domain_name": "" - } - ], - "alias_ip_range": [], - "internal_ipv6_prefix_length": 0, - "ipv6_access_config": [], - "ipv6_access_type": "", - "ipv6_address": "", - "name": "nic0", - "network": "https://www.googleapis.com/compute/v1/projects/PROJECT-408009/global/networks/PROJECT-408009-vpc", - "network_ip": "10.0.0.10", - "nic_type": "", - "queue_count": 0, - "stack_type": "IPV4_ONLY", - "subnetwork": "https://www.googleapis.com/compute/v1/projects/PROJECT-408009/regions/us-east4/subnetworks/PROJECT-408009", - "subnetwork_project": "PROJECT-408009" - } - ], - "network_performance_config": [], - "params": [], - "project": "PROJECT-408009", - "reservation_affinity": [], - "resource_policies": [], - "scheduling": [ - { - "automatic_restart": true, - "instance_termination_action": "", - "local_ssd_recovery_timeout": [], - "min_node_cpus": 0, - "node_affinities": [], - "on_host_maintenance": "MIGRATE", - "preemptible": false, - "provisioning_model": "STANDARD" - } - ], - "scratch_disk": [ - { - "device_name": "local-ssd-0", - "interface": "NVME", - "size": 375 - }, - { - "device_name": "local-ssd-1", - "interface": "NVME", - "size": 375 - }, - { - "device_name": "local-ssd-2", - "interface": "NVME", - "size": 375 - }, - { - "device_name": "local-ssd-3", - "interface": "NVME", - "size": 375 - }, - { - "device_name": "local-ssd-4", - "interface": "NVME", - "size": 375 - }, - { - "device_name": "local-ssd-5", - "interface": "NVME", - "size": 375 - }, - { - "device_name": "local-ssd-6", - "interface": "NVME", - "size": 375 - }, - { - "device_name": "local-ssd-7", - "interface": "NVME", - "size": 375 - }, - { - "device_name": "local-ssd-8", - "interface": "NVME", - "size": 375 - }, - { - "device_name": "local-ssd-9", - "interface": "NVME", - "size": 375 - }, - { - "device_name": "local-ssd-10", - "interface": "NVME", - "size": 375 - }, - { - "device_name": "local-ssd-11", - "interface": "NVME", - "size": 375 - }, - { - "device_name": "local-ssd-12", - "interface": "NVME", - "size": 375 - }, - { - "device_name": "local-ssd-13", - "interface": "NVME", - "size": 375 - }, - { - "device_name": "local-ssd-14", - "interface": "NVME", - "size": 375 - }, - { - "device_name": "local-ssd-15", - "interface": "NVME", - "size": 375 - } - ], - "self_link": "https://www.googleapis.com/compute/v1/projects/PROJECT-408009/zones/us-east4-a/instances/solana-rpc", - "service_account": [ - { - "email": "compute-service-account@PROJECT-408009.iam.gserviceaccount.com", - "scopes": [ - "https://www.googleapis.com/auth/devstorage.read_only", - "https://www.googleapis.com/auth/logging.write", - "https://www.googleapis.com/auth/monitoring.write", - "https://www.googleapis.com/auth/service.management.readonly", - "https://www.googleapis.com/auth/servicecontrol", - "https://www.googleapis.com/auth/trace.append" - ] - } - ], - "shielded_instance_config": [ - { - "enable_integrity_monitoring": true, - "enable_secure_boot": false, - "enable_vtpm": true - } - ], - "tags": [ - "solana" - ], - "tags_fingerprint": "y08fa8ZiHQ8=", - "terraform_labels": {}, - "timeouts": null, - "zone": "us-east4-a" - }, - "sensitive_attributes": [], - "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjoxMjAwMDAwMDAwMDAwLCJkZWxldGUiOjEyMDAwMDAwMDAwMDAsInVwZGF0ZSI6MTIwMDAwMDAwMDAwMH0sInNjaGVtYV92ZXJzaW9uIjoiNiJ9", - "dependencies": [ - "google_compute_address.solana_rpc_internal", - "google_compute_address.solana_rpc_public", - "google_compute_network.solana_etl", - "google_compute_subnetwork.solana_etl", - "google_service_account.solana_etl" - ] - } - ] - }, - { - "mode": "managed", - "type": "google_compute_network", - "name": "solana_etl", - "provider": "provider[\"registry.terraform.io/hashicorp/google\"]", - "instances": [ - { - "schema_version": 0, - "attributes": { - "auto_create_subnetworks": false, - "delete_default_routes_on_create": false, - "description": "", - "enable_ula_internal_ipv6": false, - "gateway_ipv4": "", - "id": "projects/PROJECT-408009/global/networks/PROJECT-408009-vpc", - "internal_ipv6_range": "", - "mtu": 1460, - "name": "PROJECT-408009-vpc", - "network_firewall_policy_enforcement_order": "AFTER_CLASSIC_FIREWALL", - "project": "PROJECT-408009", - "routing_mode": "REGIONAL", - "self_link": "https://www.googleapis.com/compute/v1/projects/PROJECT-408009/global/networks/PROJECT-408009-vpc", - "timeouts": null - }, - "sensitive_attributes": [], - "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjoxMjAwMDAwMDAwMDAwLCJkZWxldGUiOjEyMDAwMDAwMDAwMDAsInVwZGF0ZSI6MTIwMDAwMDAwMDAwMH19" - } - ] - }, - { - "mode": "managed", - "type": "google_compute_subnetwork", - "name": "solana_etl", - "provider": "provider[\"registry.terraform.io/hashicorp/google\"]", - "instances": [ - { - "schema_version": 0, - "attributes": { - "creation_timestamp": "2023-12-13T02:04:25.555-08:00", - "description": "", - "external_ipv6_prefix": "", - "fingerprint": null, - "gateway_address": "10.0.0.1", - "id": "projects/PROJECT-408009/regions/us-east4/subnetworks/PROJECT-408009", - "internal_ipv6_prefix": "", - "ip_cidr_range": "10.0.0.0/24", - "ipv6_access_type": "", - "ipv6_cidr_range": "", - "log_config": [], - "name": "PROJECT-408009", - "network": "https://www.googleapis.com/compute/v1/projects/PROJECT-408009/global/networks/PROJECT-408009-vpc", - "private_ip_google_access": true, - "private_ipv6_google_access": "DISABLE_GOOGLE_ACCESS", - "project": "PROJECT-408009", - "purpose": "PRIVATE", - "region": "us-east4", - "role": "", - "secondary_ip_range": [], - "self_link": "https://www.googleapis.com/compute/v1/projects/PROJECT-408009/regions/us-east4/subnetworks/PROJECT-408009", - "stack_type": "IPV4_ONLY", - "timeouts": null - }, - "sensitive_attributes": [], - "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjoxMjAwMDAwMDAwMDAwLCJkZWxldGUiOjEyMDAwMDAwMDAwMDAsInVwZGF0ZSI6MTIwMDAwMDAwMDAwMH19", - "dependencies": [ - "google_compute_network.solana_etl" - ] - } - ] - }, - { - "mode": "managed", - "type": "google_project_iam_member", - "name": "service_account_roles", - "provider": "provider[\"registry.terraform.io/hashicorp/google\"]", - "instances": [ - { - "index_key": "roles/logging.logWriter", - "schema_version": 0, - "attributes": { - "condition": [], - "etag": "BwYMYWzjhY4=", - "id": "PROJECT-408009/roles/logging.logWriter/serviceAccount:compute-service-account@PROJECT-408009.iam.gserviceaccount.com", - "member": "serviceAccount:compute-service-account@PROJECT-408009.iam.gserviceaccount.com", - "project": "PROJECT-408009", - "role": "roles/logging.logWriter" - }, - "sensitive_attributes": [], - "private": "bnVsbA==", - "dependencies": [ - "google_service_account.solana_etl" - ] - }, - { - "index_key": "roles/monitoring.metricWriter", - "schema_version": 0, - "attributes": { - "condition": [], - "etag": "BwYMYWzjhY4=", - "id": "PROJECT-408009/roles/monitoring.metricWriter/serviceAccount:compute-service-account@PROJECT-408009.iam.gserviceaccount.com", - "member": "serviceAccount:compute-service-account@PROJECT-408009.iam.gserviceaccount.com", - "project": "PROJECT-408009", - "role": "roles/monitoring.metricWriter" - }, - "sensitive_attributes": [], - "private": "bnVsbA==", - "dependencies": [ - "google_service_account.solana_etl" - ] - }, - { - "index_key": "roles/storage.objectViewer", - "schema_version": 0, - "attributes": { - "condition": [], - "etag": "BwYMYWzjhY4=", - "id": "PROJECT-408009/roles/storage.objectViewer/serviceAccount:compute-service-account@PROJECT-408009.iam.gserviceaccount.com", - "member": "serviceAccount:compute-service-account@PROJECT-408009.iam.gserviceaccount.com", - "project": "PROJECT-408009", - "role": "roles/storage.objectViewer" - }, - "sensitive_attributes": [], - "private": "bnVsbA==", - "dependencies": [ - "google_service_account.solana_etl" - ] - } - ] - }, - { - "mode": "managed", - "type": "google_service_account", - "name": "solana_etl", - "provider": "provider[\"registry.terraform.io/hashicorp/google\"]", - "instances": [ - { - "schema_version": 0, - "attributes": { - "account_id": "compute-service-account", - "description": "", - "disabled": false, - "display_name": "Compute Service Account", - "email": "compute-service-account@PROJECT-408009.iam.gserviceaccount.com", - "id": "projects/PROJECT-408009/serviceAccounts/compute-service-account@PROJECT-408009.iam.gserviceaccount.com", - "member": "serviceAccount:compute-service-account@PROJECT-408009.iam.gserviceaccount.com", - "name": "projects/PROJECT-408009/serviceAccounts/compute-service-account@PROJECT-408009.iam.gserviceaccount.com", - "project": "PROJECT-408009", - "timeouts": null, - "unique_id": "114338025937302118981" - }, - "sensitive_attributes": [], - "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjozMDAwMDAwMDAwMDB9fQ==" - } - ] - } - ], - "check_results": null -} From c6206881882aaadad186c9e121666e0022798773 Mon Sep 17 00:00:00 2001 From: Toko Machavariani <109960534+tornikebcw@users.noreply.github.com> Date: Wed, 7 Aug 2024 12:56:05 +0400 Subject: [PATCH 4/5] Update README.md --- iac/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/iac/README.md b/iac/README.md index a78dc3c..b1df4b3 100644 --- a/iac/README.md +++ b/iac/README.md @@ -55,7 +55,7 @@ This repository contains the necessary infrastructure as code to set up a Solana ### 5. Solana Node Configuration 1. After provisioning the infrastructure, download and run the scripts located in the `scripts/solana-rpc` directory on the Solana node. -2. These scripts handle disk configuration, Solana RPC node setup, and will expose the RPC port on `8899`. +2. These scripts handle disk configuration, Solana RPC node setup, and will expose the RPC port on `8899` internally which will be proxied through Nginx on port 80. 3. Execute the `configure-solana-disks.sh` script first and ensure it completes successfully: ```bash chmod +x configure-solana-disks.sh From c838a33c6940daee1e065e537712e79258fc0921 Mon Sep 17 00:00:00 2001 From: Toko Machavariani <109960534+tornikebcw@users.noreply.github.com> Date: Wed, 7 Aug 2024 12:56:41 +0400 Subject: [PATCH 5/5] Update README.md --- iac/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/iac/README.md b/iac/README.md index b1df4b3..666e62e 100644 --- a/iac/README.md +++ b/iac/README.md @@ -55,7 +55,7 @@ This repository contains the necessary infrastructure as code to set up a Solana ### 5. Solana Node Configuration 1. After provisioning the infrastructure, download and run the scripts located in the `scripts/solana-rpc` directory on the Solana node. -2. These scripts handle disk configuration, Solana RPC node setup, and will expose the RPC port on `8899` internally which will be proxied through Nginx on port 80. +2. These scripts handle disk configuration, Solana RPC node setup,Nginx configuraiton and will expose the RPC port on `8899` internally which will be proxied through Nginx on port 80. 3. Execute the `configure-solana-disks.sh` script first and ensure it completes successfully: ```bash chmod +x configure-solana-disks.sh