Add a jump-host in the project to continue terraform installation

docs/INSTALLATION.md

@@ -64,7 +64,7 @@ We recommend starting with a brand new GCP project.
 
 Run the following to generate a new solution skeleton at the current directory:
 ```
-st new my-solution .
+sb new my-solution .
 ```
 
 This will prompt options and variables:
@@ -102,7 +102,7 @@ Once filled, it will generate a new folder `my-solution` with the following file
 ```
 
 - **README.md**: The default empty README for your project.
-- **components/**: List of component/service folders. When adding components by running `st components add [COMPONENT_NAME]`, it will add a component subfolder here.
+- **components/**: List of component/service folders. When adding components by running `sb components add [COMPONENT_NAME]`, it will add a component subfolder here.
 - **setup.cfg**: yapf python formatter.
 - **skaffold.yaml**: Root skaffold YAML that manages deployment to Cloud Run and GKE.
 - **sb.yaml**: Root Solutions Builder YAML that contains metadata for this solution folder.
@@ -118,7 +118,7 @@ Run the following to update an existing solution folder:
 
 ```
 cd my-solution
-st update .
+sb update .
 ```
 
 This will prompt the same questions like in creating a new solution. You can provide a different values like a new GCP project ID or region. Once complete, it will replace the project ID and region to the exising solution folder.
@@ -146,14 +146,14 @@ The `1-bootstrap` stage creates a GCS bucket for persisting terraform state file
 
 Run the following to initialize both stages:
 ```
-st infra init
+sb infra init
 ```
 - Terraform will prompt for approval before proceed the terraform apply.
 - You can also pass a `--yes` to automatically approve the changes.
 
 Alternatively, to run a particular stage:
 ```
-st infra apply [STAGE_NAME]
+sb infra apply [STAGE_NAME]
 ```
 
 ## Add a component
@@ -174,10 +174,10 @@ Available module names:
 
 To add a component to a solution:
 ```
-st components add [COMPONENT_NAME]
+sb components add [COMPONENT_NAME]
 ```
 
-This will show the prompt quesitons from this particular component.
+This will show the prompt questions from this particular component.
 
 ### Example: Add a Todo List RESTful API microservice
 
@@ -248,7 +248,7 @@ Once complete, it adds the `todo_service` folder to `my-solution/components`.
 
 At this point, we'll deploy this service to Cloud Run with the following command:
 ```
-st deploy
+sb deploy
 ```
 - This will run `skaffold run` to deploy all services with `default` profile.
 
@@ -274,14 +274,14 @@ An infra component is nothing but another component. Some component contains jus
 
 To add an infra component, run the same command like adding a regular component.
 ```
-st components add [COMPONENT_NAME]
+sb components add [COMPONENT_NAME]
 ```
 
 ### Example: Add a HTTP load balancer
 
 Run the following to add a HTTP load balancer that supports Google-managed cert and a domain name.
 ```
-st components add terraform_httplb_cloudrun
+sb components add terraform_httplb_cloudrun
 ```
 
 Fill in the answers in the prompt.
@@ -302,7 +302,7 @@ This component creates a stage `3-httplb-cloudrun`, which you can find in `terra
 
 To apply the infra terraform code:
 ```
-st infra apply 3-httplb-cloudrun
+sb infra apply 3-httplb-cloudrun
 
 ... (terraform execution)
 

solutions_builder/template_root/copier.yaml

@@ -23,6 +23,17 @@ gcp_region:
   help: Which Google Cloud region?
   default: us-central1
 
+use_jump_host:
+  type: bool
+  help: Use a jump-host for running terraform foundations?
+  default: true
+
+jump_host_zone:
+  type: str
+  help: Google Cloud zone for creating the jump host?
+  default: us-central1-a
+  when: "{{use_jump_host}}"
+
 has_common:
   type: bool
   help: Include a common container image for shared libraries, data models, utils, etc?
@@ -35,7 +46,7 @@ terraform_backend_gcs:
 
 advanced_settings:
   type: bool
-  help: Show advanced settings (VPC, Storage region, etc)?
+  help: Show advanced settings (VPC, Storage, Region, etc)?
   default: false
 
 create_vpc_network:

solutions_builder/template_root/terraform/modules/bastion/main.tf

@@ -0,0 +1,17 @@
+module "iap_bastion" {
+  source                     = "terraform-google-modules/bastion-host/google"
+  version                    = "5.3.0"
+  name                       = "jump-host"
+  machine_type               = var.machine_type
+  project                    = var.project_id
+  zone                       = var.zone
+  network                    = var.vpc_network_self_link
+  subnet                     = var.vpc_subnetworks_self_link
+  image                      = var.image
+  image_family               = var.image_family
+  image_project              = var.image_project
+  disk_size_gb               = var.disk_size_gb
+  disk_type                  = "pd-balanced"
+  fw_name_allow_ssh_from_iap = "allow-ssh-ingress-from-iap"
+  startup_script             = var.startup_script
+}

solutions_builder/template_root/terraform/modules/bastion/variables.tf

@@ -0,0 +1,55 @@
+variable "project_id" {
+  type        = string
+  description = "specify the project name"
+}
+
+variable "zone" {
+  type        = string
+  description = "zone for the bastion host"
+}
+
+variable "vpc_network_self_link" {
+  type        = string
+  description = "specify the vpc network self link"
+}
+
+variable "vpc_subnetworks_self_link" {
+  type        = string
+  description = "specify the vpc subnetworks self link"
+}
+
+variable "machine_type" {
+  type        = string
+  description = "Instance type for the Bastion host"
+  default     = "n2-standard-4"
+}
+
+variable "image" {
+  type        = string
+  description = "Source image for the Bastion"
+  default     = "ubuntu-2004-focal-v20230302"
+}
+
+variable "image_family" {
+  type        = string
+  description = "Source image family for the Bastion"
+  default     = "ubuntu-2004"
+}
+
+variable "image_project" {
+  type        = string
+  description = "Project where the source image for the Bastion comes from"
+  default     = "ubuntu-os-cloud"
+}
+
+variable "disk_size_gb" {
+  type        = number
+  description = "Boot disk size in GB"
+  default     = 200
+}
+
+variable "startup_script" {
+  type        = string
+  description = "startup script to be executed upon provisioning"
+  default     = ""
+}

solutions_builder/template_root/terraform/modules/vpc_network/main.tf

@@ -43,3 +43,16 @@ module "vpc" {
   }
 
 }
+
+module "cloud-nat" {
+  source            = "terraform-google-modules/cloud-nat/google"
+  version           = "~> 1.2"
+  name              = format("%s-%s-nat", var.project_id, var.region)
+  create_router     = true
+  router            = format("%s-%s-router", var.project_id, var.region)
+  project_id        = var.project_id
+  region            = var.region
+  network           = module.vpc.network_id
+  log_config_enable = true
+  log_config_filter = "ERRORS_ONLY"
+}

solutions_builder/template_root/terraform/modules/vpc_network/outputs.tf

@@ -0,0 +1,27 @@
+/**
+ * Copyright 2023 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+# network and subnetwork self links are used as input to create a bastion host
+output "network_self_link" {
+  value       = module.vpc.network_self_link
+  description = "The URI of the VPC being created"
+}
+
+output "subnets_self_link" {
+  value       = module.vpc.subnets_self_links
+  description = "The URI of the VPC subnetworks being created"
+}

solutions_builder/template_root/terraform/stages/1-bootstrap/main.tf

@@ -1,3 +1,20 @@
+/**
+ * Copyright 2023 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
 locals {
   services = [
     "appengine.googleapis.com",            # App Engine
@@ -20,7 +37,7 @@ resource "google_project_service" "project-apis" {
 
 # add timer to avoid errors on new project creation and API enables
 resource "time_sleep" "wait_60_seconds" {
-  depends_on = [google_project_service.project-apis]
+  depends_on      = [google_project_service.project-apis]
   create_duration = "60s"
 }
 
@@ -46,7 +63,6 @@ module "terraform_runner_service_account" {
     "roles/iam.serviceAccountTokenCreator",
     "roles/iam.serviceAccountUser",
     "roles/iam.workloadIdentityUser",
-    "roles/iam.workloadIdentityUser",
     "roles/logging.admin",
     "roles/logging.viewer",
     "roles/resourcemanager.projectIamAdmin",

solutions_builder/template_root/terraform/stages/2-foundation/main.tf

@@ -62,6 +62,10 @@ locals {
 # Used to retrieve project_number later
 data "google_project" "project" {}
 
+resource "google_identity_platform_config" "default" {
+  project = var.project_id
+}
+
 module "project_services" {
   source     = "../../modules/project_services"
   project_id = var.project_id
@@ -81,6 +85,23 @@ module "vpc_network" {
   secondary_ranges_services = var.secondary_ranges_services
 }
 
+# The startup script continues to run while the jump host is deemed ready
+# It may take a up to 15 minutes for the script to complete
+data "template_file" "startup_script" {
+  template = file("${path.module}/../scripts/bastion_startup.sh")
+}
+
+module "bastion_host" {
+  depends_on                = [module.vpc_network]
+  source                    = "../../modules/bastion"
+  count                     = var.use_jump_host ? 0 : 1
+  project_id                = var.project_id
+  zone                      = var.jump_host_zone
+  vpc_network_self_link     = module.vpc_network[0].network_self_link
+  vpc_subnetworks_self_link = module.vpc_network[0].subnets_self_link[0]
+  startup_script            = data.template_file.startup_script.rendered
+}
+
 resource "google_project_iam_member" "cloudbuild-sa-iam" {
   depends_on = [module.project_services]
   for_each   = toset(local.roles_for_default_sa)

solutions_builder/template_root/terraform/stages/2-foundation/terraform.tfvars

@@ -2,6 +2,8 @@ project_id              = "{{project_id}}"
 project_number          = "{{project_number}}"
 region                  = "{{gcp_region}}"
 storage_multiregion     = "US"
+use_jump_host           = "{{use_jump_host}}"
+jump_host_zone          = "{{jump_host_zone}}"
 vpc_network             = "{{vpc_network}}"
 vpc_subnetwork          = "{{vpc_subnetwork}}"
 ip_cidr_range           = "{{ip_cidr_range}}"

solutions_builder/template_root/terraform/stages/2-foundation/variables.tf

@@ -50,6 +50,17 @@ variable "region" {
   }
 }
 
+variable "use_jump_host" {
+  type        = bool
+  description = "Whether to create a jump host for running the foundation step"
+  default     = true
+}
+
+variable "jump_host_zone" {
+  type        = string
+  description = "GCP Zone for the jump host"
+}
+
 variable "vpc_network" {
   type    = string
   default = "default-vpc"

solutions_builder/template_root/terraform/stages/scripts/bastion_startup.sh

@@ -0,0 +1,60 @@
+#!/bin/bash
+# Copyright 2022 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Startup script for the bastion host
+# Installs terraform, gcloud and other dependencies
+
+cd /tmp
+
+# Update system packages
+sudo apt-get update -y && sudo apt-get autoremove -y
+sudo DEBIAN_FRONTEND=noninteractive apt-get upgrade -yq
+sudo apt-get install apt-transport-https python3-pip pipenv redis-tools unzip -y
+
+sudo addgroup --system docker
+sudo snap install docker && sudo snap start docker
+
+# Install Kustomize
+export KUSTOMIZE_VERSION=4.5.7
+wget https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2Fv"$KUSTOMIZE_VERSION"/kustomize_v"$KUSTOMIZE_VERSION"_linux_amd64.tar.gz
+tar -xzf ./kustomize_v"$KUSTOMIZE_VERSION"_linux_amd64.tar.gz
+sudo cp kustomize /usr/local/bin/
+
+# Install Skaffold
+export SKAFFOLD_VERSION=1.39.2
+export SKAFFOLD_PLATFORM=linux
+curl -Lo skaffold https://storage.googleapis.com/skaffold/releases/v"$SKAFFOLD_VERSION"/skaffold-"$SKAFFOLD_PLATFORM"-amd64
+sudo install skaffold /usr/local/bin/
+
+# Re-install gcloud
+sudo snap remove google-cloud-cli
+echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | \
+  sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
+curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -
+sudo apt-get update -y && sudo apt-get install google-cloud-cli google-cloud-sdk-gke-gcloud-auth-plugin -y
+gcloud components list
+
+# Install terraform, helm, kubectl and other tools
+sudo snap install terraform --classic
+sudo snap install helm --classic
+sudo snap install kubectl --classic
+pushd /usr/bin
+sudo ln -s python3 python
+popd
+python -m pip install --upgrade pip
+python -m pip install google-cloud-firestore google-cloud-bigquery firestore-admin
+touch completed
+cd
+