Autoscaler tool for Cloud Spanner

Set up the Autoscaler in Cloud Run functions in a distributed deployment using Terraform
Home · Scaler component · Poller component · Forwarder component · Terraform configuration · Monitoring
Cloud Run functions · Google Kubernetes Engine
Per-Project · Centralized · Distributed

Overview

This directory contains Terraform configuration files to quickly set up the infrastructure for your Autoscaler with a distributed deployment.

In this deployment option all the components of the Autoscaler reside in a single project, with the exception of Cloud Scheduler (step 1) and the Forwarder topic and function

This deployment is the best of both worlds between the per-project and the centralized deployments: Teams who own the Spanner instances, called Application teams, are able to manage the Autoscaler configuration parameters for their instances with their own Cloud Scheduler jobs. On the other hand, the rest of the Autoscaler infrastructure is managed by a central team.

Architecture

For an explanation of the components of the Autoscaler and the interaction flow, please read the main Architecture section.

Cloud Scheduler can only publish messages to topics in the same project. Therefore in step 2, we transparently introduce an intermediate component to make this architecture possible. For more information, see the Forwarder function.

The distributed deployment has the following pros and cons:

Pros

Configuration and infrastructure: application teams are in control of their config and schedules
Maintenance: Scaler infrastructure is centralized, reducing up-keep overhead
Policies and audit: Best practices across teams might be easier to specify and enact. Audits might be easier to execute.

Cons

Configuration: application teams need to provide service accounts to write to the polling topic.
Risk: the centralized team itself may become a single point of failure even if the infrastructure is designed with high availability in mind.

Before you begin

Open the Cloud Console
Activate Cloud Shell
At the bottom of the Cloud Console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Cloud SDK already installed, including the gcloud command-line tool, and with values already set for your current project. It can take a few seconds for the session to initialize.

In Cloud Shell, clone this repository

git clone https://github.com/cloudspannerecosystem/autoscaler.git

Export variables for the working directories

export AUTOSCALER_DIR="$(pwd)/autoscaler/terraform/cloud-functions/distributed/autoscaler-project"
export APP_DIR="$(pwd)/autoscaler/terraform/cloud-functions/distributed/app-project"

Preparing the Autoscaler Project

In this section you prepare the deployment of the project where the centralized Autoscaler infrastructure, with the exception of Cloud Scheduler, lives.

Go to the project selector page in the Cloud Console. Select or create a Cloud project.
Make sure that billing is enabled for your Google Cloud project. Learn how to confirm billing is enabled for your project.

In Cloud Shell, set environment variables with the ID of your autoscaler project:

export AUTOSCALER_PROJECT_ID=<INSERT_YOUR_PROJECT_ID>
gcloud config set project "${AUTOSCALER_PROJECT_ID}"

Choose the region and App Engine Location where the Autoscaler infrastructure will be located.
```
export AUTOSCALER_REGION=us-central1
export AUTOSCALER_APP_ENGINE_LOCATION=us-central
```

Enable the required Cloud APIs :

gcloud services enable \
    appengine.googleapis.com \
    cloudbuild.googleapis.com \
    cloudfunctions.googleapis.com  \
    cloudresourcemanager.googleapis.com \
    compute.googleapis.com \
    eventarc.googleapis.com \
    firestore.googleapis.com \
    iam.googleapis.com \
    logging.googleapis.com \
    monitoring.googleapis.com \
    pubsub.googleapis.com \
    spanner.googleapis.com

Create a Google App Engine app, to enable the APIs for Cloud Scheduler and Firestore.
```
gcloud app create --region="${AUTOSCALER_APP_ENGINE_LOCATION}"
```
The Autoscaler state can be stored in either Firestore or Cloud Spanner.

In case you want to use Firestore, update the database created with the Google App Engine app to use Firestore native mode.
```
gcloud firestore databases update --type=firestore-native
```
In case you want to use Cloud Spanner, no action is needed at this point.

Deploying the Autoscaler

Set the project ID, region, and App Engine location in the corresponding Terraform environment variables

export TF_VAR_project_id="${AUTOSCALER_PROJECT_ID}"
export TF_VAR_region="${AUTOSCALER_REGION}"
export TF_VAR_location="${AUTOSCALER_APP_ENGINE_LOCATION}"

Change directory into the Terraform scaler-project directory and initialize it.
```
cd "${AUTOSCALER_DIR}"
terraform init
```
Create the Autoscaler infrastructure. Answer yes when prompted, after reviewing the resources that Terraform intends to create.
```
terraform apply -parallelism=2
```
If you are running this command in Cloud Shell and encounter errors of the form "Error: cannot assign requested address", this is a known issue in the Terraform Google provider, please retry with -parallelism=1.

Preparing the Application Project

In this section you prepare the deployment of the Cloud Scheduler, Forwarder topic and function in the project where the Spanner instances live.

Go to the project selector page in the Cloud Console. Select or create a Cloud project.
Make sure that billing is enabled for your Google Cloud project. Learn how to confirm billing is enabled for your project.

In Cloud Shell, set the environment variables with the ID of your application project:

export APP_PROJECT_ID=<INSERT_YOUR_APP_PROJECT_ID>
gcloud config set project "${APP_PROJECT_ID}"

Choose the region and App Engine Location where the Application project will be located.
```
export APP_REGION=us-central1
export APP_APP_ENGINE_LOCATION=us-central
```

Use the following command to enable the Cloud APIs:

gcloud services enable
    appengine.googleapis.com \
    cloudbuild.googleapis.com \
    cloudfunctions.googleapis.com \
    cloudresourcemanager.googleapis.com \
    cloudscheduler.googleapis.com \
    compute.googleapis.com \
    eventarc.googleapis.com \
    iam.googleapis.com \
    logging.googleapis.com \
    monitoring.googleapis.com \
    pubsub.googleapis.com \
    run.googleapis.com \
    spanner.googleapis.com

Create an App to enable Cloud Scheduler, but do not create a Firestore database:
```
gcloud app create --region="${APP_APP_ENGINE_LOCATION}"
```

Deploy the Application infrastructure

Set the project ID, region, and App Engine location in the corresponding Terraform environment variables

export TF_VAR_project_id="${APP_PROJECT_ID}"
export TF_VAR_region="${APP_REGION}"
export TF_VAR_location="${APP_APP_ENGINE_LOCATION}"

If you want to create a new Spanner instance for testing the Autoscaler, set the following variable. The spanner instance that Terraform creates is named autoscale-test.
```
export TF_VAR_terraform_spanner_test=true
```
On the other hand, if you do not want to create a new Spanner instance because you already have an instance for the Autoscaler to monitor, set the name name of your instance in the following variable
```
export TF_VAR_spanner_name=<INSERT_YOUR_SPANNER_INSTANCE_NAME>
```
For more information on how to make your Spanner instance to be managed by Terraform, see Importing your Spanner instances

You have the option to store the Autoscaler state either in Firestore or in Cloud Spanner.

If you want to store the state in Firestore, set the project ID where the Firestore instance resides.

export TF_VAR_state_project_id="${AUTOSCALER_PROJECT_ID}"

On the other hand, if you want to store the state in Cloud Spanner and you don't have a Spanner instance yet for that, then set the following two variables so that Terraform creates an instance for you named autoscale-test-state:

export TF_VAR_terraform_spanner_state=true
export TF_VAR_state_project_id="${APP_PROJECT_ID}"

It is a best practice not to store the Autoscaler state in the same instance that is being monitored by the Autoscaler.

Optionally, you can change the name of the instance that Terraform will create:

export TF_VAR_spanner_state_name=<INSERT_STATE_SPANNER_INSTANCE_NAME>

If you already have an Spanner instance where state must be stored, only set the project id and the name of your instance:

export TF_VAR_state_project_id=<INSERT_YOUR_STATE_PROJECT_ID>
export TF_VAR_spanner_state_name=<INSERT_YOUR_STATE_SPANNER_INSTANCE_NAME>

In your own instance, make sure you create the the database spanner-autoscaler-state with the following table:

CREATE TABLE spannerAutoscaler (
  id STRING(MAX),
  lastScalingTimestamp TIMESTAMP,
  createdOn TIMESTAMP,
  updatedOn TIMESTAMP,
  lastScalingCompleteTimestamp TIMESTAMP,
  scalingOperationId STRING(MAX),
  scalingRequestedSize INT64,
  scalingMethod STRING(MAX),
  scalingPreviousSize INT64,
) PRIMARY KEY (id)

Note: If you are upgrading from v1.x, then you need to add the 5 new columns to the spanner schema using the following DDL statements

ALTER TABLE spannerAutoscaler ADD COLUMN IF NOT EXISTS lastScalingCompleteTimestamp TIMESTAMP;
ALTER TABLE spannerAutoscaler ADD COLUMN IF NOT EXISTS scalingOperationId STRING(MAX);
ALTER TABLE spannerAutoscaler ADD COLUMN IF NOT EXISTS scalingRequestedSize INT64;
ALTER TABLE spannerAutoscaler ADD COLUMN IF NOT EXISTS scalingMethod STRING(MAX);
ALTER TABLE spannerAutoscaler ADD COLUMN IF NOT EXISTS scalingPreviousSize INT64;

Note: If you are upgrading from V2.0.x, then you need to add the 3 new columns to the spanner schema using the following DDL statements

ALTER TABLE spannerAutoscaler ADD COLUMN IF NOT EXISTS scalingRequestedSize INT64;
ALTER TABLE spannerAutoscaler ADD COLUMN IF NOT EXISTS scalingMethod STRING(MAX);
ALTER TABLE spannerAutoscaler ADD COLUMN IF NOT EXISTS scalingPreviousSize INT64;

For more information on how to make your existing Spanner instance to be managed by Terraform, see Importing your Spanner instances

Change directory into the Terraform app-project directory and initialize it.
```
cd "${APP_DIR}"
terraform init
```
Create the infrastructure in the application project. Answer yes when prompted, after reviewing the resources that Terraform intends to create.
```
terraform import module.scheduler.google_app_engine_application.app "${APP_PROJECT_ID}"
terraform apply -parallelism=2
```
If you are running this command in Cloud Shell and encounter errors of the form "Error: cannot assign requested address", this is a known issue in the Terraform Google provider, please retry with -parallelism=1

Authorize the Forwarder function to publish to the Poller topic

Switch back to the Autoscaler project and ensure that Terraform variables are correctly set.

cd "${AUTOSCALER_DIR}"

export TF_VAR_project_id="${AUTOSCALER_PROJECT_ID}"
export TF_VAR_region="${AUTOSCALER_REGION}"
export TF_VAR_location="${AUTOSCALER_APP_ENGINE_LOCATION}"

Set the Terraform variables for your Forwarder service accounts, updating and adding your service accounts as needed. Answer yes when prompted, after reviewing the resources that Terraform intends to create.
```
export TF_VAR_forwarder_sa_emails='["serviceAccount:forwarder-sa@'"${APP_PROJECT_ID}"'.iam.gserviceaccount.com"]'
terraform apply -parallelism=2
```

If you are running this command in Cloud Shell and encounter errors of the form "Error: cannot assign requested address", this is a known issue in the Terraform Google provider, please retry with -parallelism=1

Verifying your deployment

Your Autoscaler infrastructure is ready, follow the instructions in the main page to configure your Autoscaler. Please take in account that In a distributed deployment: Logs from the Poller and Scaler functions will appear in the Logs Viewer for the Autoscaler project. Logs about syntax errors in the JSON configuration of the Cloud Scheduler payload will appear in the Logs viewer of each Application project, so that the team responsible for a specific Cloud Spanner instance can troubleshoot its configuration issues independently.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

README.md

README.md

Autoscaler tool for Cloud Spanner

Table of Contents

Overview

Architecture

Pros

Cons

Before you begin

Preparing the Autoscaler Project

Deploying the Autoscaler

Preparing the Application Project

Deploy the Application infrastructure

Authorize the Forwarder function to publish to the Poller topic

Verifying your deployment

Files

README.md

Latest commit

History

README.md

File metadata and controls

Autoscaler tool for Cloud Spanner

Table of Contents

Overview

Architecture

Pros

Cons

Before you begin

Preparing the Autoscaler Project

Deploying the Autoscaler

Preparing the Application Project

Deploy the Application infrastructure

Authorize the Forwarder function to publish to the Poller topic

Verifying your deployment