Skip to content

obs-gh-colinhutchinson/terraform-google-collection

 
 

Repository files navigation

Observe Google Collection

This module creates a log sink, pub/sub topic, and pub/sub subscription needed to facilitate the collection of asset inventory records, metrics and logs from GCP for a given project.

This module also creates a Cloud Function to fetch some data through the GCP REST API.

Usage

Here is an example manifest for collecting data from a Google Cloud organization.

After running terraform apply, data should start flowing into Pub/Sub. In the Observe UI, one would set up the GCP app. The info from the terraform output and terraform output -raw service_account_private_key are needed to set up the GCP App pollers.

provider "google" {
  project = "YOUR_PROJECT_ID"
  region  = "YOUR_DEFAULT_REGION"
}

module "observe_gcp_collection" {
  source  = "observeinc/collection/google"
  name    = "observe"

  resource = "projects/YOUR_PROJECT_ID"
}

output "project" {
  description = "The Pub/Sub project of the subcription (to be passed to the Pub/Sub poller)"
  value       = module.observe_gcp_collection.project
}

# To extract correct value - terraform output -json | jq -r '.subscription.value.name' 
output "subscription" {
  description = "The Pub/Sub subscription created by this module (to be passed to the Pub/Sub poller)"
  value       = module.observe_gcp_collection.subscription
}

# To extract properly formatted string - terraform output -json | jq -r '.service_account_private_key.value'
output "service_account_private_key" {
  description = "A service account key to be passed to the pollers for Pub/Sub and Cloud Monitoring"
  value       = base64decode(module.observe_gcp_collection.service_account_key.private_key)
  sensitive   = true
}

Requirements

Name Version
terraform >= 0.12.21
google >= 4.15

Providers

Name Version
google 4.71.0

Modules

No modules.

Resources

Name Type
google_cloud_scheduler_job.this resource
google_cloudfunctions_function.gcs_function resource
google_cloudfunctions_function.this resource
google_cloudfunctions_function_iam_member.cloud_scheduler resource
google_folder_iam_member.cloudfunction resource
google_logging_folder_sink.this resource
google_logging_organization_sink.this resource
google_logging_project_sink.this resource
google_organization_iam_member.cloudfunction resource
google_project_iam_member.cloudfunction resource
google_project_iam_member.poller resource
google_pubsub_subscription.this resource
google_pubsub_subscription_iam_member.poller_pubsub resource
google_pubsub_topic.this resource
google_pubsub_topic_iam_member.cloudfunction_pubsub resource
google_pubsub_topic_iam_member.sink_pubsub resource
google_service_account.cloud_scheduler resource
google_service_account.cloudfunction resource
google_service_account.poller resource
google_service_account_key.poller resource
google_storage_bucket.this resource
google_storage_bucket_iam_member.bucket_iam resource
google_storage_bucket_iam_member.gcs_function_bucket_iam resource
google_folder.this data source
google_project.this data source

Inputs

Name Description Type Default Required
enable_function Whether to enable the Cloud function bool true no
folder_include_children Whether to include all children Projects of a Folder when collecting logs bool true no
function_available_memory_mb Memory (in MB), available to the function. Default value is 512. Possible values include 128, 256, 512, 1024, etc. number 512 no
function_bucket GCS bucket containing the Cloud Function source code string "observeinc" no
function_disable_logging Whether to disable function logging. bool false no
function_max_instances The limit on the maximum number of function instances that may coexist at a given time. number 5 no
function_object GCS object key of the Cloud Function source code zip file string "google-cloud-functions-v0.3.0-alpha.8.zip" no
function_roles A list of IAM roles to give the Cloud Function. set(string)
[
"roles/compute.viewer",
"roles/iam.serviceAccountViewer",
"roles/cloudscheduler.viewer",
"roles/cloudasset.viewer",
"roles/browser",
"roles/logging.viewer",
"roles/monitoring.viewer",
"roles/storage.objectCreator",
"roles/storage.objectViewer",
"roles/storage.objectAdmin",
"roles/storage.admin"
]
no
function_schedule_frequency Cron schedule for the job string "0 * * * *" no
function_timeout Timeout (in seconds) for the function. Default value is 300 seconds. Cannot be more than 540 seconds. number 300 no
labels A map of labels to add to resources (https://cloud.google.com/resource-manager/docs/creating-managing-labels)"

Note: Many, but not all, Google Cloud SDK resources support labels.
map(string) {} no
logging_exclusions Log entries that match any of these exclusion filters will not be exported.

If a log entry is matched by both logging_filter and one of logging_exclusions it will not be exported.

Relevant docs: https://cloud.google.com/logging/docs/reference/v2/rest/v2/billingAccounts.exclusions#LogExclusion
list(object({
name = string
description = string
filter = string
disabled = string
}))
[] no
logging_filter An advanced logs filter. The only exported log entries are those that are
in the resource owning the sink and that match the filter.

Relevant docs: https://cloud.google.com/logging/docs/view/building-queries
string "" no
name Module name. Used as a name prefix. string "observe-collection" no
poller_roles A list of IAM roles to give the Observe poller (through the service account key output). set(string)
[
"roles/monitoring.viewer"
]
no
pubsub_ack_deadline_seconds Ack deadline for the Pub/Sub subscription (https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.subscriptions) number 60 no
pubsub_maximum_backoff Retry policy maximum backoff for the Pub/Sub subscription (https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.subscriptions) string "600s" no
pubsub_message_retention_duration Message retention for the Pub/Sub subscription (https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.subscriptions) string "86400s" no
pubsub_minimum_backoff Retry policy minimum backoff for the Pub/Sub subscription (https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.subscriptions) string "10s" no
resource The identifier of the GCP Resource to monitor.

The resource can be a project, folder, or organization.

Examples: "projects/my_project-123", "folders/1234567899", "organizations/34739118321"
string n/a yes

Outputs

Name Description
project The ID of the Project in which resources were created
service_account_key A service account key to be passed to the pollers for Pub/Sub and Cloud Monitoring
subscription The Pub/Sub subscription created by this module.
topic The Pub/Sub topic created by this module.

About

Terraform module which streamlines collection from multiple sources within GCP

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • HCL 94.5%
  • Makefile 5.5%