Skip to content

Latest commit

 

History

History
127 lines (104 loc) · 11.7 KB

README.md

File metadata and controls

127 lines (104 loc) · 11.7 KB
Raw

Observe Google Collection

This module creates a log sink, pub/sub topic, and pub/sub subscription needed to facilitate the collection of asset inventory records, metrics and logs from GCP for a given project.

This module also creates a Cloud Function to fetch some data through the GCP REST API.

Usage

Here is an example manifest for collecting data from a Google Cloud organization.

After running terraform apply, data should start flowing into Pub/Sub. In the Observe UI, one would set up the GCP app. The info from the terraform output and terraform output -raw service_account_private_key are needed to set up the GCP App pollers.

provider "google" {
  project = "YOUR_PROJECT_ID"
  region  = "YOUR_DEFAULT_REGION"
}

module "observe_gcp_collection" {
  source  = "observeinc/collection/google"
  name    = "observe"

  resource = "projects/YOUR_PROJECT_ID"
}

output "project" {
  description = "The Pub/Sub project of the subcription (to be passed to the Pub/Sub poller)"
  value       = module.observe_gcp_collection.project
}

# To extract correct value - terraform output -json | jq -r '.subscription.value.name' 
output "subscription" {
  description = "The Pub/Sub subscription created by this module (to be passed to the Pub/Sub poller)"
  value       = module.observe_gcp_collection.subscription
}

# To extract properly formatted string - terraform output -json | jq -r '.service_account_private_key.value'
output "service_account_private_key" {
  description = "A service account key to be passed to the pollers for Pub/Sub and Cloud Monitoring"
  value       = base64decode(module.observe_gcp_collection.service_account_key.private_key)
  sensitive   = true
}

Requirements

Name Version
terraform >= 0.12.21
google >= 4.15

Providers

Name Version
google 4.71.0

Modules

No modules.

Resources

Name Type
google_cloud_scheduler_job.this resource
google_cloudfunctions_function.gcs_function resource
google_cloudfunctions_function.this resource
google_cloudfunctions_function_iam_member.cloud_scheduler resource
google_folder_iam_member.cloudfunction resource
google_logging_folder_sink.this resource
google_logging_organization_sink.this resource
google_logging_project_sink.this resource
google_organization_iam_member.cloudfunction resource
google_project_iam_member.cloudfunction resource
google_project_iam_member.poller resource
google_pubsub_subscription.this resource
google_pubsub_subscription_iam_member.poller_pubsub resource
google_pubsub_topic.this resource
google_pubsub_topic_iam_member.cloudfunction_pubsub resource
google_pubsub_topic_iam_member.sink_pubsub resource
google_service_account.cloud_scheduler resource
google_service_account.cloudfunction resource
google_service_account.poller resource
google_service_account_key.poller resource
google_storage_bucket.this resource
google_storage_bucket_iam_member.bucket_iam resource
google_storage_bucket_iam_member.gcs_function_bucket_iam resource
google_folder.this data source
google_project.this data source

Inputs

Name Description Type Default Required
enable_function Whether to enable the Cloud function bool true no
folder_include_children Whether to include all children Projects of a Folder when collecting logs bool true no
function_available_memory_mb Memory (in MB), available to the function. Default value is 512. Possible values include 128, 256, 512, 1024, etc. number 512 no
function_bucket GCS bucket containing the Cloud Function source code string "observeinc" no
function_disable_logging Whether to disable function logging. bool false no
function_max_instances The limit on the maximum number of function instances that may coexist at a given time. number 5 no
function_object GCS object key of the Cloud Function source code zip file string "google-cloud-functions-v0.3.0-alpha.8.zip" no
function_roles A list of IAM roles to give the Cloud Function. set(string) 
[
  "roles/compute.viewer",
  "roles/iam.serviceAccountViewer",
  "roles/cloudscheduler.viewer",
  "roles/cloudasset.viewer",
  "roles/browser",
  "roles/logging.viewer",
  "roles/monitoring.viewer",
  "roles/storage.objectCreator",
  "roles/storage.objectViewer",
  "roles/storage.objectAdmin",
  "roles/storage.admin"
]
 no
function_schedule_frequency Cron schedule for the job string "0 * * * *" no
function_timeout Timeout (in seconds) for the function. Default value is 300 seconds. Cannot be more than 540 seconds. number 300 no
labels A map of labels to add to resources (https://cloud.google.com/resource-manager/docs/creating-managing-labels)"

Note: Many, but not all, Google Cloud SDK resources support labels.		 map(string) {} no
logging_exclusions Log entries that match any of these exclusion filters will not be exported.

If a log entry is matched by both logging_filter and one of logging_exclusions it will not be exported.

Relevant docs: https://cloud.google.com/logging/docs/reference/v2/rest/v2/billingAccounts.exclusions#LogExclusion		 
list(object({
    name        = string
    description = string
    filter      = string
    disabled    = string
  }))
 [] no
logging_filter An advanced logs filter. The only exported log entries are those that are
in the resource owning the sink and that match the filter.

Relevant docs: https://cloud.google.com/logging/docs/view/building-queries		 string "" no
name Module name. Used as a name prefix. string "observe-collection" no
poller_roles A list of IAM roles to give the Observe poller (through the service account key output). set(string) 
[
  "roles/monitoring.viewer"
]
 no
pubsub_ack_deadline_seconds Ack deadline for the Pub/Sub subscription (https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.subscriptions) number 60 no
pubsub_maximum_backoff Retry policy maximum backoff for the Pub/Sub subscription (https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.subscriptions) string "600s" no
pubsub_message_retention_duration Message retention for the Pub/Sub subscription (https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.subscriptions) string "86400s" no
pubsub_minimum_backoff Retry policy minimum backoff for the Pub/Sub subscription (https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.subscriptions) string "10s" no
resource The identifier of the GCP Resource to monitor.

The resource can be a project, folder, or organization.

Examples: "projects/my_project-123", "folders/1234567899", "organizations/34739118321"		 string n/a yes

Outputs

Name Description
project The ID of the Project in which resources were created
service_account_key A service account key to be passed to the pollers for Pub/Sub and Cloud Monitoring
subscription The Pub/Sub subscription created by this module.
topic The Pub/Sub topic created by this module.