Google Cloud Collector

Plugin to collect Google Cloud

SpaceONE's plugin-google-cloud-inven-collector is a convenient tool to get cloud service data from Google Cloud platform.

Find us also at Dockerhub

Please contact us if you need any further information. ([email protected])

---

Google Service Endpoint (in use)

There is an endpoints used to collect resources information of GCP. Endpoint of served GCP is a URL consisting of a service code.

https://[service-code].googleapis.com

We use dozens of endpoints because we collect information from many services.

Service list

The following is a list of services being collected and service code information.

No.	Service name	Service Code
1	Compute Engine	compute
2	Networking	compute
3	Cloud SQL	sqladmin
4	Storage	storage
5	BigQuery	bigquery
6	Cloud Pub/Sub	pubsub
7	Cloud Functions	cloudfunctions
8	Recommender	recommender

If you want to know the detailed service endpoint, please check the content details below.

Content details

• Table of Contents
  • Compute Engine
    • VM Instance
    • Instance Template
    • Instance Group
    • Machine Images
    • Disk
    • Snapshot
  • Networking
    • VPC Network
    • Route
    • External IP Address
    • Firewall
    • LoadBalancing
  • Cloud SQL
    • Instance
  • Storage
    • Buckets
  • BigQuery
    • SQLWorkspace
  • Cloud Pub/Sub
    • Topic
    • Subscription
    • Snapshot
    • Schema
  • Cloud Fuctions
    • Function
  • Recommender
    • Recommendation
    • Insight
  • Options
    • CloudServiceType
    • ServiceCodeMapper

Authentication Overview

Registered service account on SpaceONE must have certain permissions to collect cloud service data Please, set authentication privilege for followings:

Compute Engine

• VM Instance

  • Scopes

    • https://www.googleapis.com/auth/compute
    • https://www.googleapis.com/auth/cloud-platform

  • IAM

    • compute.zones.list
    • compute.regions.list
    • compute.instances.list
    • compute.machineTypes.list
    • compute.urlMaps.list
    • compute.backendServices.list
    • compute.disks.list
    • compute.diskTypes.list
    • compute.autoscalers.list
    • compute.images.list
    • compute.subnetworks.list
    • compute.regionUrlMaps.list
    • compute.backendServices.list
    • compute.targetPools.list
    • compute.forwardingRules.list

  • Service Endpoint

    • https://compute.googleapis.com/compute/v1/projects/{project}/aggregated/{resource_name}
    • https://compute.googleapis.com/compute/v1/projects/{project}/global/{resource_name}
    • https://compute.googleapis.com/compute/v1/projects/{project}/zone/{zone}/{resource_name}
    • https://compute.googleapis.com/compute/v1/projects/{project}/regions/{region}/{resource_name}

• Instance Template

  • Scopes

    • https://www.googleapis.com/auth/compute
    • https://www.googleapis.com/auth/cloud-platform

  • IAM

    • compute.instanceGroupManagers.list
    • compute.machineTypes.list
    • compute.disks.list
    • compute.instanceTemplates.list

  • Service Endpoint

    • https://compute.googleapis.com/compute/v1/projects/{project}/aggregated/{resource_name}
    • https://compute.googleapis.com/compute/v1/projects/{project}/global/{resource_name}
    • https://compute.googleapis.com/compute/v1/projects/{project}/zone/{zone}/{resource_name}
    • https://compute.googleapis.com/compute/v1/projects/{project}/regions/{region}/{resource_name}

• Instance Group

  • Scopes

    • https://www.googleapis.com/auth/compute
    • https://www.googleapis.com/auth/cloud-platform

  • IAM

    • compute.instanceGroups.list
    • compute.instanceGroupManagers.list
    • compute.instances.list
    • compute.autoscalers.list
    • compute.instanceTemplates.list

  • Service Endpoint

    • https://compute.googleapis.com/compute/v1/projects/{project}/aggregated/{resource_name}
    • https://compute.googleapis.com/compute/v1/projects/{project}/global/{resource_name}
    • https://compute.googleapis.com/compute/v1/projects/{project}/zone/{zone}/{resource_name}
    • https://compute.googleapis.com/compute/v1/projects/{project}/regions/{region}/{resource_name}

• Machine Images

  • Scopes

    • https://www.googleapis.com/auth/compute
    • https://www.googleapis.com/auth/cloud-platform

  • IAM

    • compute.machineImages.list
    • compute.machineTypes.list
    • compute.disks.list
    • compute.images.list

  • Service Endpoint

    • https://compute.googleapis.com/compute/v1/projects/{project}/aggregated/{resource_name}
    • https://compute.googleapis.com/compute/v1/projects/{project}/global/{resource_name}
    • https://compute.googleapis.com/compute/v1/projects/{project}/zone/{zone}/{resource_name}
    • https://compute.googleapis.com/compute/v1/projects/{project}/regions/{region}/{resource_name}

• Disk

  • Scopes

    • https://www.googleapis.com/auth/compute
    • https://www.googleapis.com/auth/cloud-platform

  • IAM

    • compute.disks.list
    • compute.resourcePolicies.list

  • Service Endpoint

    • https://compute.googleapis.com/compute/v1/projects/{project}/aggregated/{resource_name}
    • https://compute.googleapis.com/compute/v1/projects/{project}/global/{resource_name}
    • https://compute.googleapis.com/compute/v1/projects/{project}/zone/{zone}/{resource_name}
    • https://compute.googleapis.com/compute/v1/projects/{project}/regions/{region}/{resource_name}

• Snapshot

  • Scopes

    • https://www.googleapis.com/auth/compute
    • https://www.googleapis.com/auth/cloud-platform

  • IAM

    • compute.snapshots.list
    • compute.resourcePolicies.list
    • compute.disks.list

  • Service Endpoint

    • https://compute.googleapis.com/compute/v1/projects/{project}/aggregated/{resource_name}
    • https://compute.googleapis.com/compute/v1/projects/{project}/global/{resource_name}
    • https://compute.googleapis.com/compute/v1/projects/{project}/zone/{zone}/{resource_name}
    • https://compute.googleapis.com/compute/v1/projects/{project}/regions/{region}/{resource_name}

Networking

• VPC Network

  • Scopes

    • https://www.googleapis.com/auth/compute
    • https://www.googleapis.com/auth/cloud-platform

  • IAM

    • compute.instances.list
    • compute.forwardingRules.list
    • compute.networks.list
    • compute.addresses.list
    • compute.globalAddresses.list
    • compute.subnetworks.list
    • compute.firewalls.list
    • compute.routes.list

  • Service Endpoint

    • https://compute.googleapis.com/compute/v1/projects/{project}/aggregated/{resource_name}
    • https://compute.googleapis.com/compute/v1/projects/{project}/global/{resource_name}
    • https://compute.googleapis.com/compute/v1/projects/{project}/zone/{zone}/{resource_name}
    • https://compute.googleapis.com/compute/v1/projects/{project}/regions/{region}/{resource_name}

• Route

  • Scopes

    • https://www.googleapis.com/auth/compute
    • https://www.googleapis.com/auth/cloud-platform

  • IAM

    • compute.routes.list
    • compute.instances.list

  • Service Endpoint

    • https://compute.googleapis.com/compute/v1/projects/{project}/aggregated/{resource_name}
    • https://compute.googleapis.com/compute/v1/projects/{project}/global/{resource_name}
    • https://compute.googleapis.com/compute/v1/projects/{project}/zone/{zone}/{resource_name}
    • https://compute.googleapis.com/compute/v1/projects/{project}/regions/{region}/{resource_name}

• External IP Address

  • Scopes

    • https://www.googleapis.com/auth/compute
    • https://www.googleapis.com/auth/cloud-platform

  • IAM

    • compute.instances.list
    • compute.forwardingRules.list
    • compute.addresses.list

  • Service Endpoint

    • https://compute.googleapis.com/compute/v1/projects/{project}/aggregated/{resource_name}
    • https://compute.googleapis.com/compute/v1/projects/{project}/global/{resource_name}
    • https://compute.googleapis.com/compute/v1/projects/{project}/zone/{zone}/{resource_name}
    • https://compute.googleapis.com/compute/v1/projects/{project}/regions/{region}/{resource_name}

• Firewall

  • Scopes

    • https://www.googleapis.com/auth/compute
    • https://www.googleapis.com/auth/cloud-platform

  • IAM

    • compute.instances.list
    • compute.firewalls.list

  • Service Endpoint

    • https://compute.googleapis.com/compute/v1/projects/{project}/aggregated/{resource_name}
    • https://compute.googleapis.com/compute/v1/projects/{project}/global/{resource_name}
    • https://compute.googleapis.com/compute/v1/projects/{project}/zone/{zone}/{resource_name}
    • https://compute.googleapis.com/compute/v1/projects/{project}/regions/{region}/{resource_name}

• LoadBalancing

  • Scopes

    • https://www.googleapis.com/auth/compute
    • https://www.googleapis.com/auth/cloud-platform

  • IAM

    • compute.urlMaps.list
    • compute.backendBuckets.list
    • compute.backendServices.list
    • compute.targetPools.list
    • compute.forwardingRules.list
    • compute.targetGrpcProxies.list
    • compute.targetHttpProxies.list
    • compute.targetHttpsProxies.list
    • compute.targetGrpcProxies.list
    • compute.healthChecks.list
    • compute.httpHealthChecks.list
    • compute.httpsHealthChecks.list
    • compute.autoscalers.list

  • Service Endpoint

    • https://compute.googleapis.com/compute/v1/projects/{project}/aggregated/{resource_name}
    • https://compute.googleapis.com/compute/v1/projects/{project}/global/{resource_name}
    • https://compute.googleapis.com/compute/v1/projects/{project}/zone/{zone}/{resource_name}
    • https://compute.googleapis.com/compute/v1/projects/{project}/regions/{region}/{resource_name}

Cloud SQL

• Instance

  • Scopes

    • https://www.googleapis.com/auth/cloud-platform
    • https://www.googleapis.com/auth/sqlservice.admin

  • IAM

    • sqladmin.instances.list
    • sqladmin.databases.list
    • sqladmin.users.list
    • sqladmin.backup_runs.list

  • Service Endpoint

    • https://sqladmin.googleapis.com/v1/projects/{project}/{resources}
    • https://sqladmin.googleapis.com/v1/projects/{project}/instances/{instance}/{resources}

Storage

• Bucket

  • IAM

    • storage.buckets.get
    • storage.objects.list
    • storage.objects.getIamPolicy

  • Service Endpoint

    • https://storage.googleapis.com/storage/v1/b/{resource}

BigQuery

• SQLWorkspace

  • IAM

    • bigquery.datasets.get
    • bigquery.tables.get
    • bigquery.tables.list
    • bigquery.jobs.list
    • resourcemanager.projects.get

  • Service Endpoint

    • https://bigquery.googleapis.com/bigquery/v2/projects/{projectId}/{resource}

Pub/Sub

• Topic

  • IAM

    • pubsub.topics.list
    • pubsub.subscriptions.get
    • pubsub.snapshots.get

  • Service Endpoint

    • https://pubsub.googleapis.com/v1/{project}/topics
    • https://pubsub.googleapis.com/v1/{subscription}
    • https://pubsub.googleapis.com/v1/{snapshot}

• Subscription

  • IAM

    • pubsub.subscriptions.list

  • Service Endpoint

    • https://pubsub.googleapis.com/v1/{project}/subscriptions

• Snapshot

  • IAM

    • pubsub.snapshots.list

  • Service Endpoint

    • https://pubsub.googleapis.com/v1/{project}/snapshots

• Schema

  • IAM

    • pubsub.schemas.list

  • Service Endpoint

    • https://pubsub.googleapis.com/v1/{parent}/schemas

Functions

• Function

  • IAM

    • 1st Generation
      • cloudfunctions.functions.list
      • storage.bucket.get
    • 2nd Generation
      • cloudfunctions.functions.list
      • storage.bucket.get
      • eventarc.providers.list

  • Service Endpoint

    • 1st Generation
      • https://cloudfunctions.googleapis.com/v1/{parent=projects/*/locations/*}/functions
      • https://storage.googleapis.com/storage/v1/b/{bucket}
    • 2nd Generation
      • https://cloudfunctions.googleapis.com/v2/{parent=projects/*/locations/*}/functions
      • https://storage.googleapis.com/storage/v1/b/{bucket}
      • https://eventarc.googleapis.com/v1/{parent=projects/*/locations/*}/providers

Recommender

• Recommendation & Insight

  • IAM

    • cloudasset.assets.listResource
    • cloudasset.assets.listIamPolicy
    • cloudasset.assets.listOrgPolicy
    • cloudasset.assets.listAccessPolicy
    • cloudasset.assets.listOSInventories
    • recommender.*.get
    • recommender.*.list

  • Recommendation Service Endpoint

    • https://recommender.googleapis.com/v1/{name=projects/*/locations/*/recommenders/*/recommendations/*}

  • Insight Service Endpoint

    • https://cloudasset.googleapis.com/v1/{parent=*/*}/assets
    • https://recommender.googleapis.com/v1/{parent=projects/*/locations/*/insightTypes/*}/insights

---

Options

Cloud Service Type : Specify what to collect

If cloud_service_types is added to the list elements in options, only the specified cloud service type is collected. By default, if cloud_service_types is not specified in options, all services are collected.

The cloud_service_types items that can be specified are as follows.

{
    "cloud_service_types": [
    'ComputeEngine'
    'CloudSQL',
    'BigQuery',
    'CloudStorage',
    'Networking'
    ]
}

How to update plugin information using spacectl is as follows. First, create a yaml file to set options.

> cat update_collector.yaml
---
collector_id: collector-xxxxxxx
options:
  cloud_service_types:
    - CloudSQL
    - VPCNetwork

Update plugin through spacectl command with the created yaml file.

> spacectl exec update_plugin inventory.Collector -f update_collector.yaml

Service Code Mapper : Update service code in Cloud Service Type.

If service_code_mapper is in options, You can replace the existed service code into new value one. The default service code is listed below service code list

{
    "service_code_mappers": {
        "Compute Engine": "Your new service code",
        "Cloud SQL": "Your new service code",
    }
}

Custom Asset URL : Update ASSET_URL in Cloud Service Type.

If custom_asset_url is in options, You can change it to an asset_url that users will use instead of the default asset_url.
The default ASSET_URL in cloud_service_conf is https://spaceone-custom-assets.s3.ap-northeast-2.amazonaws.com/console-assets/icons/cloud-services/google_cloud.

{
    "custom_asset_url": "https://xxxxx.spaceone.dev/icon/google"
}

---