Use Case: POC Serverless Canary Application (frontend/backend/persistence) as a Profile 3 LZ workload with PSC, PSA and VPC-SC

[fmichaelobrien]

202401 update

• https://cloud.google.com/architecture/serverless-blueprint
• HCLS specific

Use Cases

• the uc/customer-app drives the lz architecture subset. In this case we will start with 1 or more serverless use cases around profile 3.
• docai, biometric telemetry, GPS tracking, event stream, live video ML/AI intelligence
• see Canary onboarding: SaaS use case: Add KCC/Gcloud deployment options limited roles around BigQuery/AutoML/DocumentAI #220 for DocAI use case - https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/canary/solutions/document-processing/gcloud
• See LZ V1 traffic generation app https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/blob/main/docs/google-cloud-landingzone-traffic-generation.md
• See earlier V1 architecture discussions https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/dev/solutions/landing-zone/architecture.md
• See V2 architecture diagrams at https://docs.google.com/presentation/d/1Ztqn7G2rWFlssVwC7r5Xl-WFSFCbVo6MoHJffrXScmw/edit#slide=id.g25262d569c1_0_198
• See GCP network-patterns share at https://docs.google.com/presentation/d/13sjT2tJ4yLIYGRREE3wBrylB1OvcEMpKdquVuJB_nX4/edit?resourcekey=0-N3DruQaiutFvZ98HTT7-vQ#slide=id.g1154b3b950f_2_3458
• see Guardrails V1 https://github.com/canada-ca/cloud-guardrails/blob/master/EN/00_Applicable-Scope.md#applicability-of-guardrails-to-cloud-usage-profiles
• see Guardrails V2 https://github.com/CloudLandingZone/cloud-guardrails-v2

Serverless Reference Architecture

Requirements: LZ workload Canary for Profile 3

• P3 specific - we run a selected subset of all packages (armor yes, IDS/NGFW perimeter and extensive hub/spoke - no) for example

• a VPC Service Control egress perimeter is involved

• • 2+ microservices/containers - to exercise ingress/egress tracking

• VPC Serverless Connector

• No NGFW for east-west traffic

• Armor attached to an LB (off the GFE)

• SWP - Secure Web Proxy

• External Cloud Load Balancer - https://cloud.google.com/load-balancing/docs/https

• Artifact Registry and cloud build pipelines will work with the vpc-sc

• hierarchical firewall policies at the 5 org/folder/vpc/region levels

• the ALB in cloud run is not the Global LB

• cloud run will use either a SQL proxy or vpc connector for private IP based cloud SQL connections

• Ideally Guardrails V2 over V1

• https://github.com/canada-ca/cloud-guardrails-v2/tree/main/EN

• https://github.com/canada-ca/cloud-guardrails/blob/master/EN/00_Applicable-Scope.md#applicability-of-guardrails-to-cloud-usage-profiles

Design

Reference

Implementation

• add to https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/solutions/experimentation

Package Coverage Required

• https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/solutions/core-landing-zone
• https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/solutions/client-landing-zone
  or
• https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/solutions/experimentation/client-landing-zone
• https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/solutions/experimentation/client-project

Package Coverage Optional

Design Issues

• watch CR Cloud Run service fails to apply update k8s-config-connector#832
• lightweight JS frontend - minimal jquery/ajax - either encapsulated in container 1 or separate cloud functions knative app
• container 1 : backend spring boot REST api (may contain javascript frontend as well)
• container 2: backend Relational or key/value DB - may use Cloud SQL or Spanner

Template
Introduction
• Deliverables
• MVP - Immediate Minimum Viable Product
• Quickstart
• Artifacts
• Requirements
• Features
• Analysis
• API
• Architecture
• Use Cases
• Design
• Design Issues
• DevOps
• Deployment
• Testing
• Security
• Monitoring
• Releases
• Documentation
• Development Log
• Keywords
• References
• Links

reference as well https://github.com/ssc-spc-ccoe-cei/gcp-tier34-template

[obriensystems]

Implementation

• review https://cloud.google.com/architecture/serverless-blueprint

[fmichaelobrien]

https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/dev/solutions/landing-zone/architecture.md#di-25-continuous-delivery-pipelines
ref: #432
ref: #430
ref: #418
ref: #339
ref: #344
ref: #192
ref: #409

[fmichaelobrien]

Verify VPC-SC across 2 service projects in a shared VPC

Goal/Requirements

• prevent exfiltration of data from a cloud storage bucket or bigquery instance in two of the service projects hosting both of these services

Testing

• source-svc can see target-svc selected services and vice versa

• outside-svc - cannot see anything

• outside-host (separate host project and VPC) - cannot see anything

• follow slide 31 of the network-patterns deck https://docs.google.com/presentation/d/13sjT2tJ4yLIYGRREE3wBrylB1OvcEMpKdquVuJB_nX4/edit?resourcekey=0-N3DruQaiutFvZ98HTT7-vQ#slide=id.g1154b3b950f_2_3767

• https://cloud.google.com/vpc-service-controls/docs/set-up-service-perimeter

VPC Service control zone must include both service projects
Inventory

• canary-host host project
• canary-source-svc project
• canary-target-svc project
• vm-docker-source
• vm-docker-target

• enable compute api on all 3 projects
  In setup shared VPC - enable host on canary-host-ol -but first make a custom VPC and delete the default VPC

gcloud compute networks create dev --project=canary-host-ol --subnet-mode=custom --mtu=1460 --bgp-routing-mode=regional && gcloud compute networks subnets create dev-sn --project=canary-host-ol --range=10.10.0.0/24 --stack-type=IPV4_ONLY --network=dev --region=northamerica-northeast1 --enable-private-ip-google-access && gcloud compute firewall-rules create dev-allow-custom --project=canary-host-ol --network=projects/canary-host-ol/global/networks/dev --description=Allows\ connection\ from\ any\ source\ to\ any\ instance\ on\ the\ network\ using\ custom\ protocols. --direction=INGRESS --priority=65534 --source-ranges=10.10.0.0/24 --action=ALLOW --rules=all && gcloud compute firewall-rules create dev-allow-icmp --project=canary-host-ol --network=projects/canary-host-ol/global/networks/dev --description=Allows\ ICMP\ connections\ from\ any\ source\ to\ any\ instance\ on\ the\ network. --direction=INGRESS --priority=65534 --source-ranges=0.0.0.0/0 --action=ALLOW --rules=icmp && gcloud compute firewall-rules create dev-allow-rdp --project=canary-host-ol --network=projects/canary-host-ol/global/networks/dev --description=Allows\ RDP\ connections\ from\ any\ source\ to\ any\ instance\ on\ the\ network\ using\ port\ 3389. --direction=INGRESS --priority=65534 --source-ranges=0.0.0.0/0 --action=ALLOW --rules=tcp:3389 && gcloud compute firewall-rules create dev-allow-ssh --project=canary-host-ol --network=projects/canary-host-ol/global/networks/dev --description=Allows\ TCP\ connections\ from\ any\ source\ to\ any\ instance\ on\ the\ network\ using\ port\ 22. --direction=INGRESS --priority=65534 --source-ranges=0.0.0.0/0 --action=ALLOW --rules=tcp:22

share 2 svc projects

create vpc service control access policy at the org level

grant uses to the shared subnets

add projects to the perimeter

add resources to the perimeter to protect

Create VMs in subnets

gcloud compute instances create source-svc --project=canary-source-svc-ol --zone=northamerica-northeast1-a --machine-type=e2-small --network-interface=network-tier=PREMIUM,stack-type=IPV4_ONLY,subnet=projects/canary-host-ol/regions/northamerica-northeast1/subnetworks/dev-sn --maintenance-policy=MIGRATE --provisioning-model=STANDARD --service-account=531569623787-compute@developer.gserviceaccount.com --scopes=https://www.googleapis.com/auth/cloud-platform --create-disk=auto-delete=yes,boot=yes,device-name=source-svc,image=projects/debian-cloud/global/images/debian-11-bullseye-v20230814,mode=rw,size=10,type=projects/canary-source-svc-ol/zones/us-central1-a/diskTypes/pd-balanced --no-shielded-secure-boot --shielded-vtpm --shielded-integrity-monitoring --labels=goog-ec-src=vm_add-gcloud --reservation-affinity=any

gcloud compute instances create ext-svc --project=canary-ext-svc-ol --zone=northamerica-northeast1-a --machine-type=e2-small --network-interface=network-tier=PREMIUM,stack-type=IPV4_ONLY,subnet=projects/canary-ext-host-ol/regions/northamerica-northeast1/subnetworks/dev-sn --maintenance-policy=MIGRATE --provisioning-model=STANDARD --service-account=910458337292-compute@developer.gserviceaccount.com --scopes=https://www.googleapis.com/auth/cloud-platform --create-disk=auto-delete=yes,boot=yes,device-name=ext-svc,image=projects/debian-cloud/global/images/debian-11-bullseye-v20230814,mode=rw,size=10,type=projects/canary-ext-svc-ol/zones/us-central1-a/diskTypes/pd-balanced --no-shielded-secure-boot --shielded-vtpm --shielded-integrity-monitoring --labels=goog-ec-src=vm_add-gcloud --reservation-affinity=any

create a bucket with files to be accessed both inside and outside the perimeter in the source-svc project

Test access from ext-svc before/after perimeter is set

[obriensystems]

check VM access to bucket inside future perimeter

michael@source-svc:~$ gsutil ls gs://canary-source-svc-ol/
gs://canary-source-svc-ol/Easter Brunch Global Tables Takehome Notes 2023 Letter.pdf

michael@ext-svc:~$ gsutil ls
michael@ext-svc:~$ 
michael@ext-svc:~$ gsutil ls gs://canary-source-svc-ol/
AccessDeniedException: 403 [email protected] does not have storage.objects.list access to the Google Cloud Storage bucket. Permission 'storage.objects.list' denied on resource (or it may not exist).

add the VM service account as reader

visible

michael@ext-svc:~$ gsutil ls gs://canary-source-svc-ol/
gs://canary-source-svc-ol/Easter Brunch Global Tables Takehome Notes 2023 Letter.pdf

set VPC service perimeter around the bucket specific to the source service project

add 2 projects to the perimeter

add cloud storage to the perimeter

checking - the reverse of what I expected - source is blocked, ext is not

michael@source-svc:~$ gsutil ls gs://canary-source-svc-ol/
gs://canary-source-svc-ol/Easter Brunch Global Tables Takehome Notes 2023 Letter.pdf

set projects in VPC-SC
michael@source-svc:~$ gsutil ls gs://canary-source-svc-ol/
AccessDeniedException: 403 Request is prohibited by organization's policy. vpcServiceControlsUniqueIdentifier: su8wcnumiqi2Urdl9NTTaymJAk_9AsbDaQef3wG8-wn_9zZwBuCSeQ

michael@ext-svc:~$ gsutil ls gs://canary-source-svc-ol/
gs://canary-source-svc-ol/Easter Brunch Global Tables Takehome Notes 2023 Letter.pdf

however turning on ext on and off in the perimeter sets the VPC SC flag - but I cannot get it reset

michael@ext-svc:~$ gsutil ls gs://canary-source-svc-ol/
AccessDeniedException: 403 Request is prohibited by organization's policy. vpcServiceControlsUniqueIdentifier: bPOAHVzoAq-3TnXxWL2yUnkOa8oLN9ub1sCEvKKqhbydgmzhWuYBV

test reduction only to target

delete perimeter but keep default policy - to reset

recreate perimeter

michael@source-svc:~$ gsutil ls gs://canary-source-svc-ol/
AccessDeniedException: 403 Request is prohibited by organization's policy. vpcServiceControlsUniqueIdentifier: i7PWPHxZaJjhRx5b_zD6ZoCZQ1TVRU8CC2-CwQU77xBceJ77wse7XA

michael@ext-svc:~$ gsutil ls gs://canary-source-svc-ol/
gs://canary-source-svc-ol/Easter Brunch Global Tables Takehome Notes 2023 Letter.pdf

VPC-SC: There are additional ingress and VPC owning project inclusion attributes to VPC Service Control - Perimeters in the policy such as per vpc/perimeter. Triaging VPC-SC config using a gs bucket in the notes starting at #418 (comment)

reference
https://cloud.google.com/vpc-service-controls/docs/overview#isolate

Missing the ingress rule and we also need the perimeter all the way around the host project as well - or else we use bridges

after the ingress rule we have the proper internal gsutil usage and blocked external gsutil usage

michael@ext-svc:~$ gsutil ls gs://canary-source-svc-ol/
AccessDeniedException: 403 Request is prohibited by organization's policy. vpcServiceControlsUniqueIdentifier: 5hL_K1kXZRP8W1icoLReQSeY3DlkWplevoH5Ga6YwvESsEC_OVdXBA

michael@source-svc:~$ gsutil ls gs://canary-source-svc-ol/
gs://canary-source-svc-ol/Easter Brunch Global Tables Takehome Notes 2023 Letter.pdf