GKE: Using pod dns names (service scoped) doesn't work on GKE thus preventing ControlPlane -> DP's Admin API traffic

[pmalek]

Current Behavior

Currently when ControlPlane creates Admin API address for a particular Gateway it uses Pods A record (ref: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#a-aaaa-records-1) which works on kind (which uses coredns) but doesn't work on GKE (which uses kube-dns).

Related piece of code:

gateway-operator/controllers/controlplane_controller_utils.go

Lines 166 to 169 in b1bda5c

	func controllerKongAdminURL(podIP, adminServiceName, podNamespace string) string {
	return fmt.Sprintf("https://%s.%s.%s.svc:%d",
	strings.ReplaceAll(podIP, ".", "-"), adminServiceName, podNamespace, dataplaneutils.DefaultKongAdminPort)
	}

.

This effectively make the ControlPlane -> DataPlane's Admin API traffic to not work on GKE.

The following is a log from KIC using GD on GKE but the same happens when using operator with ControlPlane bound to a DataPlane (e.g. via a Gateway object).

time="2023-05-24T12:51:26Z" level=info msg="Retrying kong admin api client call after error" error="making HTTP request: Get \"https://10-12-56-9.kong-admin.kong.svc:8444/\": dial tcp: lookup 10-12-56-9.kong-admin.kong.svc on 10.12.48.42:53: no such host" logger=setup retries=119/120

This is important to think about this in the context of KIC and KGO collaboration because:

• gateway-operator/controllers/controlplane_controller_utils.go

  Lines 166 to 169 in b1bda5c

  	func controllerKongAdminURL(podIP, adminServiceName, podNamespace string) string {
  	return fmt.Sprintf("https://%s.%s.%s.svc:%d",
  	strings.ReplaceAll(podIP, ".", "-"), adminServiceName, podNamespace, dataplaneutils.DefaultKongAdminPort)
  	}

  creates Admin API addresses
• https://github.com/Kong/gateway-operator/blob/156ab31f27779ccff92686001519fdd2d0193799/controllers/dataplane_controller_reconciler_utils.go#L168-L176 creates a single, wildcard certificate per DataPlane for fmt.Sprintf("*.%s.%s.svc", adminServiceName, dataplane.Namespace) subject
• KIC's Gateway Discovery, when used for Release KGO OSS v1.2 #8, will ideally use the same addresses which will work with TLS certificates (generated above)

Related KIC issue: Kong/kubernetes-ingress-controller#4065

Expected Behavior

ControlPlane <-> DataPlane traffic works on GKE.

Operator Version

9a7d17a

[shaneutt]

If the resolution to this issue results in there still being since the resolution of this issue will result in there being different options needing to be applied based on cloud provider, we should consider employing https://github.com/Kong/gateway-operator/issues/867 to automate this as a follow-up.

[pmalek]

The problem with this on GKE is that:

kube-dns only creates DNS records for Services that have Endpoints.

https://cloud.google.com/kubernetes-engine/docs/how-to/kube-dns#service-dns-records.

Which would require us to not only use different methods of creating URLs, but also different Service types because currently we use a headless Service which due to the above limitation cannot work on GKE.

Hence we'd need to use a ClusterIP Service.

[gAmUssA]

Found a «workaround» in one of [SO answers](https://stackoverflow.com/questions/55122234/installing-coredns-on-gke.
Basically, installing CoreDNS on GKE instead of kube-dns

git clone https://github.com/coredns/deployment.git
cd deployment/kubernetes
./deploy.sh > corends-deployment.yaml
kubectl apply -f corends-deployment.yaml
kubectl scale --replicas=0 deployment/kube-dns-autoscaler --namespace=kube-system
kubectl scale --replicas=0 deployment/kube-dns --namespace=kube-system

This seems to work for me on GKE

Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.27.3-gke.100

[pmalek]

Created an issue for kube-dns to request providing service scoped dns names: kubernetes/dns#633