Make Gateway Discovery work with GKE and kube-dns

[mlavacca]

Problem Statement

https://github.com/Kong/gateway-operator/pull/1261 enabled Gateway Discovery in ControlPlanes with Service DNSStrategy, setting the admin services spec.publishNotReadyAddresses field to true. This means that the admin service endpoints are created before the ready probes of the pod are successful. This implementation allows us to solve the chicken-egg problem (caused by the enablement of the dataplane /status/ready endpoint) that follows:

• the dataplane needs a configuration to get ready
• the controlplane needs a dataplane pod connection to get ready

This solution has been tested and has some problems in GKE

• when kube-dns is used, this solution does not work, because kube-dns needs the pod to be ready to resolve the service-based pod-ip-address.service-name.namespace.svc address
• when cloud-dns is enabled this solution works

Proposed Solution

To make KGO work with GKE and kube-dns, we may want to do the follows:

• introduce a new flag --controlplane-dns-strategy with default value "service" and allowed values {"service", "pod"}
• the controlplane DNS strategy is configured according to the flag's value
• in case the DNS strategy is pod, we need to create a looser certificate with the *.namespace.pod subject instead of *.service-name.namespace.svc
• in case a certificate already existed, we need to replace the old certificate with the new one and restart the dataplane pods to properly mount the new certificate.

Acceptance Criteria

• ControlPlane Gateway Discovery properly works in GKE with kube-dns

[aoktox]

I'm facing same issue with Kong/charts#1068, this workaround is working for me

kind: GatewayConfiguration
apiVersion: gateway-operator.konghq.com/v1beta1
metadata:
  name: kong
  namespace: kong-system
spec:
  <truncated>
  controlPlaneOptions:
    deployment:
      podTemplateSpec:
        spec:
          containers:
            - name: controller
              image: kong/kubernetes-ingress-controller:3.3.1
              env:
                - name: CONTROLLER_GATEWAY_DISCOVERY_DNS_STRATEGY
                  value: ip
                - name: CONTROLLER_KONG_ADMIN_TLS_SKIP_VERIFY
                  value: "true"
<truncated>

[joran-fonjallaz]

Hello @aoktox and @mlavacca,

Could it be that KIC suffers the same issue ? We are running KIC in GKE on preemptible nodes in our non-prod environment and we regularly have Kong gateway pods unhealthy with errors like the follwing from the controller:

performing update for https://10.xxx.xxx.21:8444 failed: failed to detect configuration change: failed to verify kong readiness: making HTTP request: Get "https://10.xxx.xxx.21:8444/status": context deadline exceeded

or

error retrieveing schema for plugin foo: making HTTP request: Get "https://10.xxx.xxx.21:8444/schemas/plugins/foo": dial tcp 10.xxx.xxx.21:8444: i/o timeout"

The gateway never gets the config pushed to them and the gateway pods remain unhealthy until we manually restarted them all (including the controller).

This weekend, we just had an incident in production after that GKE replaced an unhealthy node. The KIC failed to push the config to the Kong pods that were running on that node after they restarted.

Our versions

non-prod

• GKE 1.30
• KIC 3.3.1
• Kong 3.7

prod

• GKE 1.29
• KIC 3.3.1
• Kong 3.8

[pmalek]

@joran-fonjallaz What you're describing is a different issue: seems like Kong/kubernetes-ingress-controller#6567.