Controller: multiple replicas fail: invalid spiffe path

[Poweranimal]

Overview of the Issue

Running the consul-api-gateway-controller with more than 1 replica and deploying a Gateway with spec.listeners[*].protocol = "HTTPS" causes the following error on all consul-api-gateway-controller replicas that have not acquired the leader lease:

2022-10-13T14:59:47.784Z [WARN]  envoy/middleware.go:101: consul-api-gateway-server.sds-server: gateway not found: namespace="" service=consul-api-gateway
2022-10-13T14:59:47.784Z [ERROR] envoy/middleware.go:89: consul-api-gateway-server.sds-server: error parsing spiffe path, skipping: error="invalid spiffe path" path=""
2022-10-13T15:00:00.885Z [WARN]  envoy/middleware.go:101: consul-api-gateway-server.sds-server: gateway not found: namespace="" service=consul-api-gateway
2022-10-13T15:00:00.885Z [ERROR] envoy/middleware.go:89: consul-api-gateway-server.sds-server: error parsing spiffe path, skipping: error="invalid spiffe path" path=""

Additionally, this has the effect that all replicas of consul-api-gateway that are in contact with the failing consul-api-gateway-controller replicas reject ingress traffic.

1. When creating a gateway with the following configuration:

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: Gateway
metadata:
  name: consul-api-gateway
  namespace: consul
spec:
  gatewayClassName: consul-api-gateway
  listeners:
    - allowedRoutes:
        kinds:
          - kind: HTTPRoute
        namespaces:
          from: Selector
          selector:
            matchLabels:
              shared-gateway-access: "true"
      name: prod-api
      port: 8080
      protocol: HTTPS
      tls:
        certificateRefs:
          - kind: Secret
            name: consul-api-gateway
        mode: Terminate

2. View error in consul-api-gateway-controller

2022-10-13T15:00:00.885Z [WARN]  envoy/middleware.go:101: consul-api-gateway-server.sds-server: gateway not found: namespace="" service=consul-api-gateway
2022-10-13T15:00:00.885Z [ERROR] envoy/middleware.go:89: consul-api-gateway-server.sds-server: error parsing spiffe path, skipping: error="invalid spiffe path" path=""

Provide log files from the gateway controller component by providing output from kubectl logs from the pod and container that is surfacing the issue.

Logs

2022-10-13T14:53:24.540Z [INFO]  grpc/logging.go:40: consul-api-gateway-server.sds-server: [core][Server #1] Server created
2022-10-13T14:53:24.540Z [INFO]  grpc/logging.go:40: consul-api-gateway-server.sds-server: [core][Server #1 ListenSocket #2] ListenSocket created
2022-10-13T14:53:24.540Z [INFO]  k8s/logger.go:30: consul-api-gateway-server.controller-runtime: Starting server: addr=[::]:8081 kind="health probe" info="Starting server"
2022-10-13T14:53:24.541Z [INFO]  k8s/logger.go:30: consul-api-gateway-server.kubernetes-client: attempting to acquire leader lease consul/consul-api-gateway.consul.hashicorp.com...
:
  info=
  | attempting to acquire leader lease consul/consul-api-gateway.consul.hashicorp.com...
  
2022-10-13T14:53:24.541Z [INFO]  k8s/logger.go:30: consul-api-gateway-server.controller-runtime: Starting server: addr=[::]:8080 kind=metrics path=/metrics info="Starting server"
2022-10-13T14:54:00.671Z [WARN]  envoy/middleware.go:101: consul-api-gateway-server.sds-server: gateway not found: namespace="" service=consul-api-gateway
2022-10-13T14:54:00.671Z [ERROR] envoy/middleware.go:89: consul-api-gateway-server.sds-server: error parsing spiffe path, skipping: error="invalid spiffe path" path=""
2022-10-13T14:54:00.985Z [WARN]  envoy/middleware.go:101: consul-api-gateway-server.sds-server: gateway not found: namespace="" service=consul-api-gateway
2022-10-13T14:54:00.985Z [ERROR] envoy/middleware.go:89: consul-api-gateway-server.sds-server: error parsing spiffe path, skipping: error="invalid spiffe path" path=""

If not already included, please provide the following:

• consul-api-gateway version: v0.4.0
• Kubernetes version: v1.23.x
• Consul Server version: v1.13.x