Consul does not ignore services with consul.hashicorp.com/service-ignore=true

[kbreit-insight]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview of the Issue

I am deploying RabbitMQ and integrating it into a Consul service mesh. RabbitMQ ships with multiple services pointing towards it server. The consul-connect-inject-init init container is giving errors that there are multiple services pointing to the pod and I should set the consul.hashicorp.com/service-ignore=true label. All but one of the services has this label assigned but it's still giving the error.

2024-07-09T21:09:29.076Z [ERROR] There are multiple Consul services registered for this pod when there must only be one. Check if there are multiple Kubernetes services selecting this pod and add the label `consul.hashicorp.com/service-ignore: "true"` to all services except the one used by Consul for handling requests.
2024-07-09T21:09:29.076Z [ERROR] Timed out waiting for service registration: error="did not find correct number of services, found: 4, services: &{Node:0xc001095050 Services:[0xc000bf4ea0 0xc000bf50e0 0xc000bf5320 0xc000bf57a0]}"
2024-07-09T21:09:29.076Z [INFO]  consul-server-connection-manager: stopping
2024-07-09T21:09:29.078Z [INFO]  consul-server-connection-manager: ACL auth method logout succeeded

on aiocp2(rabbitmq) pcp-dev-sandbox/kbreit/crossplane on  main [!?]
➜ kubectl get svc --show-labels
NAME             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                        AGE    LABELS
cluster          ClusterIP   172.30.237.192   <none>        15672/TCP,15692/TCP,5672/TCP   63d    app.kubernetes.io/component=rabbitmq,app.kubernetes.io/name=cluster,app.kubernetes.io/part-of=rabbitmq,consul.hashicorp.com/service-ignore=true
cluster-nodes    ClusterIP   None             <none>        4369/TCP,25672/TCP             63d    app.kubernetes.io/component=rabbitmq,app.kubernetes.io/name=cluster,app.kubernetes.io/part-of=rabbitmq,consul.hashicorp.com/service-ignore=true
cluster-server   ClusterIP   172.30.99.122    <none>        15672/TCP,15692/TCP,5672/TCP   7h3m   app.kubernetes.io/component=rabbitmq,app.kubernetes.io/name=cluster-server,app.kubernetes.io/part-of=rabbitmq

Reproduction Steps

Deploy RabbitMQ operator using the instructions at https://www.rabbitmq.com/kubernetes/operator/operator-overview. Alternatively, a statefulset could be deployed with a similar configuration for POC, but I haven't tested it with that setup. Deploy the operator controller with the following manifest. Note, some of this may be populated automatically.

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  annotations:
  labels:
    consul.hashicorp.com/service-ignore: "true"
  name: cluster
  namespace: rabbitmq
spec:
  delayStartSeconds: 30
  image: rabbitmq:3.13.1-management
  override:
    statefulSet:
      spec:
        template:
          metadata:
            annotations:
              consul.hashicorp.com/connect-inject: "true"
              consul.hashicorp.com/transparent-proxy: "true"
              k8s.v1.cni.cncf.io/networks: '[{ "name":"consul-cni","namespace": "rabbitmq"
                }]'
          spec:
            containers: []
            securityContext: {}
  persistence:
    storage: 10Gi
  rabbitmq: {}
  replicas: 1
  service:
    type: ClusterIP
  terminationGracePeriodSeconds: 604800
  tls: {}

Logs

Expected behavior

The pod should come up as it should not see the services which have this label.

Environment details

• consul-k8s version: v1.18.2
• values.yaml used to deploy the helm chart:

global:
  name: consul
  datacenter: aiocp2
  image: registry.connect.redhat.com/hashicorp/consul:1.18.2-ubi
  imageK8S: registry.connect.redhat.com/hashicorp/consul-k8s-control-plane:1.4.2-ubi
  imageConsulDataplane: registry.connect.redhat.com/hashicorp/consul-dataplane:1.4.2-ubi
  openshift:
    enabled: true
  tls:
    enabled: true
    caCert:
      secretName: consul-hubs-ca-cert
      secretKey: tls.crt
  peering:
    enabled: true
  acls:
    manageSystemACLs: true
ui:
  enabled: true
  ingress:
    enabled: true
    pathType: ImplementationSpecific
    hosts:
      - host: consul-ui.apps.aiocp2.example.com
        paths:
          - ""
    annotations: |
      "route.openshift.io/termination": "passthrough"
server:
  replicas: 1
  bootstrapExpect: 1
  disruptionBudget:
    enabled: true
    maxUnavailable: 0
  exposeService:
    enabled: true
    type: LoadBalancer
    annotations: |
      metallb.universe.tf/loadBalancerIPs: 10.204.90.103
  serverCert:
    secretName: consul-hubs-server-cert
meshGateway:
  enabled: true
  wanAddress:
    source: "Static"
    static: "aiocp2-mesh-gateway.example.com"
  service:
    annotations: |
      metallb.universe.tf/loadBalancerIPs: 10.204.90.104
connect:
  enabled: true
connectInject:
  enabled: true
  default: false
  transparentProxy:
    defaultEnabled: true
  cni:
    enabled: true
    multus: true
    cniBinDir: /var/lib/cni/bin
    cniNetDir: /etc/kubernetes/cni/net.d
dns:
  enabled: true
  enableRedirection: true
  type: LoadBalancer
  annotations: |
    metallb.universe.tf/loadBalancerIPs: 10.204.90.105

• Kubernetes version: v1.28.10
• Cloud Provider (If self-hosted, the Kubernetes provider utilized): OpenShift 4.15.17 on vSphere
• Networking CNI plugin in use: Multus

[kbreit-insight]

I tried to set my cluster-server service to have only a single port and it continued to exhibit this behavior.

2024-07-11T16:06:32.475Z [ERROR] There are multiple Consul services registered for this pod when there must only be one. Check if there are multiple Kubernetes services selecting this pod and add the label `consul.hashicorp.com/service-ignore: "true"` to all services except the one used by Consul for handling requests.

on aiocp2(rabbitmq) ~
➜ kubectl get svc --show-labels
NAME             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                        AGE    LABELS
cluster          ClusterIP   172.30.237.192   <none>        15672/TCP,15692/TCP,5672/TCP   64d    app.kubernetes.io/component=rabbitmq,app.kubernetes.io/name=cluster,app.kubernetes.io/part-of=rabbitmq,consul.hashicorp.com/service-ignore=true
cluster-nodes    ClusterIP   None             <none>        4369/TCP,25672/TCP             64d    app.kubernetes.io/component=rabbitmq,app.kubernetes.io/name=cluster,app.kubernetes.io/part-of=rabbitmq,consul.hashicorp.com/service-ignore=true
cluster-server   ClusterIP   172.30.99.122    <none>        5672/TCP                       2d1h   app.kubernetes.io/component=rabbitmq,app.kubernetes.io/name=cluster-server,app.kubernetes.io/part-of=rabbitmq