Connect Inject Init Container fails with An ACL token must be provided

[ghost]

Question

I am trying to figure out what I am missing or if there is something larger taking place.

I have consul deployed using the helm chart and have ACL's enabled. I am NOT using "manageSystemACLs"

Here are my values:

consul:
  global:
    datacenter: my-consul
    enabled: true
    image: "hashicorp/consul:1.15.2"
    gossipEncryption:
      autoGenerate: true
    acls:
      bootstrapToken:
        secretName: consul-master-tokens
        secretKey: consul-bootstrap-acl-token
  ui:
    service:
      type: 'ClusterIP'
  server:
    extraVolumes:
      - type: secret
        name: consul-acl
        load: true
    disruptionBudget:
      maxUnavailable: 0
  client:
    enabled: true
    extraVolumes:
      - type: secret
        name: consul-acl
        load: true
  connectInject:
    aclInjectToken:
      secretName: consul-master-tokens
      secretKey: consul-connect-token

Everything is working fine. The consul-clients are communicating to the consul-server. The consul-connect-injector is using the aclInjectToken and is registered with the consul-server as well.

I am testing the use of connect inject by using the demo deployments at:

https://github.com/hashicorp/consul-k8s/tree/main/charts/demo/templates

The services for these deployments register in consul without issue, and say "Defined in proxy registration". However they are not available due to failing health checks. They fail due to pods not being ready E.G. "Pod "namespace/nginx-5b6d5c7fdc-45nqp" is not ready"

The reason for this is that the consul-connect-inject-init container never finishes. It gets stuck with the following error message:

2023-04-19T20:28:10.270Z [INFO]  consul-server-connection-manager: trying to connect to a Consul server
2023-04-19T20:28:10.274Z [INFO]  consul-server-connection-manager: discovered Consul servers: addresses=[172.16.49.26:8502]
2023-04-19T20:28:10.274Z [INFO]  consul-server-connection-manager: current prioritized list of known Consul servers: addresses=[172.16.49.26:8502]
2023-04-19T20:28:10.476Z [ERROR] consul-server-connection-manager: connection error: error="fetching supported dataplane features: rpc error: code = Unauthenticated desc = An ACL token must be provided (via the `x-consul-token` metadata field) to call this endpoint"

This error is familiar to me because I received the same error in the "consul-connect-injector" deployment until I created \ added the token in the helm chart under "aclInjectToken". That said, I am not seeing a viable way to do something similar to the init containers.

I could be missing something either in the chart or in the ACL policy, or somewhere else. I could really use the help in figuring out how to resolve this.

If its helpful, my injectToken policy looks like this:

node_prefix "" {
  policy = "write"
}
service_prefix "" {
  policy = "write"
}
operator = "write"

Thanks for the assistance.

[ghost]

I could have been wrong, but I was going down the road of believing that this was due to consul lacking an auth method. I decided this was too much manual work, so I converted to using manageSystemACLs which installs the right auth methods. Once I did that, then I got passed the failure with the init container and everything worked as expected.

Updated values:

consul:
  global:
    datacenter: my-consul
    enabled: true
    image: "hashicorp/consul:1.15.2"
    gossipEncryption:
      autoGenerate: true
    acls:
      manageSystemACLs: true
  ui:
    service:
      type: 'ClusterIP'
  server:
    disruptionBudget:
      maxUnavailable: 0
  client:
    enabled: true

[abckey]

But, if I would set manageSystemACLs: false, how to make connect inject init container to work? connectInject.aclInjectToken works for consul-connect-injector pod, but not for the connect injected init container...

[DenagaKomadina]

I believe you need to manually create k8s auth method on Consul server with all the k8s host, cacert...etc and then use

connectInject:
      overrideAuthMethodName: <auth_name>

in helm chart. I just base it on the following chart entry

                {{- if .Values.connectInject.overrideAuthMethodName }}
                -acl-auth-method="{{ .Values.connectInject.overrideAuthMethodName }}" \
                {{- else if .Values.global.acls.manageSystemACLs }}
                -acl-auth-method="{{ template "consul.fullname" . }}-k8s-auth-method" \
                {{- end }}