Can't deploy multiple Consul installations in separated namespaces

[ansromanov]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview of the Issue

Context: I need to deploy multiple Consul installations in separate namespaces as we have several isolated development environments within the Kube cluster, where the Consul is a part of each isolated environment.

Reproduction Steps

1. helm install -n dev1 consul ...
2. helm install -n dev2 consul ...

The second command will fail with the following error:

Error: rendered manifests contain a resource that already exists. Unable to continue with install: existing resource conflict: kind: CustomResourceDefinition, namespace: , name: "apigateways.mesh.consul.hashicorp.com"

The issue is that the chart contains some cluster-scoped resources - CRDs, ClusterRole's, ClusterRoleBinding's.

I was able to deploy multiple Consul instances by performing the following steps:

• disable the service mesh feature, which creates a lot of CRDs

connectInject:
  enabled: false

• fork chart and substitute some chart resources
  ClusterRole -> Role
  ClusterRoleBinding -> RoleBinding

So, my questions are:

1. Are there any cornerstones for such a multi-tenant Consul scenario?
2. Is it possible to deploy CRDs separately from Consul installation?
3. Are there plans to support ability to skip Cluster roles installation?

Expected behavior

The chart has native support to deploy multiple Consul instances.

[aokhotnikovsisu]

Same is true for syncCatalog as it is also uses clusterrole and clusterrolebinding, modified them to role and rolebinding and got errors related to cluster scoped resources, despite I limited syncCatalog to a single namespace:

W1004 16:14:52.527132       1 reflector.go:539] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: failed to list *v1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:test-sandbox:consul-sync-catalog" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope
E1004 16:14:52.527485       1 reflector.go:147] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *v1.Ingress: failed to list *v1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:test-sandbox:consul-sync-catalog" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope
W1004 16:14:52.528030       1 reflector.go:539] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:serviceaccount:test-sandbox:consul-sync-catalog" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope
E1004 16:14:52.528071       1 reflector.go:147] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:serviceaccount:test-sandbox:consul-sync-catalog" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope
W1004 16:14:52.528370       1 reflector.go:539] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: failed to list *v1.Service: services is forbidden: User "system:serviceaccount:test-sandbox:consul-sync-catalog" cannot list resource "services" in API group "" at the cluster scope
E1004 16:14:52.528391       1 reflector.go:147] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User "system:serviceaccount:test-sandbox:consul-sync-catalog" cannot list resource "services" in API group "" at the cluster scope
W1004 16:14:53.591440       1 reflector.go:539] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:serviceaccount:test-sandbox:consul-sync-catalog" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope
E1004 16:14:53.591509       1 reflector.go:147] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:serviceaccount:test-sandbox:consul-sync-catalog" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope

As a workaround I have updated syncCatalog clusterrole and clusterrolebinding names to be unique (added namespace name), but it surely not a good way to solve this

[andrewnazarov]

We're experiencing the same issue. Curiously, the identical question was raised two years ago (#1250 for version 0.42), but we've been using version 1.1.x successfully to install Consul in multiple namespaces within one cluster.