GatewayClass: Multi-Tenancy

[shaneutt]

Consider this hypothetical scenario:

Team1 and Team2 at company ACME both share a large Kubernetes cluster and both use the acme-gateway-controller for ingress to their apps.

Let's assume the following about the acme-gateway-controller:

• the controller manager uses a default --gateway-class of acme.io/gateway-controller
• has a Helm chart which deploys the GatewayClass
• the Helm chart provides a generated metadata name for the GatewayClass by default
• the Helm chart provides the default acme.io/gateway-controller for the GatewayClass's .spec.controller by default

Thereafter Team1 and Team2 do the following:

• deploy their own acme-gateway-controller using mostly defaults
• deploy their apps, which include Gateway resources configured for the name of the GatewayClass that Helm generated

In the above scenario you would have two acme-gateway-controller controller managers which would both be referenced by controller name in each other's GatewayClass, the implication being that they would both provision copies of each others Gateway resources, and then each others Routes in those Gateways.

I see the potential for this to seem like a bit of an edge case and perhaps maybe a problem people would assume needs to be dealt with by Team1 and Team2 being more deliberate in their implementations and/or the acme-gateway-controller needs to not use a static default, but maybe this is a fundamental problem others see too and we need to dig in deeper? I bring this up because I feel there's a bit more nuance as we (Kong) have run into some issues like this in the past with collisions using the current ingress class name.

So there's some obvious potential solutions given the current spec:

a. the chart should use a generated unique controller reference
b. the controller implementation could be configured to correspond only with the metadata names of a GatewayClass, thus ensuring no accidental coverage of others we didn't know about

But rather than picking one of these I'm mostly looking for feedback for what solution we currently prescribe (or should) and if/how we document that currently. I did a couple passes over the specs, the guides, and the concepts to see if there was already some documentation which covers this but I didn't latch on to anything.

My thinking is that we may need a specific GatewayClass implementation guide (e.g. site-src/v1alpha2/guides/gatewayclass-multitenancy.md)? Maybe ultimately this is something that the spec doesn't cover for, and then the result should be updates to the spec and then we add a guide? Let me know your thoughts 🤔 .

[robscott]

This is a great topic, thanks for raising it! Our documentation really needs to cover this - I completely agree with your suggestion for a docs page about this.

I think that each instance of a controller should have a unique name. We currently validate that each controller name is a domain-prefixed path. Given that, my recommendation would be for each implementation to support controller name values with custom paths. For example, the Acme controller may support controller names matching this prefix: acme.io/gateway/*, defaulting to just acme.io/gateway. If a user wanted to provision multiple instances of the Acme Gateway controller in the same cluster, they would need to specify a custom controller name for any additional ones, such as acme.io/gateway/custom.

[shaneutt]

I completely agree with your suggestion for a docs page about this.
I think that each instance of a controller should have a unique name.

This seems sensible to me, if none of the maintainers seem to object then I'm +1 for making this one of our action items for this post.

[jpeach]

• deploy their own acme-gateway-controller using mostly defaults

Remember, in almost all cases, this will be a cluster-admin operation. Any groups with that level of access that are making changes to cluster infrastructure will need to co-ordinate.

[youngnick]

Part of the spec's purpose is to provide guidelines for things that we don't mandate, so that people can be doing similar things. That's what I see this as. "If you have an implementation that can be installed more than once, you should do something like this". That will ensure that people are starting from the same place, and we don't end up with completely separate uses, and maybe someday we can do something more.

[jpeach]

There must be a 1-1 binding between gatewayclass controller names and controllers. I agree that the spec should make it crystal clear that this is required, but the question here is whether the spec should say what happens if that constraint is violated.

[youngnick]

I think that's there's huge value in mandating the exact behavior if a conflict like this is detected, because it means that we don't end up with between-implementation drift. I think that saying "Conflict == everything out" is simple but adds a vector for shenanigans. Said shenanigans require cluster-admin, so they're not really a huge risk, and I think that the benefit of having implementations consistent in their behavior outweighs it.

[jpeach]

OK, I'm not convinced that it's reasonable for implementations to be able to detect when some other controller is also using their controller name.

[stevesloka]

There must be a 1-1 binding between gatewayclass controller names and controllers.

Then should gatewayclass just move into a Gateway's spec? That way you are guaranteed that one implementation of a Gateway matches a single Controller? Doing this eliminates the need for folks to follow a "guide" and the API would then enforce this behavior which I think will end up with better "core" functionality vs having various implementations do different things.

[shaneutt]

Related but a bit of an aside, and consider this a thought experiment:

In our implementation of Gateways so far it seems that we would want a strict 1:1 mapping of actual controllers to the controller field in the GatewayClass (I know this is not true for all implementations, and in fact may become untrue later for our own).

Given the above we've at least thought about linking controllers by the metadata name of a GatewayClass rather than the controller field in the spec, as doing so consequently enforces uniqueness (e.g. in the original scenario team2 would get a CONFLICT when trying to deploy, rather than not realize there was overlap).

I'm well aware of the implications here, and this is NOT what we're planning to do so I re-iterate that this is a throught experiment only, but what are your thoughts on this? If nothing else then for posterity what are the problems you would anticipate coming from this?

(the purpose of this would be to see if there's something in the spec that might need to be changed OR to gather insights that we can then document in the spec to help reinforce it)

[jpeach]

You have to match on the controller name. The gateway class is an arbitrary string, so I don't see any way to actively prevent a different controller from looking at it.

[shaneutt]

You have to match on the controller name. The gateway class is an arbitrary string

This doesn't seem like a forgone conclusion to me, the spec.controller is an arbitrary string in some rights despite there being more intended structure to it is it not?

so I don't see any way to actively prevent a different controller from looking at it.

Preventing other controllers from looking at a GatewayClass isn't a goal I intended with this thought experiment.

I'm trying to explore whether having that "arbitrary string" be unique (in an enforceable way) provides gains or causes any losses.

[evankanderson]

I was just glancing through back discussions before starting my own, but I'm wondering about the premise here.

This seems to suggest that Team1 and Team2 both have permissions on the cluster to deploy jobs which have RBAC access to all CRUD operations on Gateways in each others namespaces. If so, that feels like both teams are close to cluster-admin in terms of Gateways-related objects on their cluster (since they could install some other chart that gave them an interactive shell with those permissions).

If not, then there's some other access control (Helm chart allowlist, Gatekeeper / Kyverno or whatnot) going on here by the cluster owner which could be leveraged to prevent the two Teams from being able to deploy any Pod which has permissions to control or view Gateways in the other team's namespaces.