HTTPRoute Path Protection

[joranlager]

It looks like the creation of an HTTPRoute might enable a Team (the creator of a new HTTPRoute) to "steal" traffic already setup for another Team's Application (created earlier by an HTTPRoute)?

I think it would reduce risk and team friction if the HTTPRoutes could be optionally protected such that new HTTPRoutes created that will overlap protected paths will be rejected at the time of creation.

Consider this scenario:
Team A manages Application A on Gateway with hostname mycompany.com and an HTTPRoute with this path: http://mycompany.com/applicationa/*
... directing traffic to the "TeamAApplicationAService" Service.

Traffic flows into Application A for these requests:
http://mycompany.com/applicationa/index.html
http://mycompany.com/applicationa/api/*

Team B manages Application B on the same Gateway with hostname mycompany.com and an HTTPRoute with this path: http://mycompany.com/applicationb/*
... directing traffic to the "TeamBApplicationBService" Service.

Traffic flows into Application B for these requests:
http://mycompany.com/applicationb/index.html
http://mycompany.com/applicationb/api/*

Then, a new HTTPRoute is defined by Team B with the following path that includes a mistake done by Team B:
/applicationa/api/*
... directing traffic to the "TeamBApplicationBAPIService" Service.

After that change, traffic flows into Application B for these requests:
http://mycompany.com/applicationb/index.html
http://mycompany.com/applicationb/api/*
... and into Application B API for these requests:
http://mycompany.com/applicationa/api/*

And Application A looses this traffic:
http://mycompany.com/applicationa/api/*

I think it would reduce risk and team friction if the HTTPRoutes could be optionally protected such that new HTTPRoutes created that will overlap protected paths will be rejected at the time of creation.

The rules for merging Route Rules are lightly mentioned here: https://gateway-api.sigs.k8s.io/api-types/httproute/#merging
Details for hostname conflicts are documented here: https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io/v1.HTTPRouteSpec
And HTTPRouteRule is documented here: https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io/v1.HTTPRouteRule

[joranlager]

There is also some similar discussion on the level of Gateways and protecting the Gateway listeners here: #1218

[youngnick]

Thanks for this writeup @joranlager.

We specifically define that, in the case of a path conflict, the oldest Route (by creation time) must be chosen, not the newest, precisely to prevent this sort of thing. Adding a new Route that matches the same hostname and path as another should be rejected by conformant impelmentations.

If that's not the case, then we need to tighten up the conformance testing here and ensure that this contract is respected.

[joranlager]

Thank you, @youngnick.

I would like to read the section/details of the spec defining that behaviour - can you please share a link to it?

I found this section that talks about the runtime enforcement/rules to follow by a Proxy or Load Balancer - and the oldest Route is the to be chosen only if there is a tie (link here: https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io/v1.HTTPRouteRule). :

Proxy or Load Balancer routing configuration generated from HTTPRoutes MUST prioritize matches based on the following criteria, continuing on ties. Across all rules specified on applicable Routes, precedence must be given to the match having:

“Exact” path match.
“Prefix” path match with largest number of characters.
Method match.
Largest number of header matches.
Largest number of query param matches.
Note: The precedence of RegularExpression path matches are implementation-specific.

If ties still exist across multiple Routes, matching precedence MUST be determined in order of the following criteria, continuing on ties:

The oldest Route based on creation timestamp.
The Route appearing first in alphabetical order by “{namespace}/{name}”.

My proposal is to prevent the creation of the HTTPRoute if it contains a path that is chosen to be protected.
This will prevent routing configuration from propagating to a Proxy or a Load Balancer in the first place.

[youngnick]

The general rules around conflict resolution are defined here: https://gateway-api.sigs.k8s.io/guides/api-design/#conflicts

I understand that preventing resources being created is worthwhile, but traditionally we've biased towards allowing the objects to be created and then adding information in their status as to why they were unable to apply.

Changing the existing behavior to stop the application of resources instead of allowing them to apply is a breaking change and so is not really possible as part of the specification. I think we even have some conformance tests that validate that things are created in the apiserver, and end up with status indicators saying that there is a conflict.

If the objects were not created in the apiserver at all (because they were rejected by a rule like you describe), then it would be much more difficult to validate that the implementation is behaving correctly.

In short, I get what you are saying, but I don't think it's doable at this point in the API's lifespan.

Interested to hear what others have to say here though.

[joranlager]

Thanks @youngnick. Thank you for sharing your insights. Highly appreciated.

As for allowing the objects to be created in the apiserver and how handling is documented in the Gateway API spec related to showing status, that is understandable.
On that topic, I suspect that the actual "filtering"/rejection of routing rules when they are manifested into a Proxy or a Load Balancer's configurations can be done by the Gateway Controller. So that part might not be a part of the Gateway API spec per se.

There might however be a potential discrepancy on the topic of avoiding team friction related to paths (that is the intention to address in the HTTPRoute Path Protection idea) in the Gateway API spec;

From https://gateway-api.sigs.k8s.io/guides/api-design/#conflicts :

Separation and delegation of responsibility among independent actors (e.g between cluster ops and application developers) can result in conflicts in the configuration. For example, two application teams may inadvertently submit configuration for the same HTTP path.

In most cases, guidance for conflict resolution is provided along with the documentation for fields that may have a conflict. If a conflict does not have a prescribed resolution, the following guiding principles should be applied:

• Prefer not to break things that are working.
• Drop as little traffic as possible.
• Provide a consistent experience when conflicts occur.
• Make it clear which path has been chosen when a conflict has been identified. Where possible, this should be communicated by setting appropriate status conditions on relevant resources.
• More specific matches should be given precedence over less specific ones.
• The resource with the oldest creation timestamp wins.
• If everything else is equivalent (including creation timestamp), precedences should be given to the resource appearing first in alphabetical order (namespace/name). For example, foo/bar would be given precedence over foo/baz.

I am focusing on these two entries:

• More specific matches should be given precedence over less specific ones.
• The resource with the oldest creation timestamp wins.

Is "The resource with the oldest creation timestamp wins." to be enforced even when there is no tie?

I am asking, because when looking at the guidance for conflict resolution in the documentation for the HTTPRouteRule's matches field (https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io/v1.HTTPRouteRule), I find that the rule related to "The resource with the oldest creation timestamp wins." only applies if there are ties.

Also, the rules in that documentation could lead to teams "stealing" traffic from each other;
Imagine the "Prefix" path match /teama/ being created first by team a.
Then, team b creates the "Exact" path match /teama/appb.
As far as I can understand, that will be allowed by the below rules?

(And it will be allowed by the rule as documented in the Conflicts section discussed above too ("More specific matches should be given precedence over less specific ones.") - given that the timestamp rule only applies for ties.)

[...] Across all rules specified on applicable Routes, precedence must be given to the match having:

• “Exact” path match.
• “Prefix” path match with largest number of characters.
• Method match.
• Largest number of header matches.
• Largest number of query param matches.

Note: The precedence of RegularExpression path matches are implementation-specific.

If ties still exist across multiple Routes, matching precedence MUST be determined in order of the following criteria, continuing on ties:

• The oldest Route based on creation timestamp.
• The Route appearing first in alphabetical order by “{namespace}/{name}”.

If ties still exist within an HTTPRoute, matching precedence MUST be granted to the FIRST matching rule (in list order) with a match meeting the above criteria.

[youngnick]

You're right, these rules are only enforced on exact match ties.

Otherwise, the specificity precedence rules apply.

So yes, the example you give would mean that Team B takes that path away from Team A, assuming that they both attach to the same Gateway Listener. Unfortunately, this is where we run up against the limits of technical restrictions - we need the rules about specificity for many other reasons, but they do mean that we can't specifically stop that.

At some point though, if folks are using shared resources in this way, we do need to expect some level of good intent.

Addressing this would require a lot of mechanisms like rule priority or something that add too large amount of complexity on top of an already complex API, sadly.

[joranlager]

Thanks, @youngnick, looking at the role-oriented approach of Gateway API specification, enabling teams to operate with as little friction as possible looks like an inherent goal of the Gateway API.

From first-hand experience on working with shared resources (like F5 BIG-IP LB) and different teams sharing a Virtual Server there, I know that all of the teams had good intent. Regardless, several production incidents occurred due to sharing the same Virtual Server; the routing rules (iRules) applied by one team had side-effects that caused outage of another team's service.

This scenario is also relevant for Kubernetes Ingress, where different teams can apply rules for http paths that conflicts with another team's rules.

My suggestion is that the Gateway API specification itself includes and acknowledges this scenario.
This way, implementors of the specification can put efforts into solving this major factor of friction between teams.
This is our chance of ensuring that teams can be autonomous and knowing that they cannot unintentionally cause interruptions of other team's services.

[youngnick]

I don't disagree with your conclusion @joranlager, but I can't see a way that we could accomplish that goal without adding way too much complexity into HTTPRoute.

In Contour's HTTPProxy, we had a similar concept, route delegation, that would allow a higher object to delegate paths to a child object, and which would not allow paths outside of the delegated path - which would solve this scenario.

However, that specific mechanism meant that there needed to be able to be multiple layers of object hierarchy - in Gateway API terms, you would create a HTTPRoute that then served as the parent for another HTTPRoute (and any children of the first HTTPRoute would need to be within the defined paths there). We've talked about this before (in GEP #1058), and ran into serious problems with reusing the same object for both parent and child (does this allow arbitrary depth of a hierarchy? What happens if there are delegation loops?).

Other ways I can think of here are something like an allowedPaths or something like that in the Listener - but that wouldn't work in this case, because the Routes are all attaching to the same Listener.

To put this another way - I agree that this is a problem, and I would like to see Gateway API do something about it, but every previous attempt to handle this sort of config has stumbled on the added complexity. Without someone being willing to put in the work to drive a GEP to completion to solve this, I don't see it happening.