Gateway API v1.3 Release Scoping: New Experimental Features

[robscott]

In v1.3, we're continuing to work with our new release process, and we're currently in the first phase of that process - scoping. This discussion is intended to be a central place for proposed additions to the experimental channel in v1.3.

If you have an enhancement that you would like to see included in Gateway API v1.3, and you personally have time to lead the development of the feature, please add a comment below using the following template:

# Proposal Name Goes Here

**1. Describe the proposed feature**

**2. Why is this feature important for Gateway API?**

**3. How broadly supportable would this feature be? (Look at Envoy, HAProxy, and NGINX as a baseline)**

**4. Do you have sufficient time to lead the development of this feature in Gateway API? Do you have anyone else that can help you?**

**5. Are there any relevant links for previous discussions about this topic in Gateway API? (It would be particularly helpful if you can provide links showing strong interest for this feature)**

Ultimately, we'll likely have more proposals here than the project has room for. Each feature that leaves experimental channel in a release gives us an opening to add a feature to experimental channel, and we're expecting somewhere between 3 and 5 in this cycle. (You can help the project move more quickly and accept more new features by helping us move features out of experimental channel - see #3404 for details). Even if you don't have time to lead a feature, you can help us prioritize the features proposed in this thread by:

• Upvoting proposals that you would like to see move forward
• Adding a comment below a specific proposal to note that your implementation would work to support the feature
• Adding a comment below a specific proposal that you would be able to help lead a proposal to completion
• Adding a comment below a specific proposal to make a case for the inclusion of this proposal

The scoping process is scheduled to wrap up on November 19, 2024. Please have all proposals in place at least 1 week before that point so everyone has sufficient time to comment and vote on proposals.

It's worth noting that maintainers will still have some discretion in terms of which proposals are accepted into the scope of a release, but the information gathered here will be the primary consideration for which proposals are included in v1.3. Essentially we're hoping to prioritize features that:

1. Are broadly implementable
2. Have significant community support
3. Will encourage more adoption of the API
4. Introduce minimal cost to our "complexity budget"
5. Have strong leadership that will follow through with the proposal

[mikemorris]

Retry Budgets

1. Describe the proposed feature
Add the capability to configure budgeted-style retries in Gateway API.

2. Why is this feature important for Gateway API?
Budgeted retries offer benefits over typical "max count" retries in mitigating "retry storm" scenarios, and has been a recurring requested capability by Istio users which we would prefer to implement through Gateway API configuration.

3. How broadly supportable would this feature be? (Look at Envoy, HAProxy, and NGINX as a baseline)

• Envoy supports retry budget thresholds as a circuit breaker configuration.
• Linkerd implemented retry budgets through a "service profile" such that a retry policy defined by a service owner should be respected by "smart" clients. Linkerd until quite recently exclusively offered retry functionality only through this budgeted configuration and I expect the project would like to offer that functionality through their Gateway API implementation.

The UX for how this functionality should be implemented in Gateway API will require some discovery work to determine whether the HTTPRoute retry stanza or an alternative such as a policy attachment implementation is most suitable, and may require implementation work in dataplane proxies to accommodate this configuration.

4. Do you have sufficient time to lead the development of this feature in Gateway API? Do you have anyone else that can help you?
Yes, I can lead. I've heard demand for this functionality from several community members (@ericdbishop, @rajatsharma94, @zirain, @ramaraochavali) who may be able to help support, and I would seek review from @kflynn on API design.

5. Are there any relevant links for previous discussions about this topic in Gateway API? (It would be particularly helpful if you can provide links showing strong interest for this feature)

• Retry Budgets in HTTPRouteRetry #3388
• This was originally part of the scope of Configurable Retries in HTTPRoute #1731 and has substantial discussion on ecosystem and implementation questions in that issue, but was omitted due to timeline/capacity constraints in the design and implementation phase.

[dprotaso]

Ready Condition

1. Describe the proposed feature

Currently a Ready condition on Routes is 'reserved'. It would be great to put some work into making the condition functional.

2. Why is this feature important for Gateway API?

There's no reliable way for clients to know when it is safe to make an request given the propagation delay of the config and actuation of the proxy.

3. How broadly supportable would this feature be? (Look at Envoy, HAProxy, and NGINX as a baseline)

Would need help determining this.

4. Do you have sufficient time to lead the development of this feature in Gateway API? Do you have anyone else that can help you?

Possibly.

5. Are there any relevant links for previous discussions about this topic in Gateway API? (It would be particularly helpful if you can provide links showing strong interest for this feature)

Historical
#1156
#1364

Still open:
#1870

[howardjohn]

Before we go about adding this to the official API, I would love to see that an implementation can correctly implement this first. I think it would be a very poor position to be in if the only implementations that can support it have fundamental flaws in their implementation. Happy to discuss more in depth than #3403 (reply in thread) on this topic -- Istio spent literally 4 years trying to build this out before deciding it was a fundamentally impossible problem to meaningfully solve in a distributed system.

[arkodg]

@howardjohn any links you can share around the effort mentioned above ?

[arkodg]

if the control plane knows the number of data planes running, it can confirm if the configuration has persisted in them or not. The reliability of the status information depends on the CAP theorem tradeoffs (CP or AP systems).

From a user perspective, some information is far more useful than no information

[kflynn]

My thinking tends to line up with @howardjohn's here, though I would actually go farther because I don't think this is just a technical issue: the question of "can I send traffic down a particular route" veers almost instantly into the application-specific realm. For many routes, calling it Ready when any endpoint is available will be fine, but for some critical things, it will need to be all endpoints... or 75% of them, or all but one, or all endpoints that aren't part of a rollout, or any number of other possible conditions.

Getting Gateway API in the middle of that kind of debate isn't going to be productive, so I'm not in favor of this one.

[arkodg]

@kflynn the notion of overprovisioning factor to decide what fraction of total endpoints/collection is good enough (healthy enough or ready enough) is something the project will have to inevitably tackle to solve routing features like health checks and active - passive failovers

[youngnick]

Auth*

1. Describe the proposed feature

Make forward progress on some sort of construct to describe authentication and/or authorization.

Note that my personal opinion is that we should address the simplest possible case first - the Ingress, north/south case, and leave exact definition of GAMMA interactions for a later date.

2. Why is this feature important for Gateway API?

This is a table-stakes feature for most Ingress implementations, and the lack of a solution here is a barrier to Gateway API adoption.

3. How broadly supportable would this feature be? (Look at Envoy, HAProxy, and NGINX as a baseline)

Very, this is a table stakes feature. Envoy, HAProxy, and NGINX all support some variation of this already, and numerous Gateway API implementations have their own implementation specific config as well.

4. Do you have sufficient time to lead the development of this feature in Gateway API? Do you have anyone else that can help you?
Yes, but I know that other folks, (@LiorLieberman, @robscott, @kflynn, Gui, @howardjohn) are also interested.

5. Are there any relevant links for previous discussions about this topic in Gateway API? (It would be particularly helpful if you can provide links showing strong interest for this feature)

We've talked about this already quite a bit:

• The existing GEP is GEP: Gateway/HTTPRoute level authentication #1494
• @kflynn summarized our Paris discusion: GEP: Gateway/HTTPRoute level authentication #1494 (comment)
• I'm currently working with @jgao1025 on the introduction and prior art section of an update to this GEP

[thejosephstevens]

Huge +1 here, we have multiple places we're using Ingress that I would love to migrate, but auth is currently handled by the ingress, and it doesn't make sense to replace one existing tool with two new ones (particularly an extra that we don't actually want and will immediately want to deprecate).

[LiorLieberman]

Agree on the importance. Would like to see some version of auth as we discussed in KubeCon. My preference is policy attachment and not route/gateway level auth.

[kflynn]

I am very very strongly opposed to policy attachment for table-stakes features, even more so where we're not interacting with core resources.

[kflynn]

(I very much agree with prioritizing N/S auth though. 🙂)

[jgao1025]

I am very very strongly opposed to policy attachment for table-stakes features, even more so where we're not interacting with core resources.

Is there a good reason for this? The task appears to be scaling up significantly, and using policy attachment could effectively manage the hierarchy while also offering better portability.

[shadialtarsha]

Gateway API Support for gRPC Retries

1. Describe the proposed feature
Add support for gRPC retries to GRPCRoute.

2. Why is this feature important for Gateway API?
Retries are a fundamental feature of keeping services reliable and resilient. Currently, HTTPRoute supports a retry API and I am proposing to bring this to GRPCRoute as well.

3. How broadly supportable would this feature be? (Look at Envoy, HAProxy, and NGINX as a baseline)
Envoy, HAProxy, and Nginx are all support retries for gRPC in some form. However, Envoy is the only proxy supporting retrying on gRPC status codes. The other proxies are retrying on timeouts, network errors, and HTTP status codes.

4. Do you have sufficient time to lead the development of this feature in Gateway API? Do you have anyone else that can help you?
Yes, I do.

5. Are there any relevant links for previous discussions about this topic in Gateway API? (It would be particularly helpful if you can provide links showing strong interest for this feature)
GEP-3440

[lianglli]

CORS Filter

1. Describe the proposed feature
Provide a way to configure CORS in a HTTPRoute.

2. Why is this feature important for Gateway API?
The RFC 6454 defines the concept of an "origin". W3C recommend REC-cors-20140116. Then, RFC 7480 describes the registration data access protocol (RDAP) specifically.

Cross-Origin Resource Sharing (CORS) is supported by all browsers. Complex applications often reference third-party APIs and resources in their client-side code. CORS allows browsers to enforce the same-origin security policy for preventing client-side from accessing sensitive resources of a server on a domain different than the domain that served the web page.

3. How broadly supportable would this feature be? (Look at Envoy, HAProxy, and NGINX as a baseline)

• NGINX supports it using the add_header directive.
• Envoy has a CORS filter envoy.filters.http.cors.
• Traefik supports this with the customResponseHeaders.
• KONG supports adding cross-origin resource sharing (CORS) to a service or a route with a CORS plugin.
• HAProxy can enable CORS functionality by using the HAProxy CORS Lua Library.
• ingress-nginx supports this with CORS annotation.
• NGINX Ingress Controller supports this with CRD ResponseHeaders.
• Tengine-Ingress supports this with CORS annotations.

4. Do you have sufficient time to lead the development of this feature in Gateway API? Do you have anyone else that can help you?
Yes.

5. Are there any relevant links for previous discussions about this topic in Gateway API? (It would be particularly helpful if you can provide links showing strong interest for this feature)

• issue GEP: Add support for CORS
• PR GEP 1767: CORS Filter
• relevant issue:
  Support the necessary functions of load balancing to gateway and route
  Can HTTPRoute support global filters

Originally posted by @lianglli in #3103 (comment)

[jgao1025]

It's a security feature, and by its very nature, security tends to be invisible. I'm wondering if there would be a good way to make testing more visible? I mean, it's a configuration setting, but to truly achieve the desired goal, could we ensure through testing that a malicious attacker won't be able to exploit this vulnerability after the setting is applied?

[lianglli]

HTTP Cookie Match

1. Describe the proposed feature

1. Support cookie matching in a HTTPRoute
2. Add a new match type List

2. Why is this feature important for Gateway API?
Cookie is an essential part of the HTTP request. The Cookie header has multiple cookie-pair which contains the cookie-name and cookie-value. Just like HTTP route based on header and query parameter is common, it’d be helpful to have a HTTPCookieMatch field in HTTPRouteMatch which would let gateway forwards request based on cookie to the certain backends.

Cookies are used to maintain state and identify specific users. Moreover, cookies, headers and query parameters are common techniques used in a canary release.
Currently HTTPRouteMatch API supports the following condition: path, method, header and query parameter. This GEP proposes adding support for cookie matching in a HTTPRoute.

3. How broadly supportable would this feature be? (Look at Envoy, HAProxy, and NGINX as a baseline)

• NGINX supports it using the cookie_name variable.
• Envoy has PR #21496 and #16344 about HTTP cookie matching.
• Ingress-nginx supports this with an annotation nginx.ingress.kubernetes.io/canary-by-cookie.
• NGINX Ingress Controller supports this with CRD Condition.
• Tengine-ingress supports this with annotations nginx.ingress.kubernetes.io/canary-by-cookie and nginx.ingress.kubernetes.io/canary-by-cookie-value.

4. Do you have sufficient time to lead the development of this feature in Gateway API? Do you have anyone else that can help you?
Yes.

5. Are there any relevant links for previous discussions about this topic in Gateway API? (It would be particularly helpful if you can provide links showing strong interest for this feature)
issue

• issue GEP: Add support for cookie in the HTTPRouteMatch API
• PR GEP-2891: HTTP Cookie Match
• relevant issue:
  HTTPCookieMatching
  httproute add ipmatch and cookiematch
  bug: multiple header value matching semantics

Originally posted by @lianglli in #3103 (comment)

[lianglli]

Query Parameter Filter

1. Describe the proposed feature
Provide a way to modify query parameters of an incoming request in a HTTPRoute.

2. Why is this feature important for Gateway API?
Just like modify header is useful, the same goes for query parameters.
It's helpful to have a HTTPQueryParamFilter field in HTTPRouteFilter to set,
add and remove a query parameter of the HTTP request before it is sent to the upstream target.

The query parameters are an important part of the request URL.
The developers can use query parameters to filter, sort or customize data of request body.
Backend service can enable different function based on the query parameters.
Query parameters are important information about search and track.
Moreover, query parameter, headers and cookies are common techniques used in a canary release.

3. How broadly supportable would this feature be? (Look at Envoy, HAProxy, and NGINX as a baseline)

• NGINX supports it using the set directive.
• Envoy has a feature request Support adding/removing query params and PR #33977 about query parameters filter.
• Traefik supports this with a Query Parameter Modification plugin.
• KONG supports adding/removing query params with a Request Transformer plugin.
• Tengine-Ingress supports this with annotation nginx.ingress.kubernetes.io/canary-request-add-query.

4. Do you have sufficient time to lead the development of this feature in Gateway API? Do you have anyone else that can help you?
Yes.

5. Are there any relevant links for previous discussions about this topic in Gateway API? (It would be particularly helpful if you can provide links showing strong interest for this feature)

• issue GEP: Add support for query parameter in the HTTPRouteFilter API
• PR GEP-2895: Query Parameter Filter
• relevant issue:
  gateway-api: support VirtualService-kind routes on k8s gateways

Originally posted by @lianglli in #3103 (comment)

[dprotaso]

From 1.2: #3103 (comment)

ListenerSets - Merging many listeners into a single Gateway

1. Describe the proposed feature

Support for merging Gateway listeners should have standard & documented mechanic.

2. Why is this feature important for Gateway API?

Knative generates on demand per-service certificates using HTTP-01 challenges. There can be O(1000) Knative Services in the cluster which means we have O(1000) distinct certificates. The Gateway Resource is a contention point since is the only place to attach listeners to gateways with certificates.

The spec currently has language to indicate implemenations MAY merge Gateways resources but the mechanic isn't defined.

gateway-api/apis/v1beta1/gateway_types.go

Lines 76 to 78 in 541e9fc

	// determines that the Listeners in the group are "compatible". An
	// implementation MAY also group together and collapse compatible
	// Listeners belonging to different Gateways.

Secondly, given similar use-cases/needs for control planes to manage listeners it might make sense that merging Gateways is part of extended conformance.

3. How broadly supportable would this feature be? (Look at Envoy, HAProxy, and NGINX as a baseline)

I believe Contour has prototyped this GEP. Envoy Gateway has a similar feature to merge gateways (https://gateway.envoyproxy.io/v0.6.0/api/extension_types/#envoyproxyspec)

4. Do you have sufficient time to lead the development of this feature in Gateway API? Do you have anyone else that can help you?

Yes

5. Are there any relevant links for previous discussions about this topic in Gateway API? (It would be particularly helpful if you can provide links showing strong interest for this feature)

Prior Discussions:

#1248,
#1246

GEP:

#3213

[robscott]

@dprotaso is this meant to represent ListenerSets?

[dprotaso]

@dprotaso is this meant to represent ListenerSets?

Yup - let me update the description

[xtineskim]

cont. from v1.2 (#3103 (comment))

gRPC Timeouts

1. Describe the proposed feature
adding gRPC timeouts to GRPCRoute, similar to how HTTPRoute has timeouts

2. Why is this feature important for Gateway API?
To keep xRoutes on par with each other

3. How broadly supportable would this feature be? (Look at Envoy, HAProxy, and NGINX as a baseline)
Various implementations have some method of implementing gRPC timeouts

4. Do you have sufficient time to lead the development of this feature in Gateway API? Do you have anyone else that can help you?
yes

5. Are there any relevant links for previous discussions about this topic in Gateway API? (It would be particularly helpful if you can provide links showing strong interest for this feature)
#3219

from 1.2 #3103 (comment) . We were not able to get it in last time with short notice, but will have more time this next release

[dprotaso]

HTTPRoute-level SSL redirection

1. Describe the proposed feature

There's no simple way to enforce SSL redirection solely the route level.

A solution here could be being able to match on the scheme in the HTTPRoute. I'm open to others.

2. Why is this feature important for Gateway API?

This would help with redirect loops and http => http redirects.

Currently it requires two HTTPRoutes and configuring a two Gateway listeners to accomplish this. But that approach only works well if you have the ability to configure your Gateway.

The goal here is to support this feature at the HTTPRoute level.

3. How broadly supportable would this feature be? (Look at Envoy, HAProxy, and NGINX as a baseline)

nginx/K8s Ingress has an annotation

ingress.kubernetes.io/force-ssl-redirect: "true"

Contour supports the above annotations.

I think this behaviour is pretty common in many implementaitons.

4. Do you have sufficient time to lead the development of this feature in Gateway API? Do you have anyone else that can help you?

Yes

5. Are there any relevant links for previous discussions about this topic in Gateway API? (It would be particularly helpful if you can provide links showing strong interest for this feature)

#2809
#2455
Redirect Loop #1185 (comment)

[mikemorris]

Agree that we should simplify the UX for this, but it feels like this would be more appropriate to solve at the Listener level, potentially in the tls field via https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io/v1.GatewayTLSConfig