Serving Operator should provide an option that all ingress gateways are configured

[houshengbo]

If a user creates a custom ingress gateway, istio may select a gateway at random (knative/serving#5163 (comment)). The operator should provide a way to ensure that the ingress gateway always works.

This relies a bit on knative/serving#5172.

I believe this could work by montoring gateways and adjusting the config-istio to account for the new gateway (https://github.com/knative/serving/blob/b7dfe3fdb80ffac7f7a05c20787400ce51aad2b1/config/config-istio.yaml#L45)

This issue is a clone of an existing issue in knative/serving-operator: knative/serving-operator#156

[houshengbo]

By @trshafer

Another option is to not install the `knative-ingress-gateway` (https://github.com/knative/serving/blob/b7dfe3fdb80ffac7f7a05c20787400ce51aad2b1/config/202-gateway.yaml#L19) if one is already configured, . And use that gateway instead of the `knative-ingress-gateway`.

[houshengbo]

Some discussions in the community:

Three options I could see would be:
* Package all the plugins and have something in the CR to list which plugin dukes should be included
* Package all the plugins in one fine and use labels to select them
* Package each in their own operator with some way for the serving operator to ensure at least one is installed
11:23
Serving uses annotations and a default config to determine which plugins to use for HTTP routing (and they aren't mutually exclusive)

markusthoemmes:redhatnew:  11:26 AM
we could key off of the setting @argent is referring to, to pick one of the ingresses, but each potentially comes with its own set of configuration.
So maybe we have something like this?
type IngressConfiguration struct {
  Istio *IstioConfiguration
  Contour *ContourConfiguration
  Kourier *KourierConfiguration
}
And for each non-nil configuration, we install the relevant package?
11:26
IIRC there's precedence in K8s for this

Jim Crossley:redhat:  11:27 AM
yeah, i like that, and from the install docs, it looks like it's easy enough to grab the istio, kourier, etc yaml's from somewhere, i.e. we won't have to create them specifically for the operator

[houshengbo]

@markusthoemmes This is all the information I collected for this feature so far.
Plz write up your idea in terms of an proposal for this feature. Thx.

[jcrossley3]

I think we need to have a solution in place for this before we recommend this operator. At a minimum, we need an option in the CR for disabling istio entirely. This is for downstream consumers who already have another ingress, e.g. kourier, configured.

[markusthoemmes]

@jcrossley3 while I agree we need to solve this, the existing "recommended" knative-serving-operator has the exact same behavior. I don't think the combined operator makes that any better or worse (yet).

[houshengbo]

Added the proposal @markusthoemmes drafted: https://docs.google.com/document/d/1AQBGCIoUAB8Whg02MVWdujMj8O-OWFvPbccoctMbgJA/edit#

[nak3]

@houshengbo may I ask you the status of this feature? (cc @ZhiminXiang )

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

[houshengbo]

@nak3 We have resumed the development of this feature. We are discussing how to separate the ingress out of the existing knative-serving directory in kodata, and how to align the version of the knative-serving with the version of the ingresses.

[evankanderson]

/remove-lifecycle stale