Operator support for HA deployments

[pmorie]

Problem
We are currently working on making knative controllers able to do leader-elected HA

Persona:
People using the operator to deploy serving (and possibly other parts of the project if we unify the operators), who are making decisions about how they want their knative deployment configured.

Exit Criteria
I should be able to indicate via the operator's CR that I want:

• Leader election disabled / enabled
• Number of replicas for controllers that scale horizontally
• Possibly anti-affinity
• Possibly disruption budgets

Time Estimate (optional):
Probably more dependent on design consensus than implementation time.

[jcrossley3]

It's easy enough to conceive of a new field in the CR...

spec:
  leader-election:
    - enabled: true 
      replicas: 4

...but if all the operator does in response is set ConfigMap entries as described in the design doc, then we may as well use the existing Spec.Config field instead.

[jcrossley3]

Of course, if we make the controllers consume the KnativeServing CR for their leader-election configuration instead of a ConfigMap, that may be a good first step toward migrating other controller configuration from their ConfigMaps to the KnativeServing CR.

Maybe that's your master plan. 😜

[pmorie]

I'd want to avoid serving depending on the operator's CR - it's the operator's job to make sure serving does what's in the CR.

I think the operator will wind up needing to do more than just manipulate configmaps, because there will likely be knobs that affect the resources deployed (replicas, anti-affinity etc) too.

[pmorie]

As I write this I'm thinking maybe the operator API needs to be in the design doc for this feature track

[aliok]

@pmorie
Can we resolve this ticket as following PRs are all merged, or is there anything missing?

#321
#326
#337