OCPEDGE-1191: feat: initial arbiter cluster enhancement #1674
Conversation
Signed-off-by: ehila <[email protected]>
openshift-ci-robot
added
the
jira/valid-reference
|
@eggfoobar: This pull request references OCPEDGE-1191 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the epic to target the "4.18.0" version, but no target version was set. In response to this: Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-ci
bot
added
the
do-not-merge/work-in-progress
|
Skipping CI for Draft Pull Request. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
|
||
| ### Goals | ||
|
|
||
| - Provide a new node arbiter role type that supports HA but is not a full master |
There was a problem hiding this comment.
| - Provide a new node arbiter role type that supports HA but is not a full master | |
| - Provide a new arbiter node role type that achieves HA but does not act as a full master node |
| ### Goals | ||
|
|
||
| - Provide a new node arbiter role type that supports HA but is not a full master | ||
| - Support installing OpenShift with 2 regular nodes and 1 arbiter node. |
There was a problem hiding this comment.
| - Support installing OpenShift with 2 regular nodes and 1 arbiter node. | |
| - Support installing OpenShift with 2 master nodes and 1 arbiter node. |
|
|
||
| - Provide a new node arbiter role type that supports HA but is not a full master | ||
| - Support installing OpenShift with 2 regular nodes and 1 arbiter node. | ||
| - The arbiter node hardware requirement will be lower than regular nodes. |
There was a problem hiding this comment.
| - The arbiter node hardware requirement will be lower than regular nodes. | |
| - The arbiter node hardware requirements will be lower than regular nodes in both cost and performance. |
| - Running the arbiter node offsite | ||
| - Running the arbiter node as a VM local to the cluster | ||
| - Having a single arbiter supporting multiple clusters | ||
| - Moving from 2 + 1 to conventional 3 node cluster |
There was a problem hiding this comment.
| - Moving from 2 + 1 to conventional 3 node cluster | |
| - Moving from 2 + 1 to a conventional 3 node cluster |
There was a problem hiding this comment.
Why are we stating this as a non-goal? The JIRA feature asks for this (see requirement number 6 in the description)
| be scheduled on the arbiter node. The arbiter node will be tainted to make sure | ||
| that only deployments that tolerate that taint are scheduled on the arbiter. | ||
|
|
||
| Things that we are proposing of changing. |
There was a problem hiding this comment.
| Things that we are proposing of changing. | |
| Functionality that we are proposing to change: |
|
|
||
| A few drawbacks we have is that we will be creating a new variant of OpenShift | ||
| that implements a new unique way of doing HA for kubernetes. This does mean an | ||
| increase in the test matrix and all together a different type of tests since |
There was a problem hiding this comment.
...and all together a different type of tests since
since what?
|
|
||
| - Running e2e test would be preferred but might prove to be tricky due to the | ||
| asymmetry in the control plane | ||
| - We need a strategy for validating install and test failures |
There was a problem hiding this comment.
We may need to modify a lot of the tests in origin to account for this new configuration or add a whole new test suite to accommodate it
|
|
||
| - Ability to utilize the enhancement end to end | ||
| - End user documentation, relative API stability | ||
| - Sufficient test coverage |
There was a problem hiding this comment.
Whats the reason we need a Dev Preview? Could we start with a Tech Preview right away, to reduce potential time to market? We could extend TP phase if required.
| We originally had tried using the pre-existing features in OCP, such as setting | ||
| a node as NoSchedule to avoid customer workloads going on the arbiter node. | ||
| While this whole worked as expected, the problem we faced is that the desire is | ||
| to use a very lower powered and cheap device as the arbiter, this method would |
There was a problem hiding this comment.
| to use a very lower powered and cheap device as the arbiter, this method would | |
| to use a device that is lower power and is cheaper as the arbiter. This method would |
| a node as NoSchedule to avoid customer workloads going on the arbiter node. | ||
| While this whole worked as expected, the problem we faced is that the desire is | ||
| to use a very lower powered and cheap device as the arbiter, this method would | ||
| still run a lot of the overhead on the arbiter node. |
There was a problem hiding this comment.
| still run a lot of the overhead on the arbiter node. | |
| still run most of the OCP overhead on the arbiter node. |
| - Running the arbiter node offsite | ||
| - Running the arbiter node as a VM local to the cluster | ||
| - Having a single arbiter supporting multiple clusters | ||
| - Moving from 2 + 1 to conventional 3 node cluster |
There was a problem hiding this comment.
As i know, changing topology mode is not allowed
There was a problem hiding this comment.
the etcd / ocp topology would not change to my understanding - it remains a 3 node cluster.
|
|
||
| - Provide a new node arbiter role type that supports HA but is not a full master | ||
| - Support installing OpenShift with 2 regular nodes and 1 arbiter node. | ||
| - The arbiter node hardware requirement will be lower than regular nodes. |
There was a problem hiding this comment.
What about including hardware example from Daniel’s presentation in order to give an idea where this might fit?
|
|
||
| ## Open Questions [optional] | ||
|
|
||
| 1. In the future it might be desired to add another master and convert to a |
There was a problem hiding this comment.
As i know, changing topology mode is not allowed
There was a problem hiding this comment.
Questions from my side:
- What is the concrete list of control plane components that will run on the arbiter node?
- What non-control plane or control-plane supporting components need to exist on the arbiter node? (e.g. cluster observability)
- How do we let cluster admins (or other RH operators) opt-in to adding components to the arbiter? I'm thinking in cases where a customer has a monitoring workload that they need to run on the control plane, or we have operators with agents that must run on the control plane like ACM or GitOps.
- Are there any considerations needed for components that utilize the
infrastructureTopologyfield to determine their replicas? (I believe this applies to observability components like prometheus et al)
|
|
||
| ### Goals | ||
|
|
||
| - Provide a new node arbiter role type that supports HA but is not a full master |
There was a problem hiding this comment.
What does "supports HA" mean?
| ideas for future features. | ||
|
|
||
| - Running the arbiter node offsite | ||
| - Running the arbiter node as a VM local to the cluster |
There was a problem hiding this comment.
| - Running the arbiter node as a VM local to the cluster | |
| - Running a virtualized arbiter node. |
This suggestion is a little nitpicky and could be ignored if it doesn't make sense. I interpreted this non-goal to mean that we're not intending to support a virtualized arbiter node in any capacity; from within the cluster, adjacent to the cluster, or remote to the cluster.
There was a problem hiding this comment.
I think we should not exclude running the arbiter node on a hypervisor. There might be situations where this would actually be helpful. I think the key point is really not to run the arbiter on OCPVirt on the same cluster, as this spoils the idea of 3 node redundancy.
|
|
||
| #### For Cloud Installs | ||
|
|
||
| 1. User sits down at the computer. |
There was a problem hiding this comment.
What about Gitops installations? Just kidding :-)
This can be dropped IMO. The location of the cluster admin is an implementation detail.
| #### For Cloud Installs | ||
|
|
||
| 1. User sits down at the computer. | ||
| 2. The user creates an `install-config.yaml` like normal. |
There was a problem hiding this comment.
| 2. The user creates an `install-config.yaml` like normal. | |
| 2. The user creates an `install-config.yaml`. |
|
|
||
| #### For Baremetal Installs | ||
|
|
||
| 1. User sits down at the computer. |
There was a problem hiding this comment.
Same as above - this line can be dropped.
| #### For Baremetal Installs | ||
|
|
||
| 1. User sits down at the computer. | ||
| 2. The user creates an `install-config.yaml` like normal. |
There was a problem hiding this comment.
| 2. The user creates an `install-config.yaml` like normal. | |
| 2. The user creates an `install-config.yaml`. |
| mitigations we can take against that is to make sure we are testing installs and | ||
| updates. | ||
|
|
||
| Another risk we run is customers using an arbiter node with improper disk speeds |
There was a problem hiding this comment.
I'm leaning towards suggesting that this risk be removed since we document etcd disk best practices so they'd also apply here. With that in mind, I don't think this risk is specific to this enhancement. If anyone else wants to give this idea a 👍 then maybe it can be dropped otherwise it can be left as-is.
| reviewers: | ||
| - "@tjungblu" | ||
| - "@patrickdillon" | ||
| - "@williamcaban" |
There was a problem hiding this comment.
william is no longer involved in controlplane, please tag Ramon Acedo
| ideas for future features. | ||
|
|
||
| - Running the arbiter node offsite | ||
| - Running the arbiter node as a VM local to the cluster |
There was a problem hiding this comment.
I think we should not exclude running the arbiter node on a hypervisor. There might be situations where this would actually be helpful. I think the key point is really not to run the arbiter on OCPVirt on the same cluster, as this spoils the idea of 3 node redundancy.
| - Running the arbiter node offsite | ||
| - Running the arbiter node as a VM local to the cluster | ||
| - Having a single arbiter supporting multiple clusters | ||
| - Moving from 2 + 1 to conventional 3 node cluster |
There was a problem hiding this comment.
Why are we stating this as a non-goal? The JIRA feature asks for this (see requirement number 6 in the description)
| - Running the arbiter node offsite | ||
| - Running the arbiter node as a VM local to the cluster | ||
| - Having a single arbiter supporting multiple clusters | ||
| - Moving from 2 + 1 to conventional 3 node cluster |
There was a problem hiding this comment.
the etcd / ocp topology would not change to my understanding - it remains a 3 node cluster.
|
|
||
| 1. In the future it might be desired to add another master and convert to a | ||
| compact cluster, do we want to support changing ControlPlaneTopology field | ||
| after the fact? |
There was a problem hiding this comment.
see above, this is already listed in the requirements and should be possible. Maybe not on initial release, but then on the next one.
There was a problem hiding this comment.
Feature Requirement number 8 states:
It must be possible to explicitly schedule additional workload to the arbiter node. That is important for 3d party solutions (e.g. storage provider) which also have quorum based mechanisms.
The use case is e.g. ODF in replica-2 mode, where some ODF components also need 3 deployments (the ceph mon for example).
How is this supported by this ER? I think we should mention / describe this case.
|
|
||
| - Ability to utilize the enhancement end to end | ||
| - End user documentation, relative API stability | ||
| - Sufficient test coverage |
There was a problem hiding this comment.
Whats the reason we need a Dev Preview? Could we start with a Tech Preview right away, to reduce potential time to market? We could extend TP phase if required.
No description provided.