This document proposes a new label selector representation, called
NodeSelector
, that is similar in many ways to LabelSelector
, but is a bit
more flexible and is intended to be used only for selecting nodes.
In addition, we propose to replace the map[string]string
in PodSpec
that the
scheduler currently uses as part of restricting the set of nodes onto which a
pod is eligible to schedule, with a field of type Affinity
that contains one
or more affinity specifications. In this document we discuss NodeAffinity
,
which contains one or more of the following:
- a field called
RequiredDuringSchedulingRequiredDuringExecution
that will be represented by aNodeSelector
, and thus generalizes the scheduling behavior of the currentmap[string]string
but still serves the purpose of restricting the set of nodes onto which the pod can schedule. In addition, unlike the behavior of the currentmap[string]string
, when it becomes violated the system will try to eventually evict the pod from its node. - a field called
RequiredDuringSchedulingIgnoredDuringExecution
which is identical toRequiredDuringSchedulingRequiredDuringExecution
except that the system may or may not try to eventually evict the pod from its node. - a field called
PreferredDuringSchedulingIgnoredDuringExecution
that specifies which nodes are preferred for scheduling among those that meet all scheduling requirements.
(In practice, as discussed later, we will actually add the Affinity
field
rather than replacing map[string]string
, due to backward compatibility
requirements.)
The affinity specifications described above allow a pod to request various properties that are inherent to nodes, for example "run this pod on a node with an Intel CPU" or, in a multi-zone cluster, "run this pod on a node in zone Z." (This issue describes some of the properties that a node might publish as labels, which affinity expressions can match against.) They do not allow a pod to request to schedule (or not schedule) on a node based on what other pods are running on the node. That feature is called "inter-pod topological affinity/anti-affinity" and is described here.
// A node selector represents the union of the results of one or more label queries
// over a set of nodes; that is, it represents the OR of the selectors represented
// by the nodeSelectorTerms.
type NodeSelector struct {
// nodeSelectorTerms is a list of node selector terms. The terms are ORed.
NodeSelectorTerms []NodeSelectorTerm `json:"nodeSelectorTerms,omitempty"`
}
// An empty node selector term matches all objects. A null node selector term
// matches no objects.
type NodeSelectorTerm struct {
// matchExpressions is a list of node selector requirements. The requirements are ANDed.
MatchExpressions []NodeSelectorRequirement `json:"matchExpressions,omitempty"`
}
// A node selector requirement is a selector that contains values, a key, and an operator
// that relates the key and values.
type NodeSelectorRequirement struct {
// key is the label key that the selector applies to.
Key string `json:"key" patchStrategy:"merge" patchMergeKey:"key"`
// operator represents a key's relationship to a set of values.
// Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt.
Operator NodeSelectorOperator `json:"operator"`
// values is an array of string values. If the operator is In or NotIn,
// the values array must be non-empty. If the operator is Exists or DoesNotExist,
// the values array must be empty. If the operator is Gt or Lt, the values
// array must have a single element, which will be interpreted as an integer.
// This array is replaced during a strategic merge patch.
Values []string `json:"values,omitempty"`
}
// A node selector operator is the set of operators that can be used in
// a node selector requirement.
type NodeSelectorOperator string
const (
NodeSelectorOpIn NodeSelectorOperator = "In"
NodeSelectorOpNotIn NodeSelectorOperator = "NotIn"
NodeSelectorOpExists NodeSelectorOperator = "Exists"
NodeSelectorOpDoesNotExist NodeSelectorOperator = "DoesNotExist"
NodeSelectorOpGt NodeSelectorOperator = "Gt"
NodeSelectorOpLt NodeSelectorOperator = "Lt"
)
We will add one field to PodSpec
Affinity *Affinity `json:"affinity,omitempty"`
The Affinity
type is defined as follows
type Affinity struct {
NodeAffinity *NodeAffinity `json:"nodeAffinity,omitempty"`
}
type NodeAffinity struct {
// If the affinity requirements specified by this field are not met at
// scheduling time, the pod will not be scheduled onto the node.
// If the affinity requirements specified by this field cease to be met
// at some point during pod execution (e.g. due to a node label update),
// the system will try to eventually evict the pod from its node.
RequiredDuringSchedulingRequiredDuringExecution *NodeSelector `json:"requiredDuringSchedulingRequiredDuringExecution,omitempty"`
// If the affinity requirements specified by this field are not met at
// scheduling time, the pod will not be scheduled onto the node.
// If the affinity requirements specified by this field cease to be met
// at some point during pod execution (e.g. due to a node label update),
// the system may or may not try to eventually evict the pod from its node.
RequiredDuringSchedulingIgnoredDuringExecution *NodeSelector `json:"requiredDuringSchedulingIgnoredDuringExecution,omitempty"`
// The scheduler will prefer to schedule pods to nodes that satisfy
// the affinity expressions specified by this field, but it may choose
// a node that violates one or more of the expressions. The node that is
// most preferred is the one with the greatest sum of weights, i.e.
// for each node that meets all of the scheduling requirements (resource
// request, RequiredDuringScheduling affinity expressions, etc.),
// compute a sum by iterating through the elements of this field and adding
// "weight" to the sum if the node matches the corresponding MatchExpressions; the
// node(s) with the highest sum are the most preferred.
PreferredDuringSchedulingIgnoredDuringExecution []PreferredSchedulingTerm `json:"preferredDuringSchedulingIgnoredDuringExecution,omitempty"`
}
// An empty preferred scheduling term matches all objects with implicit weight 0
// (i.e. it's a no-op). A null preferred scheduling term matches no objects.
type PreferredSchedulingTerm struct {
// weight is in the range 1-100
Weight int `json:"weight"`
// matchExpressions is a list of node selector requirements. The requirements are ANDed.
MatchExpressions []NodeSelectorRequirement `json:"matchExpressions,omitempty"`
}
Unfortunately, the name of the existing map[string]string
field in PodSpec is
NodeSelector
and we can't change it since this name is part of the API.
Hopefully this won't cause too much confusion.
** TODO: fill in this section **
-
Run this pod on a node with an Intel or AMD CPU
-
Run this pod on a node in availability zone Z
When we add Affinity
to PodSpec, we will deprecate, but not remove, the
current field in PodSpec
NodeSelector map[string]string `json:"nodeSelector,omitempty"`
Old version of the scheduler will ignore the Affinity
field. New versions of
the scheduler will apply their scheduling predicates to both Affinity
and
nodeSelector
, i.e. the pod can only schedule onto nodes that satisfy both sets
of requirements. We will not attempt to convert between Affinity
and
nodeSelector
.
Old versions of non-scheduling clients will not know how to do anything
semantically meaningful with Affinity
, but we don't expect that this will
cause a problem.
See this comment for more discussion.
Users should not start using NodeAffinity
until the full implementation has
been in Kubelet and the master for enough binary versions that we feel
comfortable that we will not need to roll back either Kubelet or master to a
version that does not support them. Longer-term we will use a programatic
approach to enforcing this (#4855).
- Add the
Affinity
field to PodSpec and theNodeAffinity
,PreferredDuringSchedulingIgnoredDuringExecution
, andRequiredDuringSchedulingIgnoredDuringExecution
types to the API. - Implement a scheduler predicate that takes
RequiredDuringSchedulingIgnoredDuringExecution
into account. - Implement a scheduler priority function that takes
PreferredDuringSchedulingIgnoredDuringExecution
into account. - At this point, the feature can be deployed and
PodSpec.NodeSelector
can be marked as deprecated. - Add the
RequiredDuringSchedulingRequiredDuringExecution
field to the API. - Modify the scheduler predicate from step 2 to also take
RequiredDuringSchedulingRequiredDuringExecution
into account. - Add
RequiredDuringSchedulingRequiredDuringExecution
to Kubelet's admission decision. - Implement code in Kubelet or the controllers that evicts a pod that no
longer satisfies
RequiredDuringSchedulingRequiredDuringExecution
(see this comment).
We assume Kubelet publishes labels describing the node's membership in all of the relevant scheduling domains (e.g. node name, rack name, availability zone name, etc.). See #9044.
The design described here is the result of careful analysis of use cases, a decade of experience with Borg at Google, and a review of similar features in other open-source container orchestration systems. We believe that it properly balances the goal of expressiveness against the goals of simplicity and efficiency of implementation. However, we recognize that use cases may arise in the future that cannot be expressed using the syntax described here. Although we are not implementing an affinity-specific extensibility mechanism for a variety of reasons (simplicity of the codebase, simplicity of cluster deployment, desire for Kubernetes users to get a consistent experience, etc.), the regular Kubernetes annotation mechanism can be used to add or replace affinity rules. The way this work would is:
- Define one or more annotations to describe the new affinity rule(s)
- User (or an admission controller) attaches the annotation(s) to pods to
request the desired scheduling behavior. If the new rule(s) replace one or
more fields of
Affinity
then the user would omit those fields fromAffinity
; if they are additional rules, then the user would fill inAffinity
as well as the annotation(s). - Scheduler takes the annotation(s) into account when scheduling.
If some particular new syntax becomes popular, we would consider upstreaming it
by integrating it into the standard Affinity
.
Are there any other fields we should convert from map[string]string
to
NodeSelector
?
The review for this proposal is in #18261.
The main related issue is #341. Issue #367 is also related. Those issues reference other related issues.