You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This example shows how to assign a [pod](/docs/user-guide/pods/) to a specific [node](/docs/admin/node/) or to one of a set of nodes using node labels and the nodeSelector field in a pod specification. Generally this is unnecessary, as the scheduler will take care of things for you, but you may want to do so in certain circumstances like to ensure that your pod ends up on a machine with an SSD attached to it.
8
+
# Constraining pods to run on particular nodes
10
9
11
-
You can find all the files for this example [in our docs
10
+
You can constrain a [pod](/docs/user-guide/pods/) to only be able to run on particular [nodes](/docs/admin/node/) or to prefer to
11
+
run on particular nodes. There are several ways to do this, and they all use
12
+
[label selectors](/docs/user-guide/labels/) to make the selection.
13
+
Generally such constraints are unnecessary, as the scheduler will automatically do a reasonable placement
14
+
(e.g. spread your pods across nodes, not place the pod on a node with insufficient free resources, etc.)
15
+
but there are some circumstances where you may want more control where a pod lands, e.g. to ensure
16
+
that a pod ends up on a machine with an SSD attached to it, or to co-locate pods from two different
17
+
services that communicate a lot into the same availability zone.
18
+
19
+
You can find all the files for these examples [in our docs
`nodeSelector` is the simplest form of constraint.
25
+
`nodeSelector` is a field of PodSpec. It specifies a map of key-value pairs. For the pod to be eligible
26
+
to run on a node, the node must have each of the indicated key-value pairs as labels (it can have
27
+
additional labels as well). The most common usage is one key-value pair.
28
+
29
+
Let's walk through an example of how to use `nodeSelector`.
30
+
14
31
### Step Zero: Prerequisites
15
32
16
33
This example assumes that you have a basic understanding of Kubernetes pods and that you have [turned up a Kubernetes cluster](https://github.com/kubernetes/kubernetes#documentation).
@@ -50,26 +67,58 @@ Then add a nodeSelector like so:
50
67
51
68
When you then run `kubectl create -f pod.yaml`, the pod will get scheduled on the node that you attached the label to! You can verify that it worked by running `kubectl get pods -o wide` and looking at the "NODE" that the pod was assigned to.
52
69
53
-
#### Alpha feature in Kubernetes v1.2: Node Affinity
70
+
## Interlude: built-in node labels
71
+
72
+
In addition to labels you [attach yourself](#step-one-attach-label-to-the-node), nodes come pre-populated
73
+
with a standard set of labels. As of Kubernetes v1.4 these labels are
74
+
75
+
* `kubernetes.io/hostname`
76
+
* `failure-domain.beta.kubernetes.io/zone`
77
+
* `failure-domain.beta.kubernetes.io/region`
78
+
* `beta.kubernetes.io/instance-type`
79
+
* `beta.kubernetes.io/os`
80
+
* `beta.kubernetes.io/arch`
81
+
82
+
## Alpha features: affinity and anti-affinity
83
+
84
+
`nodeSelector`provides a very simple way to constrain pods to nodes with particular labels. The affinity/anti-affinity
85
+
feature, currently in alpha, greatly expands the types of constraints you can express. The key enhancements are
86
+
87
+
1. the language is more expressive (not just "AND of exact match")
88
+
1. you can indicate that the rule is "soft"/"preference" rather than a hard requirement, so if the scheduler
89
+
can't satisfy it, the pod will still be scheduled
90
+
1. you can constrain against labels on other pods running on the node (or other topological domain),
91
+
rather than against labels on the node itself, which allows rules about which pods can and cannot be co-located
92
+
93
+
The affinity feature consists of two types of affinity, "node affinity" and "inter-pod affinity/anti-affinity."
94
+
Node affinity is like the existing `nodeSelector` (but with the first two benefits listed above),
95
+
while inter-pod affinity/anti-affinity constrains against pod labels rather than node labels, as
96
+
described in the three item listed above, in addition to having the first and second properties listed above.
97
+
98
+
`nodeSelector`continues to work as usual, but will eventually be deprecated, as node affinity can express
99
+
everything that `nodeSelector` can express.
100
+
101
+
### Node affinity (alpha feature)
54
102
55
-
During the first half of 2016 we are rolling out a new mechanism, called *affinity* for controlling which nodes your pods wil be scheduled onto.
56
-
Like `nodeSelector`, affinity is based on labels. But it allows you to write much more expressive rules.
57
-
`nodeSelector`wil continue to work during the transition, but will eventually be deprecated.
103
+
Node affinity was introduced in Kubernetes 1.2.
104
+
Node affinity is conceptually similar to `nodeSelector` -- it allows you to constrain which nodes your
105
+
pod is eligible to schedule on, based on labels on the node.
58
106
59
-
Kubernetes v1.2 offers an alpha version of the first piece of the affinity mechanism, called [node affinity](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/nodeaffinity.md).
60
107
There are currently two types of node affinity, called `requiredDuringSchedulingIgnoredDuringExecution` and
61
108
`preferredDuringSchedulingIgnoredDuringExecution`. You can think of them as "hard" and "soft" respectively,
62
109
in the sense that the former specifies rules that *must* be met for a pod to schedule onto a node (just like
63
110
`nodeSelector`but using a more expressive syntax), while the latter specifies *preferences* that the scheduler
64
111
will try to enforce but will not guarantee. The "IgnoredDuringExecution" part of the names means that, similar
65
-
to how `nodeSelector` works, if labels on a node change at runtime such that the rules on a pod are no longer
112
+
to how `nodeSelector` works, if labels on a node change at runtime such that the affinity rules on a pod are no longer
66
113
met, the pod will still continue to run on the node. In the future we plan to offer
67
114
`requiredDuringSchedulingRequiredDuringExecution`which will be just like `requiredDuringSchedulingIgnoredDuringExecution`
68
115
except that it will evict pods from nodes that cease to satisfy the pods' node affinity requirements.
69
116
70
-
Node affinity is currently expressed using an annotation on Pod. After the release of 1.4, node affinity will use a field, and it will introduce an additional option for affinity called "pod affinity",
71
-
which allows you to control whether a pod schedules onto a particular node based on which other pods are
72
-
running on the node, rather than the labels on the node.
117
+
Thus an example of `requiredDuringSchedulingIgnoredDuringExecution` would be "only run the pod on nodes with Intel CPUs"
118
+
and an example `preferredDuringSchedulingIgnoredDuringExecution` would be "try to run this set of pods in availability
119
+
zone XYZ, but if it's not possible, then allow some to run elsewhere".
120
+
121
+
Node affinity is currently expressed using an annotation on Pod. Once the feature goes to GA it will use a field of Pod.
73
122
74
123
Here's an example of a pod that uses node affinity:
75
124
@@ -81,20 +130,72 @@ among nodes that meet that criteria, nodes with a label whose key is `another-an
81
130
value is `another-annotation-value` should be preferred.
82
131
83
132
You can see the operator `In` being used in the example. The new node affinity syntax supports the following operators: `In`, `NotIn`, `Exists`, `DoesNotExist`, `Gt`, `Lt`.
133
+
There is no explicit "node anti-affinity" concept, but `NotIn` and `DoesNotExist` give that behavior.
84
134
85
135
If you specify both `nodeSelector` and `nodeAffinity`, *both* must be satisfied for the pod
86
136
to be scheduled onto a candidate node.
87
137
88
-
### Built-in node labels
89
-
90
-
In addition to labels you [attach yourself](#step-one-attach-label-to-the-node), nodes come pre-populated
91
-
with a standard set of labels. As of Kubernetes v1.2 these labels are
138
+
For more information on node affinity, see the design doc
### Inter-pod affinity and anti-affinity (alpha feature)
142
+
143
+
Inter-pod affinity and anti-affinity were introduced in Kubernetes 1.4.
144
+
Inter-pod affinity and anti-affinity allow you to constrain which nodes your pod is eligible to schedule on *based on
145
+
labels on pods that are already running on the node* rather than based on labels on nodes. The rules are of the form "this pod should (or, in the case of
146
+
anti-affinity, should not) run in an X if that X is already running one or more pods that meet rule Y." Y is expressed
147
+
as a LabelSelector with an associated list of namespaces (or "all" namespaces); unlike nodes, because pods are namespaced
148
+
(and therefore the labels on pods are implicitly namespaced),
149
+
a label selector over pod labels must specify which namespaces the selector should apply to. Conceptually X is a topology domain
150
+
like node, rack, cloud provider zone, cloud provider region, etc. You express it using a `topologyKey` which is the
151
+
key for the node label that the system uses to denote such a topology domain, e.g. see the label keys listed above
152
+
in the section "Interlude: built-in node labels."
153
+
154
+
As with node affinity, there are currently two types of pod affinity and anti-affinity, called `requiredDuringSchedulingIgnoredDuringExecution` and
155
+
`preferredDuringSchedulingIgnoredDuringExecution`which denote "hard" vs. "soft" requirements.
156
+
See the description in the node affinity section earlier.
157
+
An example of `requiredDuringSchedulingIgnoredDuringExecution` affinity would be "co-locate the pods of service A and service B
158
+
in the same zone, since they communicate a lot with each other"
159
+
and an example `preferredDuringSchedulingIgnoredDuringExecution` anti-affinity would be "spread the pods from this service across zones"
160
+
(a hard requirement wouldn't make sense, since you probably have more pods than zones).
161
+
162
+
Inter-pod affinity and anti-affinity are currently expressed using an annotation on Pod. Once the feature goes to GA it will use a field.
163
+
164
+
Here's an example of a pod that uses pod affinity:
165
+
166
+
{% include code.html language="yaml" file="pod-with-pod-affinity.yaml" ghlink="/docs/user-guide/node-selection/pod-with-pod-affinity.yaml" %}
167
+
168
+
The affinity annotation on this pod defines one pod affinity rule and one pod anti-affinity rule. Both
169
+
must be satisfied for the pod to schedule onto a node. The
170
+
pod affinity rule says that the pod can schedule onto a node only if that node is in the same zone
171
+
as at least one already-running pod that has a label with key "security" and value "S1". (More precisely, the pod is eligible to run
172
+
on node N if node N has a label with key `failure-domain.beta.kubernetes.io/zone` and some value V
173
+
such that there is at least one node in the cluster with key `failure-domain.beta.kubernetes.io/zone` and
174
+
value V that is running a pod that has a label with key "security" and value "S1".) The pod anti-affinity
175
+
rule says that the pod cannot schedule onto a node if that node is already running a pod with label
176
+
having key "security" and value "S2". (If the `topologyKey` were `failure-domain.beta.kuberntes.io/zone` then
177
+
it would mean that the pod cannot schedule onto a node if that node is in the same zone as a pod with
178
+
label having key "security" and value "S2".) See the [design doc](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/podaffinity.md).
179
+
for many more examples of pod affinity and anti-affinity, both the `requiredDuringSchedulingIgnoredDuringExecution`
180
+
flavor and the `preferredDuringSchedulingIgnoredDuringExecution` flavor.
181
+
182
+
As with node affinity, the legal operators for pod affinity and anti-affinity are `In`, `NotIn`, `Exists`, `DoesNotExist`, `Gt`, `Lt`.
183
+
184
+
In principle, the `topologyKey` can be any legal label value. However,
185
+
for performance reasons, only a limit set of topology keys are allowed;
186
+
they are specified in the `--failure-domain` command-line argument to the scheduler. By default the allowed topology keys are
92
187
93
188
* `kubernetes.io/hostname`
94
189
* `failure-domain.beta.kubernetes.io/zone`
95
190
* `failure-domain.beta.kubernetes.io/region`
96
-
* `beta.kubernetes.io/instance-type`
97
191
98
-
### Conclusion
192
+
In addition to `labelSelector` and `topologyKey`, you can optionally specify a list `namespaces`
193
+
of namespaces which the `labelSelector` should match against (this goes at the same level of the definition as `labelSelector` and `topologyKey`).
194
+
If omitted, it defaults to the namespace of the pod where the affinity/anti-affinity definition appears.
195
+
If defined but empty, it means "all namespaces."
196
+
197
+
All `matchExpressions` associated with `requiredDuringSchedulingIgnoredDuringExecution` affinity and anti-affinity
198
+
must be satisfied for the pod to schedule onto a node.
99
199
100
-
While this example only covered one node, you can attach labels to as many nodes as you want. Then when you schedule a pod with a nodeSelector, it can be scheduled on any of the nodes that satisfy that nodeSelector. Be careful that it will match at least one node, however, because if it doesn't the pod won't be scheduled at all.
200
+
For more information on inter-pod affinity/anti-affinity, see the design doc
0 commit comments