using kustomize to modify nodeSelector's

Question

It would be useful for me to be able to define at a higher level a way to determine a set of nodes that a set of pods should run ontop of.

Is there a way to use kustomize so that i can specify what nodeSelectors a deployment should have?

Answer 1

i ended up just defining a patch:

in kustomization.yaml:

patchesStrategicMerge:
- nodeSelectors.yaml

in nodeSelectors.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
spec:
  template:
    spec:
      nodeSelector:
        group: infra
        slurm: mysql

bit long winded; but i guess it gives the most flexibility in terms of defining which pods i care ping and where.

Answer 2

OP asks for a way for all pods to run on a pre-defined set of nodes. So, adding to @yee379, you may define a patch to be applied to multiple generic resources at once like this:

patches:
  - patch: |-
      kind: not-important
      metadata:
        name: not-important
      spec:
        template:
          spec:
            nodeSelector:
              key: value
    target:
      kind: (StatefulSet|Deployment|Job)
  # ... Include more resources
  # If you'd like to include Pods specifically:
  - patch: |-
      kind: not-important
      metadata:
        name: not-important
      spec:
        nodeSelector:
          key: value
    target:
      kind: Pod

Answer 3

I think you can tain nodes. This kind of nodes are reserved for specific pod.

Node affinity, is a property of pods that attracts them to a set of nodes (either as a preference or a hard requirement). Taints are the opposite – they allow a node to repel a set of pods.

Taints and tolerations work together to ensure that pods are not scheduled onto inappropriate nodes. One or more taints are applied to a node; this marks that the node should not accept any pods that do not tolerate the taints. Tolerations are applied to pods, and allow (but do not require) the pods to schedule onto nodes with matching taints.

Taints and tolerations are a flexible way to steer pods away from nodes or evict pods that shouldn’t be running. Here are examples few of them:

• Dedicated Nodes: If you want to dedicate a set of nodes for exclusive use by a particular set of users, you can add a taint to those nodes (say, kubectl taint nodes nodename dedicated=groupName:NoSchedule) and then add a corresponding toleration to their pods (this would be done most easily by writing a custom admission controller). The pods with the tolerations will then be allowed to use the tainted (dedicated) nodes as well as any other nodes in the cluster.
• Nodes with Special Hardware: In a cluster where a small subset of nodes have specialized hardware (for example GPUs), it is desirable to keep pods that don’t need the specialized hardware off of those nodes, thus leaving room for later-arriving pods that do need the specialized hardware. This can be done by tainting the nodes that have the specialized hardware (e.g. kubectl taint nodes nodename special=true:NoSchedule or kubectl taint nodes nodename special=true:PreferNoSchedule) and adding a corresponding toleration to pods that use the special hardware.
• Taint based Evictions (beta feature): A per-pod-configurable eviction behavior when there are node problems, which is described in the next section.

More information you can find here: Kubernetes node for specific pods.

Interesting documentation: taint-toleration-dedicated.