Kubernetes Scheduling

1.Manual Scheduling

2.Labels and Selectors

3.Taints and Tolerations

4.Node Selectors

5.Node Affinity

6.Taints and Tolerations vs Node Affinity

7.Resource Limits

8.Static Pods

9.Multiple Schedulers

10.Configuring Scheduler Profiles

Manual Scheduling

In Kubernetes, Pods can be manually scheduled using the nodeName field in the Pod specification. This allows you to specify the node where you want the Pod to be scheduled.

Here are the steps to manually schedule a Pod:

1. Set the nodeName field in the Pod specification to the name of the node where you want the Pod to be scheduled. For example:

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: my-container
    image: nginx
  nodeName: my-node

1. Create the Pod using the kubectl create command:

    kubectl create -f my-pod.yaml
   

   Where my-pod.yaml is the filename of the Pod specification file.

   1. Verify that the Pod has been scheduled on the desired node:

    kubectl get pods -o wide

This command will display the Pod name, its status, and the node where it is scheduled. If the Pod is not scheduled on the desired node, you may need to troubleshoot any issues with the node or the Pod specification.

Note that manually scheduling Pods is not recommended in most cases, as it can lead to uneven distribution of workload across nodes and may make it harder to manage and scale your cluster. Instead, it's generally better to rely on Kubernetes' built-in scheduling algorithms to ensure that Pods are scheduled optimally.

Labels and Selectors

In Kubernetes, labels are key-value pairs that are used to identify and organize objects such as Pods, Services, Deployments, and ReplicaSets. Labels are attached to objects as metadata and can be used to group related objects together or to specify how objects should be selected for certain operations.

Selectors, on the other hand, are used to filter objects based on their labels. A selector specifies a set of label requirements, and any objects with labels that match those requirements will be selected. Selectors are commonly used when working with Kubernetes objects that reference other objects, such as Services that need to select a set of Pods to route traffic to.

Here's an example of how labels and selectors can be used:

1. Attach labels to a set of Pods:

    apiVersion: v1
    kind: Pod
    metadata:
      name: my-pod-1
      labels:
        app: my-app
        env: dev
    spec:
      containers:
      - name: my-container
        image: nginx
    ---
    apiVersion: v1
    kind: Pod
    metadata:
      name: my-pod-2
      labels:
        app: my-app
        env: prod
    spec:
      containers:
      - name: my-container
        image: nginx
   

   In this example, we have created two Pods with different labels.

   1. Create a Service that selects Pods based on their labels:

    apiVersion: v1
    kind: Service
    metadata:
      name: my-service
    spec:
      selector:
        app: my-app
      ports:
      - name: http
        protocol: TCP
        port: 80
        targetPort: 80

In this example, we have created a Service that selects all Pods with the app: my-app label. This Service will route traffic to any Pod that matches this label requirement.

Labels and selectors are powerful tools in Kubernetes that allow you to organize and manage your objects in a flexible and scalable way. They can be used to implement complex deployment strategies, such as canary deployments, blue-green deployments, and rolling updates, as well as to monitor and debug your application.

Taints and Tolerations

Taints are applied to nodes, and tolerations are applied to Pods. Taints and tolerations work together to ensure that Pods are only scheduled on nodes that are compatible with their requirements.

Here's how taints and tolerations work:

• A taint is a label that is applied to a node. A taint consists of a key, a value, and an effect. The key and value are used to identify the taint, and the effect determines what happens when a Pod tries to schedule on the node. There are three possible effects:

  • NoSchedule: Pods will not be scheduled on the node unless they have a matching toleration.

  • PreferNoSchedule: Kubernetes will try to avoid scheduling Pods on the node unless no other nodes are available.

  • NoExecute: Pods that do not have a matching toleration will be evicted from the node.

• A toleration is a setting on a Pod that allows it to tolerate a matching taint on a node. A toleration consists of a key, a value, an operator, and a tolerationSeconds field. The key and value must match the taint applied to the node. The operator determines how the toleration should be applied, and the tolerationSeconds field specifies how long the Pod will tolerate the taint before it is evicted.

Here's an example of how taints and tolerations can be used:

1. Apply a taint to a node:

kubectl taint nodes my-node key=value:NoSchedule

In this example, we have applied a taint with key "key", value "value", and effect "NoSchedule" to the node "my-node".

1. Create a Pod with a matching toleration:

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: my-container
    image: nginx
  tolerations:
  - key: "key"
    value: "value"
    effect: "NoSchedule"

In this example, we have created a Pod with a toleration that matches the taint applied to the node. This Pod will be able to schedule on the node despite the taint.

Taints and tolerations are a powerful mechanism for managing node selection in Kubernetes, especially in large clusters with diverse hardware and software requirements. By carefully applying taints and tolerations, you can ensure that Pods are scheduled on the nodes that best meet their needs, while avoiding resource contention and conflicts.

Taints and tolerations can be a powerful way to manage node selection in Kubernetes, but they can also present some challenges. Here are some common challenges and ways to overcome them:

1. Taints can prevent certain Pods from scheduling: When a node has a taint applied to it, Pods that do not have a matching toleration will not be scheduled on that node. This can be a problem if you need to schedule a Pod on that node, but don't want to remove the taint. To overcome this challenge, you can use a Pod affinity or anti-affinity rule to specify which nodes the Pod should or should not be scheduled on.

2. Tolerations can allow Pods to schedule on unsuitable nodes: If a Pod has a toleration that matches a taint on a node, it will be scheduled on that node even if it is not suitable. This can lead to resource contention and conflicts. To avoid this, you can use Pod affinity or anti-affinity rules to ensure that Pods are only scheduled on nodes that meet certain criteria, such as having enough available resources or running a certain software version.

3. Managing taints and tolerations can be complex: When you have many nodes and many Pods, managing taints and tolerations can become complex and difficult to manage. To overcome this challenge, you can use tools such as Kubernetes operators or configuration management systems to automate the management of taints and tolerations.

4. Changing taints and tolerations can be disruptive: When you need to change the taints or tolerations on a node, it can be disruptive to running Pods. To avoid this, you can use Kubernetes rolling updates to gradually update the taints or tolerations on nodes, so that running Pods are not affected.

In summary, while taints and tolerations can be a powerful way to manage node selection in Kubernetes, they can also present some challenges. By using Pod affinity or anti-affinity rules, automating the management of taints and tolerations, and using rolling updates to avoid disruption, you can overcome these challenges and ensure that your applications are running efficiently and reliably in your Kubernetes cluster.

Node Selectors

In Kubernetes, node selectors are used to specify which nodes a Pod should be scheduled on based on a set of labels assigned to the nodes. Node selectors are a simple way to control the placement of Pods in a Kubernetes cluster.

Here's how node selectors work:

1. Assign labels to nodes: You can assign labels to nodes when you create them, or you can add labels later using the kubectl command-line tool.

For example, you can assign a label to a node like this:

kubectl label nodes <node-name> <label-key>=<label-value>

1. Define a node selector in the Pod spec: In the Pod specification, you can specify a node selector that consists of one or more label-key and label-value pairs. The label-key and label-value pairs must match the labels assigned to the nodes.

For example, you can specify a node selector like this:

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: my-container
    image: nginx
  nodeSelector:
    <label-key>: <label-value>

In this example, the node selector specifies that the Pod should be scheduled on a node with a label that matches the label-key and label-value pair specified.

If no nodes are found that match the node selector, the Pod will remain in a pending state until a suitable node becomes available.

Node selectors provide a simple way to control the placement of Pods in a Kubernetes cluster based on the labels assigned to the nodes. They are particularly useful in small or simple clusters where you have a limited number of nodes and a limited number of applications running. However, in larger or more complex clusters, it may be more efficient to use more advanced scheduling mechanisms such as node affinity, anti-affinity, taints, and tolerations.

Here are some common challenges and ways to overcome them:

1. Limited flexibility: Node selectors provide a limited level of flexibility when it comes to scheduling Pods. You can only use labels to match nodes, which means that you may not be able to express more complex scheduling requirements. For example, you may want to schedule a Pod on a node that has a certain amount of available resources, or that is running a specific version of an operating system. To overcome this challenge, you can use other scheduling mechanisms like node affinity or taints and tolerations, which provide more advanced scheduling options.

2. Static configuration: Node selectors are configured statically in the Pod spec, which means that you need to manually update the Pod spec if you want to change the node selector. This can be time-consuming and error-prone, especially if you have a large number of Pods. To overcome this challenge, you can use dynamic node selection mechanisms like node affinity or taints and tolerations, which allow you to specify scheduling rules that are evaluated dynamically at runtime.

3. Risk of resource contention: If you use node selectors to schedule Pods on specific nodes, you may run the risk of creating resource contention if too many Pods are scheduled on the same node. This can lead to performance problems and stability issues. To overcome this challenge, you can use Kubernetes resource management features like limits and quotas to ensure that Pods are not overloading nodes.

4. Difficulty in managing large clusters: If you have a large number of nodes and a large number of Pods, managing node selectors can become difficult and time-consuming. To overcome this challenge, you can use Kubernetes operators or other automation tools to manage node selectors and other scheduling mechanisms automatically.

In summary, while node selectors can be a simple and effective way to control the placement of Pods in a Kubernetes cluster, they can also present some challenges. By using other scheduling mechanisms like node affinity or taints and tolerations, using dynamic node selection, managing resources, and automating management tasks, you can overcome these challenges and ensure that your applications are running efficiently and reliably in your Kubernetes cluster.

Affinity

In Kubernetes, affinity is a mechanism used to schedule Pods on nodes that match certain criteria. Affinity is used to ensure that Pods are scheduled on nodes that meet specific requirements, such as running a certain type of workload, having specific labels, or having certain hardware capabilities.

There are two types of affinity in Kubernetes: node affinity and pod affinity.

1. Node affinity: Node affinity is used to schedule Pods on nodes that match certain criteria. Node affinity can be expressed as either "required" or "preferred" affinity. Required node affinity means that the Pod must be scheduled on a node that matches the criteria, while preferred node affinity means that the scheduler will try to schedule the Pod on a node that matches the criteria, but can still schedule the Pod on a node that does not match the criteria if necessary.

Node affinity can be further divided into two types:

• requiredDuringSchedulingIgnoredDuringExecution: Pods will be scheduled only on the nodes that match the node affinity criteria. If no nodes meet the criteria, the Pod will not be scheduled. If a node loses its label after the Pod has been scheduled, the Pod will continue to run on the node until it is terminated.

• preferredDuringSchedulingIgnoredDuringExecution: Pods will be scheduled on nodes that match the node affinity criteria if possible. If no nodes meet the criteria, the scheduler will still schedule the Pod on a node that does not match the criteria.

1. Pod affinity: Pod affinity is used to schedule Pods on nodes that are already running Pods that match certain criteria. Pod affinity can be expressed as either "required" or "preferred" affinity. Required pod affinity means that the Pod must be scheduled on a node that is already running Pods that match the criteria, while preferred pod affinity means that the scheduler will try to schedule the Pod on a node that is already running Pods that match the criteria, but can still schedule the Pod on a node that does not match the criteria if necessary.

Pod affinity can be further divided into two types:

• requiredDuringSchedulingIgnoredDuringExecution: Pods will be scheduled only on nodes that are already running Pods that match the pod affinity criteria. If no nodes meet the criteria, the Pod will not be scheduled. If a node loses its Pod label after the Pod has been scheduled, the Pod will continue to run on the node until it is terminated.

• preferredDuringSchedulingIgnoredDuringExecution: Pods will be scheduled on nodes that are already running Pods that match the pod affinity criteria if possible. If no nodes meet the criteria, the scheduler will still schedule the Pod on a node that does not match the criteria.

Overall, affinity provides a powerful mechanism to ensure that Pods are scheduled on nodes that meet specific requirements, which can improve the reliability and performance of your Kubernetes cluster.

Pod Affinity & AntiAffinity

• Node affinity allows you to schedule a pod on a set of nodes based on labels present on the nodes. However, in certain scenarios, we might want to schedule certain pods together or we might want to make sure that certain pods are never scheduled together. This can be achieved by PodAffinity and/or PodAntiAffinity respectively.

• Similar to node affinity, there are a couple of variants in pod affinity namely requiredDuringSchedulingIgnoredDuringExecution and preferredDuringSchedulingIgnoredDuringExecution.

Use cases

• While scheduling workload, when we need to schedule a certain set of pods together, PodAffinity makes sense. Example, a web server and a cache.

• While scheduling workload, when we need to make sure that a certain set of pods are not scheduled together, PodAntiAffinity makes sense.

Examples:

Follow through guide.

Let's begin with listing nodes.

kubectl get nodes

You should be able to see the list of nodes available in the cluster,

NAME                          STATUS    ROLES     AGE       VERSION
node1.compute.infracloud.io   Ready     <none>    25m       v1.9.4
node2.compute.infracloud.io   Ready     <none>    25m       v1.9.4
node3.compute.infracloud.io   Ready     <none>    28m       v1.9.4

Pod Affinity example

Let's deploy deployment-Affinity.yaml, which has pod affinity as,

affinity:
  podAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    - labelSelector:
        matchExpressions:
        - key: app
          operator: In
          values:
          - nginx
      topologyKey: "kubernetes.io/hostname"

Here we are specifying that all nginx pods should be scheduled together.

kubectl apply -f deployment-Affinity.yaml

Check the pods using,

kubectl get pods -o wide -w

You should be able to see that all pods are scheduled on the same node.

NAME                                READY     STATUS    RESTARTS   AGE       IP            NODE
nginx-deployment-6bc5bb7f45-49dtg   1/1       Running   0          36m       10.20.29.18   node2.compute.infracloud.io
nginx-deployment-6bc5bb7f45-4ngvr   1/1       Running   0          36m       10.20.29.20   node2.compute.infracloud.io
nginx-deployment-6bc5bb7f45-lppkn   1/1       Running   0          36m       10.20.29.19   node2.compute.infracloud.io

To clean up run,

kubectl delete -f deployment-Affinity.yaml

Pod Anti Affinity example

Let's deploy deployment-AntiAffinity.yaml, which has pod affinity as,

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    - labelSelector:
        matchExpressions:
        - key: app
          operator: In
          values:
          - nginx
      topologyKey: "kubernetes.io/hostname"

Here we are specifying that no two nginx pods should be scheduled together.

kubectl apply -f deployment-AntiAffinity.yaml

Check the pods using,

kubectl get pods -o wide -w

You should be able to see that pods are scheduled on different nodes.

NAME                                READY     STATUS    RESTARTS   AGE       IP            NODE
nginx-deployment-85d87bccff-4w7tf   1/1       Running   0          27s       10.20.29.16   node3.compute.infracloud.io
nginx-deployment-85d87bccff-7fn47   1/1       Running   0          27s       10.20.42.32   node1.compute.infracloud.io
nginx-deployment-85d87bccff-sd4lp   1/1       Running   0          27s       10.20.13.17   node2.compute.infracloud.io

Note: In above example, if number of replicas are more than number of nodes then some pods will remain in pending state.

To clean up run,

kubectl delete -f deployment-AntiAffinity.yaml

Pod Anti-Affinity is a feature in Kubernetes that allows you to ensure that your pods are spread across multiple nodes in a cluster. This helps to increase the availability of your applications by preventing multiple pods from being scheduled on the same node, which could result in a single point of failure.

Anti-affinity can be defined at the pod level, which means that you can specify rules that determine which pods can and cannot be scheduled on the same node. For example, you could define an anti-affinity rule that ensures that no two pods belonging to the same service are scheduled on the same node.

There are two types of pod anti-affinity: required and preferred. Required anti-affinity ensures that no two pods that match a particular label selector are scheduled on the same node. Preferred anti-affinity, on the other hand, attempts to avoid scheduling pods with the same label selector on the same node, but it is not a hard requirement.

You can use pod anti-affinity in combination with other Kubernetes features like node selectors, pod labels, and pod annotations to ensure that your applications are highly available and resilient to failures.