Kubernetes Requests and Limits: Scheduling and QoS

Kubernetes requests and limits are the two numbers that decide where a pod runs and what happens when it uses too much. The scheduler places pods using requests, the CPU and memory a container reserves. The kernel enforces limits, the ceiling a container can't exceed. Go over the CPU limit and the container is throttled. Go over the memory limit and it's killed. From those two settings Kubernetes also derives a QoS class, which affects which pods lose out when a node runs short of memory.

This guide covers how the scheduler filters and scores nodes, how requests and limits actually behave, QoS classes and eviction, and how to spread pods with anti-affinity, topology spread constraints and taints so one failed node doesn't take your app down.

How the Kubernetes scheduler picks a node

When a pod is created without a node assigned, kube-scheduler picks one in two phases.

1. Filtering removes nodes that can't run the pod:

• Not enough unrequested CPU or memory for the pod's requests
• A taint the pod doesn't tolerate
• A nodeSelector or required node affinity that doesn't match
• A required pod anti-affinity or topology spread rule that would be violated
• A volume that can't be attached in that zone, or a PVC that isn't bound

2. Scoring ranks the remaining nodes. Scoring plugins favor, for example, nodes with more free resources, balanced CPU and memory use, the container image already cached, and better spread across zones. The highest score wins.

If no node passes filtering, the pod stays Pending. If a lower-priority pod could be removed to make room, the scheduler may preempt it (see PriorityClass below). The scheduler only decides. It writes the chosen node to the pod through the API server, and the kubelet on that node starts the containers. For how those components fit together, see Kubernetes architecture: control plane and nodes.

Kubernetes requests vs limits

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
        - name: api
          image: registry.example.com/api:1.8.2
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
            limits:
              memory: 256Mi

• requests is what the container reserves. The scheduler adds up the requests of all pods on a node and compares the total with the node's allocatable capacity. It does not look at actual usage.
• limits is the maximum. The kubelet and container runtime configure the Linux cgroup to enforce it.

Units: 1 CPU is one vCPU or core, and 250m is a quarter of one. Memory uses Mi and Gi (powers of 2). M and G are powers of 10, and mixing them up is a common source of small errors.

Because scheduling is based on requests, setting requests too high makes nodes look full while they sit mostly idle. Setting them too low packs too many pods onto a node, and they fight over the real resources. Measure actual usage, from your metrics or from VPA in recommendation mode, and set requests near typical usage. Kubernetes autoscaling with HPA, VPA and Cluster Autoscaler explains how the autoscalers depend on the same numbers.

CPU throttling vs OOMKill

The two resources behave differently when a container hits its limit:

CPU throttling is easy to miss. The limit is enforced per 100ms period. A multi-threaded service with a 500m limit can use up its whole quota in the first 25ms of a period and then stall for 75ms, even though average CPU usage looks low. If p99 latency is bad but CPU graphs look fine, check the throttling metrics.

Repeated OOMKills show up as a restart loop. Debugging CrashLoopBackOff and probes covers how to tell an OOMKill apart from a failing probe.

Should you set CPU limits?

This is a real trade-off, not a rule:

• Memory: always set a limit, and usually set it equal to the request. Memory overcommit leads to unpredictable kills.
• CPU: many teams set a CPU request but no CPU limit for latency-sensitive services. The request still guarantees each pod its share under contention, and idle CPU on the node can absorb bursts instead of being throttled.
• Set CPU limits when you need strict isolation, such as multi-tenant clusters, batch jobs that would otherwise take over a node, or when you need Guaranteed QoS.

If you set only limits and no requests, Kubernetes copies the limits into the requests. That's safe, but it may reserve more than you meant to.

QoS classes and eviction

Kubernetes assigns every pod a QoS class based on its resources:

Check a pod's class:

kubectl get pod api-7d9f8c6b5-x2k4q -o jsonpath='{.status.qosClass}'

When a node runs low on memory, the kubelet evicts pods to protect itself. It ranks candidates by:

1. whether the pod's usage exceeds its requests,
2. pod priority,
3. how far usage exceeds requests.

In practice, BestEffort pods go first because any usage exceeds a zero request. Burstable pods running above their requests come next. Guaranteed pods, and Burstable pods staying within their requests, are evicted last. If memory runs out before the kubelet reacts, the kernel OOM killer steps in, and it also uses QoS: BestEffort processes are the most likely targets and Guaranteed the least.

For workloads that must not be evicted, make them Guaranteed and give them a higher PriorityClass. Priority also lets the scheduler preempt lower-priority pods when the cluster is full.

To stop BestEffort pods appearing by accident, add a LimitRange to each namespace. It fills in default requests and limits for containers that don't specify them. A ResourceQuota caps the total a namespace can request.

Spreading pods for reliability

If all three replicas land on one node and that node dies, the app goes down even though you "have three replicas". Kubernetes gives you three tools to prevent that.

Pod anti-affinity

Keep replicas of the same app off the same node:

    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                topologyKey: kubernetes.io/hostname
                labelSelector:
                  matchLabels:
                    app: api

required... is a hard rule, and the pod stays Pending if it can't be met. preferred... is best effort. Use required carefully. With 5 replicas and 4 nodes, the fifth pod can never schedule.

nodeAffinity works the other way round: it attracts a pod to nodes with certain labels, for example nodes with local SSDs.

Topology spread constraints

Anti-affinity only says "not together". Topology spread constraints say "keep the counts balanced" across zones, racks or nodes:

    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: api
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app: api

maxSkew: 1 means no zone may have more than one pod more than the least-populated zone. Losing one zone then costs you roughly a third of capacity in a three-zone cluster, not half or all of it. On bare metal, label nodes with a rack and use that as the topologyKey.

Taints and tolerations

Affinity is the pod choosing nodes. Taints are the node refusing pods:

kubectl taint nodes gpu-node-1 dedicated=gpu:NoSchedule

Only pods with a matching toleration can land there:

    spec:
      tolerations:
        - key: dedicated
          operator: Equal
          value: gpu
          effect: NoSchedule

A toleration only allows the pod onto the node. It doesn't send it there. To keep GPU nodes for GPU work and also make GPU pods go to them, combine the taint with node affinity on a node label.

Debugging a Pending pod

Pending almost always means filtering rejected every node. The events tell you why:

kubectl describe pod api-7d9f8c6b5-x2k4q

Look at the Events section for a message similar to:

0/6 nodes are available: 3 Insufficient memory, 3 node(s) had untolerated taint {dedicated: gpu}.

Work through the usual causes:

1. Insufficient CPU or memory: requests are larger than any node's free allocatable capacity. Lower the requests, or let the Cluster Autoscaler add a node.
2. Affinity or nodeSelector mismatch: no node carries the required label.
3. Untolerated taint: every node with room is tainted.
4. Topology or anti-affinity: a required or DoNotSchedule rule can't be satisfied with the current nodes.
5. Unbound PVC: the pod is waiting on a volume. See Kubernetes storage: PV, PVC and StorageClass.

FAQ

What is the difference between requests and limits in Kubernetes?

Requests are what a container reserves, and the scheduler uses them to place pods. Limits are the hard ceiling at runtime. Exceeding the CPU limit causes throttling, and exceeding the memory limit gets the container OOMKilled.

Why is my pod OOMKilled when the node has free memory?

The container went over its own memory limit. Node free memory doesn't matter, because the cgroup limit is per container. Raise the limit or reduce the app's memory use.

How do I make a pod Guaranteed QoS?

Every container in the pod must set CPU and memory requests and limits, with requests equal to limits. Init containers count too.

Does the scheduler use actual CPU usage?

No. It uses the sum of requests on each node. A node at 10% real CPU can still reject pods if its requests are fully allocated.

Is it bad to run without CPU limits?

Not necessarily. With accurate CPU requests, leaving CPU limits off avoids throttling and lets pods use idle CPU. Keep memory limits in place.

Requests and limits checklist

• Set CPU and memory requests on every container, based on measured usage.
• Set memory limit equal to memory request. Decide on CPU limits deliberately.
• Put a LimitRange in each namespace so nothing ends up BestEffort by accident.
• Make critical workloads Guaranteed, with a higher PriorityClass.
• Spread replicas across nodes and zones with topology spread constraints.
• Reserve special hardware with taints, and pair them with node affinity.
• Start any Pending investigation with kubectl describe pod and its events.

With these settings in place, the scheduler packs nodes efficiently and one bad node or noisy pod can't take a whole service down. To work through scheduling and resource tuning on a live cluster, Vectorkub offers free DevOps courses.