Kubernetes Probes and CrashLoopBackOff: A Debugging Guide

CrashLoopBackOff is not an error in itself. It is the status Kubernetes shows when a container keeps exiting and the kubelet is waiting longer and longer before restarting it. The real problem is whatever made the container exit, and finding it usually takes three commands: kubectl describe pod, kubectl logs --previous, and a look at the exit code.

Probes are closely related. A misconfigured liveness probe is one of the most common reasons a healthy application ends up in a restart loop, and a missing readiness probe is a common reason users see errors during a deploy. This guide covers both: how liveness, readiness and startup probes work, and a step-by-step routine for debugging CrashLoopBackOff.

What CrashLoopBackOff actually means

With restartPolicy: Always (the only option for Deployments), the kubelet restarts a container every time it exits, whether it crashed or finished cleanly. If it keeps exiting, the kubelet adds an exponential back-off between restarts: 10 seconds, then 20, 40, and so on, capped at five minutes. While the kubelet waits, the pod shows CrashLoopBackOff. The back-off timer resets once a container has run for 10 minutes without a problem.

$ kubectl get pods
NAME                     READY   STATUS             RESTARTS      AGE
api-7d9f8c6b5-x2kqp      0/1     CrashLoopBackOff   6 (2m ago)    9m

Two things follow from this:

• The container did start. Image pull problems show up as ImagePullBackOff, and a missing ConfigMap or Secret key shows up as CreateContainerConfigError. CrashLoopBackOff means the process ran and then exited.
• Deleting the pod rarely helps. The new pod runs the same image with the same config and crashes the same way.

If an init container is the one failing, the status reads Init:CrashLoopBackOff. The debugging steps are the same. Just add -c <init-container-name> to the log commands.

Liveness, readiness and startup probes

The kubelet on each node runs probes against your containers (see Kubernetes architecture for where the kubelet fits). Each probe type answers a different question and triggers a different action.

Each probe can use one of four mechanisms: httpGet (any 2xx or 3xx status counts as success), tcpSocket, exec (exit code 0 means success), or grpc for services that implement the gRPC health checking protocol.

The timing fields apply to all three probe types:

The default one-second timeout is often too short for an endpoint that touches the network. Under load, a slow health check fails three times in a row and the kubelet restarts a container that was only busy.

A sensible probe configuration

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
        - name: api
          image: registry.example.com/api:1.8.2
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
            limits:
              memory: 512Mi
          startupProbe:
            httpGet:
              path: /livez
              port: 8080
            periodSeconds: 5
            failureThreshold: 24     # up to 120s to start
          livenessProbe:
            httpGet:
              path: /livez
              port: 8080
            periodSeconds: 10
            timeoutSeconds: 2
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /readyz
              port: 8080
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 2

The startup probe gives a slow-starting app up to failureThreshold × periodSeconds (here 120 seconds) to come up. Once it succeeds, the liveness probe takes over with a much tighter budget. This replaces the old habit of setting a large initialDelaySeconds on the liveness probe, which delays failure detection for the whole life of the container.

What each health endpoint should check

The most important rule: liveness checks the process, readiness checks whether the pod can serve traffic.

package main

import (
	"context"
	"database/sql"
	"net/http"
	"sync/atomic"
	"time"
)

type health struct {
	db       *sql.DB
	draining atomic.Bool
}

// Liveness: can this process still handle a request at all?
// No external calls. If the database is down, restarting us won't fix it.
func (h *health) livez(w http.ResponseWriter, _ *http.Request) {
	w.WriteHeader(http.StatusOK)
}

// Readiness: should the Service send us traffic right now?
func (h *health) readyz(w http.ResponseWriter, r *http.Request) {
	if h.draining.Load() {
		http.Error(w, "shutting down", http.StatusServiceUnavailable)
		return
	}
	ctx, cancel := context.WithTimeout(r.Context(), time.Second)
	defer cancel()
	if err := h.db.PingContext(ctx); err != nil {
		http.Error(w, "db unavailable", http.StatusServiceUnavailable)
		return
	}
	w.WriteHeader(http.StatusOK)
}

func (h *health) register(mux *http.ServeMux) {
	mux.HandleFunc("GET /livez", h.livez)
	mux.HandleFunc("GET /readyz", h.readyz)
}

If the liveness endpoint pinged the database, a short database outage would fail liveness on every replica at once. Kubernetes would restart all of them, and they would come back into a thundering herd of reconnects. That turns a dependency blip into a full outage. A readiness failure only takes the pod out of the Service endpoints, which is the right response to "I can't serve right now".

The draining flag is useful for shutdown: set it when the process receives SIGTERM, so readiness fails and the pod stops receiving new requests before it exits. That pairs well with the techniques in zero-downtime rolling updates.

Debugging CrashLoopBackOff step by step

Work through these in order. Most cases are solved by step 3.

1. Describe the pod

kubectl describe pod api-7d9f8c6b5-x2kqp

Look at two sections. Under the container, Last State tells you how the previous run ended:

    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Thu, 24 Sep 2026 10:02:11 +0700
      Finished:     Thu, 24 Sep 2026 10:02:19 +0700

At the bottom, Events shows what the kubelet did. Lines like Liveness probe failed: ... context deadline exceeded followed by Container api failed liveness probe, will be restarted point straight at the probe. Back-off restarting failed container only confirms the loop.

2. Read the logs from the crashed run

kubectl logs api-7d9f8c6b5-x2kqp --previous
kubectl logs api-7d9f8c6b5-x2kqp -c migrate --previous   # specific container

Plain kubectl logs shows the current container, which may have just started and printed nothing. --previous shows the output of the run that died, which is where the stack trace or the missing required env DATABASE_URL line usually is.

3. Interpret the exit code

Exit code 137 with Reason: OOMKilled means the container exceeded its memory limit. Either the limit is too low for the real workload or the app has a leak. Requests, limits and QoS classes explains how to size them.

4. Check events across the namespace

kubectl get events --sort-by=.lastTimestamp
kubectl get events --field-selector involvedObject.name=api-7d9f8c6b5-x2kqp

Events expire after about an hour by default, so check them early. They also show problems that aren't attached to the container, such as a volume that failed to mount.

5. Get a shell when the logs aren't enough

A container that crashes in two seconds is hard to exec into. Two options:

# Attach an ephemeral debug container that shares the target's process namespace
kubectl debug -it api-7d9f8c6b5-x2kqp --image=busybox:1.36 --target=api

# Or copy the pod, replacing the command so it stays up
kubectl debug api-7d9f8c6b5-x2kqp -it --copy-to=api-debug --container=api -- sh

The copy runs the same image, env and volumes, so you can check config files, run the binary by hand and see the error directly. Delete the copy when you're done.

Common causes and fixes

FAQ

How long does CrashLoopBackOff wait between restarts?

The delay starts at 10 seconds and doubles after each crash, up to a cap of five minutes. It resets after the container runs for 10 minutes without exiting.

Should the liveness and readiness probes use the same endpoint?

Usually not. Liveness should only check that the process itself is responsive. Readiness can check dependencies and shutdown state. Pointing both at an endpoint that checks the database means a database outage restarts every pod.

Why does my pod restart even though the logs show no error?

Check the events for liveness probe failures. The kubelet may be killing a healthy but slow container because the probe timed out. Also check Last State for OOMKilled, which leaves no application log line.

What does exit code 137 mean in Kubernetes?

The process was killed with SIGKILL. Most often that's the kernel's OOM killer enforcing the memory limit, shown as Reason: OOMKilled. It can also mean the container ignored SIGTERM during shutdown and was force-killed after the grace period.

Do I always need a startup probe?

No. It's worth adding when startup time varies or is long, for example JVM apps, apps that warm caches, or apps that run migrations. For a Go binary that starts in under a second, liveness and readiness are enough.

A quick checklist

1. kubectl describe pod: read Last State, the exit code and the events.
2. kubectl logs --previous: read the output of the run that crashed.
3. Map the exit code to a cause: config, OOM, probe, command or permissions.
4. Keep liveness cheap and dependency-free, and use readiness to take pods out of rotation.
5. Add a startup probe for slow starters instead of a long initialDelaySeconds.

Review probe settings whenever an app's startup time or dependencies change. If you want to practise this kind of troubleshooting on a real cluster, Vectorkub runs free DevOps courses that cover it.