Running Kafka with KRaft on Kubernetes: A Practical Setup

Running Kafka with KRaft on Kubernetes comes down to a few decisions: where the controllers run, a StatefulSet with one persistent volume per broker, a headless Service that gives every broker a stable DNS name, and advertised.listeners set to that name so clients can reach the right partition leader. Get these right and Kafka behaves on Kubernetes much as it does on VMs. Get advertised.listeners wrong and clients connect, fetch metadata, and then fail on every request.

Topics, partitions and consumer groups are covered in Apache Kafka partitions and consumer groups. This article is about operating the cluster: roles, KRaft, replication settings and Kubernetes manifests.

Brokers and controllers: Kafka's data plane and control plane

A Kafka cluster has two layers, much like Kubernetes separates its control plane from its worker nodes.

Brokers (data plane) store partitions on disk, serve reads and writes from clients, and replicate data from partition leaders to followers. Each broker holds a mix of leaders and followers across many topics, and a replica is never placed on the same broker as its leader. Kafka spreads leaders evenly so that write load doesn't pile up on one machine.

Controllers (control plane) don't store your messages. They hold cluster metadata: which topics exist, which broker leads each partition, where the replicas live, and which brokers are alive. When a broker dies, the active controller elects new leaders from the in-sync replicas.

Controllers run as a quorum, usually three, or five in large clusters. One is active and the others follow the metadata log, ready to take over. A majority must be up, so three controllers tolerate one failure and five tolerate two.

ZooKeeper vs KRaft

Before KRaft, the control plane was a separate ZooKeeper ensemble, so you deployed, secured, monitored and upgraded two distributed systems.

KRaft (Kafka Raft) moves the controller into Kafka itself. Metadata lives in a Raft-replicated log inside the controller quorum.

The practical wins are one system to run, faster controller failover, and room for many more partitions per cluster. New clusters should use KRaft. Existing ZooKeeper clusters must migrate to KRaft on a 3.x release before they can upgrade to 4.x.

Process roles

Every KRaft node sets process.roles:

• broker stores data and serves clients.
• controller joins the metadata quorum.
• broker,controller is combined mode. It's simpler and fine for development and small clusters, but Kafka's documentation doesn't recommend it for critical deployments, because a busy broker can slow the controller down.

KRaft nodes use node.id, not the ZooKeeper-era broker.id, and brokers and controllers share one ID space.

Multi-broker setup checklist

• At least three brokers, so topics can use a replication factor of 3.
• Three dedicated controllers in production (five for very large clusters).
• default.replication.factor=3, plus the same for internal topics: offsets.topic.replication.factor and transaction.state.log.replication.factor.
• min.insync.replicas=2, with producers using acks=all.
• unclean.leader.election.enable=false (the default), so a stale replica can't become leader and silently drop data.
• One persistent disk per broker, never shared.
• listeners and advertised.listeners set to addresses clients can actually reach.
• Rack awareness (broker.rack) set to the availability zone, so replicas of a partition land in different zones.

What replication factor 3 with min.insync.replicas 2 gives you

With replication.factor=3, each partition has a leader and two followers on different brokers. With min.insync.replicas=2 and acks=all, a write succeeds only after at least two replicas have it. In a three-broker cluster:

The cluster keeps accepting writes through one broker failure, which covers a rolling restart or a node drain, and refuses writes rather than risk losing acknowledged data in a larger outage. Setting min.insync.replicas equal to the replication factor sounds safer, but then any single broker restart blocks writes.

How producers find the partition leader

Clients don't push every record through a load balancer. They route it themselves:

1. The client connects to any address in bootstrap.servers and requests metadata.
2. The metadata lists every broker's advertised address and the leader of each partition.
3. For each record, the producer picks the partition from the key, looks up that partition's leader, and sends the batch straight to that broker.
4. If leadership moves, the broker answers with an error such as NOT_LEADER_OR_FOLLOWER, and the client refreshes metadata and retries.

This is why advertised.listeners matters so much. The bootstrap address only gets the client in the door. After that, the client connects to whatever address each broker advertises. If a broker advertises localhost:9092, or a pod IP the client can't reach, bootstrap works and produce requests fail.

Consumers also read from partition leaders by default. If you set broker.rack on brokers, client.rack on consumers and replica.selector.class=org.apache.kafka.common.replica.RackAwareReplicaSelector, consumers can fetch from a follower in their own zone and cut cross-zone traffic.

Kafka KRaft on Kubernetes: the building blocks

A Deployment is the wrong tool: its pods get random names, have no stable identity and can't have per-pod volumes. Kubernetes workloads compares the options in detail.

One headless Service covers every broker

A common misconception is that each broker needs its own Service. It doesn't. A headless Service (clusterIP: None) has no virtual IP and does no load balancing. Paired with a StatefulSet, it creates a DNS record for every pod, such as kafka-0.kafka-headless.kafka.svc.cluster.local. Scale to four brokers and kafka-3 gets its name automatically.

Add a normal ClusterIP Service for bootstrap, since any broker can answer a metadata request. Producers and consumers don't need a Service of their own, because they only make outbound connections.

apiVersion: v1
kind: Service
metadata:
  name: kafka-headless
  namespace: kafka
spec:
  clusterIP: None
  publishNotReadyAddresses: true # controllers must resolve each other before they are Ready
  selector:
    app: kafka
  ports:
    - name: broker
      port: 9092
    - name: controller
      port: 9093
---
apiVersion: v1
kind: Service
metadata:
  name: kafka-bootstrap
  namespace: kafka
spec:
  selector:
    app: kafka
  ports:
    - name: broker
      port: 9092

A three-node KRaft StatefulSet

This manifest runs three combined-mode nodes with the official apache/kafka image. It's a good way to learn the moving parts and is fine for development. For production, split controllers and brokers into separate StatefulSets, or use an operator.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka
  namespace: kafka
spec:
  serviceName: kafka-headless
  replicas: 3
  podManagementPolicy: Parallel
  selector:
    matchLabels:
      app: kafka
  template:
    metadata:
      labels:
        app: kafka
    spec:
      securityContext:
        fsGroup: 1000
      containers:
        - name: kafka
          image: apache/kafka:3.9.1
          ports:
            - name: broker
              containerPort: 9092
            - name: controller
              containerPort: 9093
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: KAFKA_NODE_ID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.labels['apps.kubernetes.io/pod-index']
            - name: CLUSTER_ID
              value: "q1Sh-9_ISia_zwGINzRvyQ" # generate once: kafka-storage.sh random-uuid
            - name: KAFKA_PROCESS_ROLES
              value: "broker,controller"
            - name: KAFKA_LISTENERS
              value: "PLAINTEXT://:9092,CONTROLLER://:9093"
            - name: KAFKA_ADVERTISED_LISTENERS
              value: "PLAINTEXT://$(POD_NAME).kafka-headless.kafka.svc.cluster.local:9092"
            - name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
              value: "PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT"
            - name: KAFKA_CONTROLLER_LISTENER_NAMES
              value: "CONTROLLER"
            - name: KAFKA_INTER_BROKER_LISTENER_NAME
              value: "PLAINTEXT"
            - name: KAFKA_CONTROLLER_QUORUM_VOTERS
              value: "[email protected]:9093,[email protected]:9093,[email protected]:9093"
            - name: KAFKA_LOG_DIRS
              value: "/var/lib/kafka/data/logs"
            - name: KAFKA_DEFAULT_REPLICATION_FACTOR
              value: "3"
            - name: KAFKA_MIN_INSYNC_REPLICAS
              value: "2"
            - name: KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR
              value: "3"
            - name: KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR
              value: "3"
            - name: KAFKA_TRANSACTION_STATE_LOG_MIN_ISR
              value: "2"
          readinessProbe:
            tcpSocket:
              port: 9092
            periodSeconds: 10
          resources:
            requests:
              cpu: "1"
              memory: 4Gi
          volumeMounts:
            - name: data
              mountPath: /var/lib/kafka/data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 100Gi

Details that matter:

• podManagementPolicy: Parallel starts all pods together. With the default OrderedReady, kafka-0 waits to become Ready before kafka-1 starts, but the controller quorum needs a majority of pods running before anything works.
• node.id comes from the pod index. The apps.kubernetes.io/pod-index label (Kubernetes 1.28 and later) gives each pod its ordinal, which matches the IDs in the voter list.
• advertised.listeners uses the pod's own DNS name, so metadata tells clients exactly where each leader is.
• The log directory is a subdirectory of the volume. Many volumes have a lost+found directory at the root, which Kafka would try to load as a partition.
• CLUSTER_ID is generated once and never changed. The image formats empty storage with it on first start.

This setup serves clients inside the cluster only. Clients outside Kubernetes must reach each broker individually, which means a per-broker LoadBalancer or NodePort and a second listener that advertises those external addresses.

Add a PodDisruptionBudget so node drains take one broker at a time, then check the quorum and create a topic:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: kafka
  namespace: kafka
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: kafka

kubectl -n kafka exec kafka-0 -- /opt/kafka/bin/kafka-metadata-quorum.sh \
  --bootstrap-server localhost:9092 describe --status

kubectl -n kafka exec kafka-0 -- /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 --create --topic bookings \
  --partitions 12 --replication-factor 3

Running Kafka with an operator: Strimzi

Hand-written manifests leave you responsible for ISR-aware rolling restarts, certificate rotation, rack awareness, external listeners and scaling. An operator encodes that knowledge in code. Strimzi is the most widely used Kafka operator: you declare a Kafka resource and KafkaNodePool resources for controllers and brokers, and it creates the pods, volumes and Services. Recent Strimzi releases support KRaft only.

Strimzi uses the same two-Service pattern described above: a bootstrap Service (<cluster>-kafka-bootstrap) and a headless Service for the brokers (<cluster>-kafka-brokers), whether you run 3 brokers or 30. How operators work in general is covered in Kubernetes operators and CRDs.

FAQ

Does Kafka still need ZooKeeper?

No. KRaft has been production-ready since Kafka 3.3, and Kafka 4.0 removed ZooKeeper support entirely.

How many KRaft controllers do I need?

Three for most clusters, which tolerates one failure. Five tolerates two. Use an odd number, because a fourth controller adds no extra fault tolerance.

Why can my client connect to Kafka but not produce?

Almost always advertised.listeners. Bootstrap succeeds, but a broker advertises an address the client can't reach. Check what each broker advertises and resolve it from the client's network.

Should Kafka on Kubernetes use a Deployment or a StatefulSet?

A StatefulSet, or an operator that manages pods with the same guarantees. Brokers need stable names, stable IDs and their own disks.

Takeaways

• Run KRaft. ZooKeeper mode is gone in Kafka 4.0.
• Use three dedicated controllers in production and keep combined mode for development.
• Replication factor 3, min.insync.replicas=2 and acks=all is the standard durability baseline.
• StatefulSet, per-broker volumes, a headless Service and a bootstrap Service are the core Kubernetes pieces.
• Set advertised.listeners to an address every client can reach.

Kafka on Kubernetes builds on StatefulSets, storage and Services, and Vectorkub's free DevOps courses cover those basics.