Apache Kafka Partitions and Consumer Groups Explained

Apache Kafka partitions explain most of Kafka's behavior: how it scales, what it guarantees about ordering, and how many consumers can work in parallel. A topic is split into partitions, each partition is an append-only log, and within a consumer group each partition is read by exactly one consumer. Once that model is clear, the rest of Kafka follows from it.

This article covers the core model: log vs queue, topics, partitions and offsets, keys and ordering, consumer groups, and retention. Operating the brokers is covered in running Kafka with KRaft on Kubernetes.

Kafka is a log, not a queue

In a traditional queue such as RabbitMQ, a message is gone once a consumer acknowledges it. The queue works like a mailbox: take the letter out and it's no longer there.

Kafka stores messages in an append-only log. Reading a message doesn't remove it. Messages stay until the retention policy deletes them by age or size, whether anyone has read them or not. It works more like a notebook: new entries go at the end, and anyone can open it at any page.

That design choice gives you three things:

• Replay. A consumer can go back and reprocess old messages after a bug fix or a new feature.
• Independent readers. Many systems can read the same stream at their own pace without the producer knowing about them.
• Late joiners. A new service can start from the beginning of the retained log and build its state from history.

It also means Kafka doesn't track acknowledgements per message. Each consumer group tracks a single position per partition. If you need per-message routing, priorities or delayed retries, a queue broker is often the better fit, as described in RabbitMQ exchanges and reliable messaging.

Core Kafka building blocks

A record has an optional key, a value, a timestamp and optional headers. The key matters more than it first appears.

Apache Kafka partitions: the unit of scale and ordering

Partitions spread the load

A topic with six partitions is six independent logs. They are spread across the brokers, and each partition has one leader broker that handles its writes. Because leaders sit on different brokers, writes to one topic are spread across the cluster instead of going through a single machine.

Reads scale the same way: each partition can be consumed by a different consumer in parallel.

Ordering is guaranteed per partition only

Kafka guarantees order within a partition, not across a topic. Records in partition 0 are read in the order they were written. Records in partition 0 and partition 3 have no ordering relationship at all.

So if events must be processed in order, such as created, paid and cancelled for the same booking, they must land in the same partition. That's what keys are for.

Keys decide the partition

When a producer sends a record with a key, the default partitioner hashes the serialized key (murmur2 in the Java client) and takes the result modulo the partition count:

partition = hash(key) % numPartitions

The same key always maps to the same partition as long as the partition count doesn't change. Use a business identifier such as bookingId or userId as the key, and all events for that entity stay in order.

Records without a key are spread across partitions. Since Kafka 3.3, the producer fills a batch for one partition before moving to the next, instead of rotating per record. That batches better, but gives no ordering between those records.

Three consequences to plan for:

• Adding partitions remaps keys. Going from 6 to 12 partitions changes hash(key) % n for many keys, so events for one booking can end up split across two partitions during the change. Choose a partition count with headroom up front.
• You can't reduce partitions. Kafka only allows increasing the count. To go down, create a new topic and migrate.
• Hot keys make hot partitions. If one customer produces half your traffic, their partition becomes the bottleneck no matter how many partitions exist.

Producing with a key in Java

Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.ACKS_CONFIG, "all");              // default since Kafka 3.0
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true); // default since Kafka 3.0

try (var producer = new KafkaProducer<String, String>(props)) {
    var record = new ProducerRecord<>("bookings", "booking-8812", "{\"status\":\"PAID\"}");
    producer.send(record, (meta, ex) -> {
        if (ex != null) {
            System.err.println("send failed: " + ex.getMessage());
            return;
        }
        System.out.printf("partition=%d offset=%d%n", meta.partition(), meta.offset());
    });
}

Every record keyed booking-8812 goes to the same partition, so that booking's events are consumed in order.

Offsets: how consumers know where they are

A consumer group stores a committed offset for each partition: the position of the next record to read. Offsets live in Kafka itself, in the internal __consumer_offsets topic, not on the consumer's machine. When a consumer crashes and restarts, or another consumer takes over its partition, reading resumes from the last committed offset.

When you commit decides your delivery semantics:

• Commit after processing gives at-least-once delivery. A crash between processing and commit means some records are processed again, so make handlers idempotent.
• Commit before processing gives at-most-once delivery. A crash can skip records.
• Exactly-once is possible for Kafka-to-Kafka pipelines using transactions. Once an external database is involved, you're back to idempotent writes.

auto.offset.reset decides where a group starts when it has no committed offset. earliest reads from the start of the retained log, and latest, the default, reads only new records.

A consumer that commits after processing:

Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "analytics");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");

try (var consumer = new KafkaConsumer<String, String>(props)) {
    consumer.subscribe(List.of("bookings"));
    while (true) {
        ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
        for (ConsumerRecord<String, String> r : records) {
            System.out.printf("p=%d offset=%d key=%s%n", r.partition(), r.offset(), r.key());
        }
        consumer.commitSync(); // commit only after the batch is processed
    }
}

Replay is just an offset reset. Stop the group's consumers first, then:

kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --group analytics --topic bookings \
  --reset-offsets --to-datetime 2026-09-01T00:00:00.000 --execute

Kafka consumer groups: the two rules

Rule 1: consumers with the same group ID share the work. Kafka assigns each partition to exactly one consumer in the group. Three consumers in group analytics reading a six-partition topic get two partitions each, and each record is processed once within the group.

Rule 2: different group IDs each get everything. Groups analytics and email both read every record in bookings, independently, each with its own offsets. One booking event is processed once by analytics and once by email.

Put simply, the topic says what data exists, and the group says who reads it and how the work is split.

Partition count caps consumer parallelism

Because each partition goes to one consumer per group, the partition count is the ceiling on a group's parallelism. With three partitions, a fourth consumer in the same group sits idle. It can still take over if another consumer dies, but it adds no throughput.

When consumers join or leave, or one stops polling for longer than max.poll.interval.ms, the group rebalances and partitions are reassigned. Frequent rebalances are a common cause of lag. Keep the work per poll bounded and configure the CooperativeStickyAssignor, so only the partitions that actually move are paused. Kafka 4.0 also made a new consumer group protocol generally available (group.protocol=consumer), which moves assignment to the broker and further reduces disruption.

Check a group's state and lag:

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group analytics

The LAG column shows how far each partition's committed offset trails the end of the log. Lag growing on only one partition usually points to a hot key.

Retention and log compaction

Kafka deletes data by policy, never because someone read it:

• retention.ms sets how long records are kept. The broker default is 7 days.
• retention.bytes caps size per partition, and is unlimited by default.
• Deletion removes whole segment files, so records can outlive the limit somewhat.

With cleanup.policy=compact, Kafka keeps at least the latest record for each key and removes older ones. The topic becomes a changelog of current state, such as the latest profile per user. A record with a null value, called a tombstone, marks its key for deletion.

Create a topic with explicit settings (14 days of retention here):

kafka-topics.sh --bootstrap-server localhost:9092 --create \
  --topic bookings --partitions 12 --replication-factor 3 \
  --config retention.ms=1209600000 --config min.insync.replicas=2

When Kafka is the right tool

Kafka fits well when:

• You need an event stream. Bookings, searches and clicks are recorded as events that other services react to.
• Several systems consume the same events. Analytics, email, fraud detection and the data warehouse each read with their own consumer group. Realtime delivery can be one more consumer, as in scaling WebSockets with pub/sub.
• You need replay. Reprocess history after a bug fix, or build a new service from past events. This is why Kafka often sits under event sourcing and CQRS.
• Throughput is high. Sequential appends to disk, producer batching and zero-copy transfer from disk to network let Kafka handle volumes that per-message brokers struggle with.

It's a weaker fit for request/response workflows, per-message priorities or delays, and small systems where one queue would do. Running a Kafka cluster has a real cost.

FAQ

How many partitions should a Kafka topic have?

Start from the parallelism you need: at least the largest number of consumers you expect in one group, plus headroom, because adding partitions later remaps keys. Very high counts add overhead on brokers and clients, so don't default to hundreds without a reason.

Does Kafka guarantee message order?

Only within a single partition. Give related records the same key so they land in the same partition.

What happens if there are more consumers than partitions?

The extra consumers in that group get no partitions and stay idle until a rebalance gives them one.

Does Kafka delete messages after they are consumed?

No. Records are removed by retention time or size, or by compaction, whether or not they were read.

What is the difference between a topic and a consumer group?

A topic is where data is written. A consumer group is a set of readers that split a topic's partitions between them and track their own offsets.

Key takeaways

• Kafka is a log. Reading doesn't delete, so replay and many independent readers come built in.
• Partitions are the unit of both scale and ordering. Choose the count with headroom.
• Key anything that must stay in order by its business ID.
• The same group ID splits the work, and different group IDs each get everything.
• Commit after processing and make handlers idempotent.

Next, see how brokers, controllers and replication fit together in running Kafka with KRaft on Kubernetes. If you're deciding whether Kafka fits a system you're building, Vectorkub can help with the design.