Consumer Groups Are Not Queues: What Kafka Share Groups Change

After running Kafka for a while, many teams arrive at the same point.

The records are already in Kafka, and you want to add more workers. But if the topic has only a few partitions, adding more consumers to the same consumer group does not necessarily add more active work. On the other hand, creating many partitions up front brings its own cost: ordering boundaries, operational overhead, rebalances, file handles, and broker load.

At that point, people start wanting a Kafka topic to behave more like a queue.

Calm miniature illustration of people carrying small glowing spheres from a shared bowl — When the way work is shared changes, the same flow can become easier to scale.

Apache Kafka Share Groups exist to address that gap. They do not add a separate queue product beside Kafka. Instead, they add a new group model that lets multiple consumers cooperatively process records from the same Kafka topic partitions.

This post explains why Share Groups were added, what they mean, and how to start applying them, based on KIP-932: Queues for Kafka and the Apache Kafka 4.3 documentation. Version context matters: Share Groups appeared as early access in Kafka 4.0, moved to preview in 4.1, and were documented as production-ready in Kafka 4.2.

Consumer Groups scale around partition ownership. Share Groups scale around record acquisition and acknowledgement.

The short version

These are the main ideas to keep in mind:

A Consumer Group is a partition ownership model.
A Share Group is a record acquisition and acknowledgement model.
In a Share Group, having more consumers than partitions can still be useful.
The trade-off is weaker ordering within a partition.
Share Groups fit queue-like workloads where records are processed one at a time, not ordered stream processing.

In one sentence:

Consumer Groups split ordered streams across partitions.
Share Groups enable queue-like work sharing on top of Kafka topics.

What Consumer Groups do well

Kafka Consumer Groups are a strong default. Inside a group, each partition is assigned to a specific consumer, and that consumer processes records from the partition in offset order.

This model has real strengths:

Partition-level ordering is easy to reason about.
Offset commits are relatively simple.
Duplicate processing within the same group is reduced.
Increasing partitions increases available parallelism.

But this model is not the same as a queue. If a topic has three partitions and you start twenty consumers in the same group, you usually still get only three active consumers. The rest wait for work or for future partition changes.

This is not a flaw in Kafka. It is a consequence of the model. Consumer Groups are designed for parallel consumption of ordered logs. They keep partition ownership clear, but they do not split work more finely than a partition.

When teams want queue behavior

Many queue workloads are different. Each record is mostly independent.

Think about workloads like these:

image thumbnail generation
email delivery
webhook delivery
external API synchronization
AI inference job dispatch
asynchronous post-processing that can be retried later

In these cases, operators often care less about preserving partition order and more about scaling the number of workers.

work is backing up
-> add more workers
-> more workers process records from the same topic

With a traditional Consumer Group, that flow is constrained by partition count. Share Groups provide another option.

A Share Group is a new kind of group that exists alongside Consumer Groups. Multiple consumer groups and multiple share groups can independently consume from the same topic. The internal consumption model is different.

In a Share Group, a consumer does not exclusively own a partition. Multiple share consumers can receive records from the same partition. When the broker delivers a record to a consumer, that record is acquired for a limited time. While it is acquired, another consumer will not receive the same record.

The processing result is expressed through acknowledgements.

acknowledgement	Meaning
`ACCEPT`	Processing succeeded. The record is completed.
`RELEASE`	Transient failure. The record can be delivered again.
`REJECT`	Permanent failure. The record should not be redelivered.
`RENEW`	Still processing. Extend the acquisition lock.

The default lock duration is 30 seconds and can be adjusted with the group setting share.record.lock.duration.ms. In Kafka 4.3, related share group settings include share.delivery.count.limit, share.partition.max.record.locks, and share.renew.acknowledge.enable.

The important shift is that part of the mental model moves from offsets to record state.

What the code looks like

A minimal share consumer looks familiar at first.

Properties props = new Properties();
props.setProperty("bootstrap.servers", "localhost:9092");
props.setProperty("group.id", "thumbnail-workers");
props.setProperty("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.setProperty("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");

try (KafkaShareConsumer<String, String> consumer = new KafkaShareConsumer<>(props)) {
    consumer.subscribe(List.of("thumbnail-jobs"));

    while (true) {
        ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(1));
        for (ConsumerRecord<String, String> record : records) {
            createThumbnail(record.value());
        }
    }
}

In the default implicit acknowledgement mode, records returned by the previous poll() are acknowledged as successful when the next poll() happens. That is convenient for simple processing, but explicit acknowledgement is usually a better fit when failures need record-level handling.

Properties props = new Properties();
props.setProperty("bootstrap.servers", "localhost:9092");
props.setProperty("group.id", "thumbnail-workers");
props.setProperty("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.setProperty("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.setProperty("share.acknowledgement.mode", "explicit");

try (KafkaShareConsumer<String, String> consumer = new KafkaShareConsumer<>(props)) {
    consumer.subscribe(List.of("thumbnail-jobs"));

    while (true) {
        ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(1));

        for (ConsumerRecord<String, String> record : records) {
            try {
                createThumbnail(record.value());
                consumer.acknowledge(record, AcknowledgeType.ACCEPT);
            } catch (TransientStorageException e) {
                consumer.acknowledge(record, AcknowledgeType.RELEASE);
            } catch (InvalidJobException e) {
                consumer.acknowledge(record, AcknowledgeType.REJECT);
            }
        }

        consumer.commitSync();
    }
}

This code exposes the important decision: not every failure is the same. A storage timeout may be worth retrying. An invalid job payload may never succeed if delivered again.

What you gain and what you give up

The biggest benefit of Share Groups is that partition count and consumer parallelism become less tightly coupled.

Angle	Consumer Group	Share Group
Work distribution unit	partition	record
Active consumer count	strongly tied to partition count	can exceed partition count
Ordering	strong within a partition	weaker
Failure handling	offset commit and retry design	record acknowledgement and delivery count
Best fit	ordered stream processing	independent work, queue-like workloads

This table can make Share Groups look universally better. They are not. They are a different tool.

For ordered event streams, Consumer Groups remain the natural choice. Account transactions, user timeline events, and CDC streams often depend on the meaning of order. If one record must be processed before the next, use Share Groups carefully.

If records are independent, if transient and permanent failures should be handled per record, and if worker elasticity matters more than partition ordering, Share Groups become a strong candidate.

Operational settings to check first

Share Groups also introduce more broker-side and coordinator-side state. Kafka uses an internal topic named __share_group_state for share group state. In a test cluster with fewer than three brokers, you may need to set share.coordinator.state.topic.replication.factor and share.coordinator.state.topic.min.isr to 1 before using share groups.

These are the first settings I would review:

Setting	What to check
`share.record.lock.duration.ms`	Whether record processing fits inside the acquisition lock duration
`share.delivery.count.limit`	How many delivery attempts are allowed before a record stops being redelivered
`share.partition.max.record.locks`	The in-flight record limit per share partition
`share.acquire.mode`	Whether batch efficiency or strict `max.poll.records` control matters more
`share.acknowledgement.mode`	Whether implicit acknowledgement is enough or explicit per-record results are needed

If processing can take longer than the lock duration, RENEW is available. But renewal keeps a record acquired for longer and can affect delivery progress. For long-running work, first ask whether the job can be split into smaller pieces or tracked in an external job state store.

Monitoring has to change too

With Consumer Groups, operators look at lag, committed offsets, rebalances, and partition skew. With Share Groups, backlog and throughput still matter, but record-level state becomes more important.

The dashboard needs to answer questions like these:

Are acquired records building up?
Is redelivery increasing after lock timeout?
Are there more RELEASE or REJECT acknowledgements?
Are records hitting the delivery count limit?
Do failures cluster around specific partitions or key families?
Does throughput actually increase when workers are added?

Share Group operations do not end with “what is the lag?” You also need to know where records are waiting, whether failures are transient or permanent, and whether worker scaling is actually helping.

This operational angle is also central to Konduo. Konduo connects Kafka and other infrastructure resources through plugins so teams can read brokers, topics, consumer groups, metric evidence, and alert response in one operating flow.

Before applying Share Groups, walk through these questions:

Does this workload require strict ordering within a partition?
Can each record be treated as an independent unit of work?
Can failures be cleanly classified as ACCEPT, RELEASE, or REJECT?
Does record processing usually fit inside the lock duration?
Is RENEW really the right model for long-running records?
Is there a real need to scale workers beyond partition count?
Where will poison records be observed and recovered?
Are we adding a new model for a problem that Consumer Groups already solve well enough?

My rule of thumb is simple:

For ordered event streams, start with Consumer Groups.
For independent queue-like work where elastic workers matter, evaluate Share Groups.

Share Groups can remove some of the awkward patterns teams used to make Kafka behave like a queue. But they do not replace every Kafka consumption model. Their real meaning is that Kafka now has a more official choice between ordered log consumption and queue-like work sharing.