Kafka EOS: What Is Actually Exactly Once?

You enabled idempotence on a Kafka producer. A consumer application still received the same order event twice.

Both records had the same payload. Both used order-123 as the key, and both carried event_id=evt-789 in a header. Kafka stored them at two different offsets.

orders-0 offset 100: key=order-123, event_id=evt-789
orders-0 offset 101: key=order-123, event_id=evt-789

Does this mean Kafka Exactly-Once Semantics, or EOS, failed?

Not necessarily. Kafka’s definition of a duplicate is different from an application’s definition of the same business event.

A duplicate Kafka can identify
= a retried record batch with the same producer protocol identity

A duplicate the application can identify
= the same event_id, idempotency key, or business meaning

To understand Kafka EOS, we first have to ask what, exactly, is supposed to happen once.

One Phrase Carries Several Different Expectations

When developers hear “exactly once,” they often imagine all of the following:

Kafka stores only one message with the same content.
A consumer handler runs only once for each message.
A database update or payment API call happens only once.

Kafka does not promise all three at once. Its guarantees live at different layers.

Layer	Guarantee at this layer	What it does not provide automatically
Idempotent producer	Prevents a client retry from appending the same batch twice	Deduplication of two application-level sends with the same payload
Kafka transaction	Atomic commit or abort across multiple Kafka partition writes	Atomicity with an ordinary database transaction or external API
Exactly-once processing	Atomic Kafka input offsets and Kafka outputs	Exactly-once business side effects outside Kafka
Application idempotency	Identifies a previously handled event_id and makes replay safe	A deduplication store or idempotent side-effect implementation

Mixing these layers leads to two opposite mistakes. A team may assume enable.idempotence=true makes the whole service exactly once, or find one duplicate business event and conclude that Kafka transactions provide no value.

An Idempotent Producer Removes Retry Duplicates

Suppose a producer sends a record batch but loses the connection before receiving the acknowledgement.

The broker may already have appended the batch, or it may not have received it. The producer cannot know, so it retries. Without idempotence, both the original request and the retry could enter the log.

The idempotent producer protects this ambiguous failure window.

The Kafka 4.3 design documentation explains that the broker assigns an ID to each producer and uses producer sequence numbers to identify a resend. The record batch format contains these fields:

producerId
producerEpoch
baseSequence

The Kafka message format describes how the broker compares the first and last sequence numbers of an incoming batch with the last batch from that producer. It does not parse JSON fields or calculate a semantic hash of the value.

An internal retry of the first send() reuses the producer identity and sequence, allowing the broker to suppress the duplicate append. If the application calls send() twice, the second call receives a new sequence. Even an identical payload is a new, valid record to Kafka.

The KafkaProducer API explicitly warns that application-level resends cannot be deduplicated and limits the ordinary idempotent producer guarantee to messages sent within one producer session.

In the Kafka 4.3 producer configuration, enable.idempotence defaults to true. A producer may therefore receive retry protection without setting it explicitly. That default does not automatically enable Kafka transactions or EOS that also coordinates consumer offsets.

Keys and Headers Are Not Deduplication Switches

Would putting event_id in the record key or a header let Kafka recognize the duplicate?

The broker does not work that way.

A message key primarily selects a partition. Sending the same key to the same topic normally keeps records in one partition and gives them key-scoped order. On a compacted topic, the key also identifies older values that may eventually be removed while retaining the latest state.

Headers carry metadata such as trace context, schema versions, correlation IDs, and event IDs. Kafka transports these bytes but does not interpret them as application semantics.

Both calls below are therefore appended:

send(key="order-123", header.event_id="evt-789", value=orderCreated);
send(key="order-123", header.event_id="evt-789", value=orderCreated);

The key, header, and value may be identical, but the calls receive different sequences.

Value	Kafka broker’s view	Useful application meaning
Message key	Partitioning and compaction key	Aggregate ordering and state identity
event_id header	Opaque metadata bytes	Deduplication key for an Inbox or idempotent handler
Business transaction_id header	Opaque metadata bytes	Correlation across requests and events
Kafka transactional.id config	Producer transaction recovery and fencing identity	Must not be treated as a business event ID

The last row is especially important. A business header such as transaction_id=payment-123 and the Kafka producer configuration named transactional.id only sound similar.

Kafka uses transactional.id so the transaction coordinator can identify a producer, recover unfinished work after a restart, and fence an older producer instance. It is not a key for matching equal business transactions.

This does not make event_id unnecessary. A stable event ID is essential precisely because Kafka will not use it automatically. It gives consumers evidence for recognizing a previously applied business event and avoiding a duplicate side effect.

Where that event_id belongs—key, header, or value—is a separate design decision. If order-scoped ordering matters, keeping order_id as the record key and placing event_id in a header or the value may be more natural. Changing the key to event_id only for deduplication can alter the intended partition placement and aggregate-scoped ordering.

Log Compaction Is Not Write-Time Deduplication

The statement “Kafka eventually keeps one record for the same key on a compacted topic” creates another common misunderstanding.

Log compaction does not reject a duplicate when it arrives. Kafka appends all records with the same key. A consumer that reads before compaction can see every one of them.

Background compaction may later remove older values. Its purpose is to preserve the latest state for each key over the long term.

Retain the latest state for a key.
!=
Process the same event only once.

This is also why state topics and event topics need different semantics. Compaction can fit a topic representing the current membership tier of an account. For a payment authorization event, however, two records with the same key are not necessarily duplicates; each occurrence may be material.

Transactions Bind the Visibility of Kafka Writes

If an idempotent producer prevents retry duplicates, a Kafka transaction binds multiple writes into one commit or abort decision.

beginTransaction
-> produce to output topic A
-> produce to output topic B
-> commitTransaction

All writes become committed together. When the transaction aborts, its records may still exist physically in the log, but a consumer configured with isolation.level=read_committed does not receive them. The default read_uncommitted mode can return records from aborted transactions, so a transactional producer alone does not complete end-to-end visibility semantics.

read_committed is not content-based deduplication either. It hides records from aborted transactions; it does not compare keys, headers, or payloads across two committed records. It also returns non-transactional records normally.

A transaction still does not compare record values. If an application produces the same payload twice inside one transaction, both records are valid members of that transaction. The transaction coordinates the outcome of the included Kafka writes, not the uniqueness of their content.

EOS Requires the Consumer Offset in the Same Transaction

Consider a consume-transform-produce pipeline.

input topic
-> consume and transform a record
-> produce a result to an output topic
-> commit the input offset

If the output write and offset commit are separate, a failure window remains between them.

output produce succeeds
process crashes before offset commit
the input record is fetched again
the output is produced again

In a typical consume-transform-produce flow, the consumer commits its input offsets separately with commitSync() or commitAsync() after processing. For Kafka-to-Kafka EOS, sendOffsetsToTransaction replaces that separate offset commit. It adds the next input offset to the current producer transaction, making the Kafka output write and consumer offset commit one success-or-failure unit.

producer.beginTransaction();

// Process the consumed record and write Kafka output.
producer.send(outputRecord);

// Instead of a separate consumer offset commit, add the next input offset to this transaction.
producer.sendOffsetsToTransaction(nextOffsets, consumer.groupMetadata());

producer.commitTransaction();

When commitTransaction() succeeds, the Kafka output and the consumer group’s next input offset are committed together. On abort, neither the output nor the offset commit is completed, so the input can be processed again.

The KafkaProducer sendOffsetsToTransaction documentation says to use enable.auto.commit=false and not mix this flow with a separate commitSync() or commitAsync(). The committed position must also be the next record to process, not the offset of the last processed record.

The consumer itself is not transactional in this model. A transactional producer writes output records and updates the consumer group’s position on the consumer’s behalf. This distinction explains why an ordinary offset commit cannot create EOS by itself.

EOS does not limit the consumer processing code to one execution either. An aborted transaction or process restart can cause the same input to be processed again. What becomes visible once is the Kafka result committed by the transaction, not a single invocation of the processing function.

Commit Order Cannot Make an External Side Effect Atomic

The boundary changes when a consumer writes to a service database or calls a payment API instead of producing only to Kafka.

Committing the offset first allows this failure:

offset commit succeeds
process crashes before DB update
restart begins at the next offset
the DB update is lost

Committing the offset after processing allows the opposite failure:

DB update succeeds
process crashes before offset commit
the same record is processed again
the DB update or external API call is duplicated

Neither ordering makes the Kafka offset and external result one atomic state. A standard Kafka transaction does not automatically include an ordinary database transaction or HTTP API call.

The destination system has to cooperate:

• Store the result and processing position in one database transaction, then resume from that database position after a restart.
• Use an Inbox with a unique constraint on event_id + consumer identity.
• Make the handler idempotent when it receives the same event again.
• Pass a stable idempotency key when an external API supports one.
• Define query, retry, and compensation behavior when a remote result is unknown.

This boundary connects directly to the application contracts in Why Kafka Fits Between Outbox and Inbox. Kafka EOS remains useful; it simply does not erase the need for a contract beyond the Kafka transaction boundary.

What exactly_once_v2 Binds in Kafka Streams

Kafka Streams packages the consume-transform-produce transaction pattern into the framework.

With processing.guarantee=exactly_once_v2, it coordinates source offsets, changelog writes for state-store updates, and output-topic records as one processing transaction. If a failure aborts the transaction, downstream read_committed consumers do not observe partial results, and the input is processed again from the last committed position.

The Kafka Streams configuration documentation still lists at_least_once as the default. Applications must select exactly_once_v2, and production clusters need transaction-state and user topics configured with sufficient replication and min.insync.replicas.

Streams EOS does not identify duplicate business content either. If an upstream system publishes the same event_id twice as two valid Kafka records, Streams can process both input records exactly once.

Reflect two distinct input records exactly once each.
!=
Recognize that both records describe one business event.

Classify the Duplicate Before Choosing a Mechanism

The requirement “prevent duplicates” needs to name the failure that creates them.

Duplicate or partial result	Appropriate defense
Producer retries the same batch after losing an acknowledgement	Idempotent producer
Writes to multiple Kafka topics or partitions become partially visible	Kafka transaction plus read_committed
Crash causes a consume-transform-produce output to be written again	Include offset and output in one Kafka transaction
Application publishes the same business event twice	Stable event_id and a producer-side publication contract
Consumer commits its DB result but fails to commit the offset	Inbox or atomic storage of result and processing position
Payment or notification succeeds but its response is lost	External idempotency key and outcome lookup
Only the latest state for a key must be retained long-term	Log compaction

No single feature resolves every row. EOS is powerful, but its broad name has to be read through a narrow and explicit system boundary.

A Pre-Production Checklist

Before declaring a pipeline exactly once, answer these questions:

• Are you using only an idempotent producer, a Kafka transaction, or Kafka Streams EOS?
• Is each transactional.id stable and unique for its instance or task ownership model?
• Is auto commit disabled, with offsets passed through sendOffsetsToTransaction?
• Is the committed position the next record to process?
• Do downstream consumers read transactional output with read_committed?
• Do you observe transaction timeouts, abort rates, producer fencing, and commit failures?
• Are replication and ISR settings sufficient for transaction-state and user topics?
• Does event_id remain stable across a business retry, or does every retry create a new UUID?
• For how long and at what scope does a consumer remember processed event IDs?
• How are database and external API side effects made idempotent outside Kafka?

Broker health alone is not enough to operate EOS. Teams need to correlate producer transaction aborts and errors, consumer group offsets and lag, application duplicate counts, Inbox conflicts, and external idempotency outcomes.

This is also relevant to Konduo, which connects Kafka cluster resources through plugins so operators can inspect broker, topic, and consumer-group state alongside metric evidence, diagnostics, and alert response. It does not replace application contracts, but correlating transaction failures and consumer progress with surrounding infrastructure signals helps locate the actual boundary of a failure.

Conclusion

Kafka EOS is not a promise that equal content exists only once in the world.

An idempotent producer prevents a producer retry from appending the same batch twice. A Kafka transaction commits or aborts multiple Kafka writes together. Exactly-once processing places Kafka input offsets and Kafka outputs in one transaction so the read-process-write result is visible once.

Application responsibilities remain beyond that boundary.

Kafka knows producer identity and sequence.
The application knows business event identity.
The external system knows the actual side-effect outcome.

This is why putting event_id in a key or header still matters. Not because Kafka will deduplicate it, but because consumers and operators need evidence for the business duplicate that Kafka cannot identify.

The first question in an exactly-once design is not the configuration name.

Which duplicate are we preventing, and within which system boundary?

Further Reading

• Why Kafka Fits Between Outbox and Inbox - Explains application contracts for database handoff and consumer duplicates outside Kafka.
• Kafka Streams and the EIP Aggregator - Shows why exactly-once processing and business completion are separate concerns.
• Consumer Lag Is Not a Health Score - Treats offsets and lag as consumer progress signals rather than proof of completed business processing.