Why Kafka Transactions Cannot Commit with Your Database Yet: KIP-939 and 2PC Participation

Imagine an order service that must save a payment state in its database and publish an event for the delivery service to process.

The database commit succeeds, but Kafka publication fails.
Kafka publication succeeds, but the database commit fails.

If neither outcome is acceptable, a natural question follows: why not put the database in the Kafka transaction?

Today, a Kafka transaction does not solve that problem directly. It can commit multiple Kafka writes and consumer-group offsets as one unit, but an ordinary database transaction or HTTP API call remains outside its boundary. That is why Outbox, Inbox, and idempotency keys still connect the local atomic boundaries of different systems in production.

Kafka has also been exploring a wider boundary. KIP-939 proposes making Kafka a participant in an externally coordinated two-phase commit (2PC). This post separates what you can run with Kafka transactions now from KIP-939, which remains at the proposal and implementation stage.

Editorial illustration of deep navy and warm cream paper forms meeting in a smooth central connection — To connect distinct systems safely, start by being clear about where each state and final decision is recorded.

The Short Answer: What a Kafka Transaction Binds

A Kafka transaction binds the following work into one commit-or-abort unit.

Work that can be included	Result
Records written to several topics or partitions	They all commit, or `read_committed` consumers see none of them
The next offsets of consumed input	They commit with the Kafka output
Kafka Streams changelog writes and output	They commit with input processing under `exactly_once_v2`

An ordinary database COMMIT, a payment or notification HTTP call, and a file write are not automatically included in the standard Kafka transaction.

Comparison showing input offsets and output records inside a current Kafka transaction while a database stays outside, and an external coordinator completing a prepared Kafka transaction under KIP-939 — The current atomic boundary is Kafka itself. KIP-939 does not erase that boundary; it proposes a way for Kafka to participate in an external 2PC.

This does not make Kafka transactions weak. They can atomically reflect a path that reads from Kafka, transforms data, and writes back to Kafka. The important part is reading the system boundary precisely enough to choose the right tool.

Common Use 1: Commit Multiple Kafka Writes Together

Suppose an order approval must be recorded in both orders-confirmed and billing-audit. If a record is visible in only one topic, services that consume them can process the same order from inconsistent state.

A producer with a transactional.id can call initTransactions() once, then place its sends between beginTransaction() and commitTransaction().

Properties properties = new Properties();
properties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
properties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
properties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
properties.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "order-confirmation-0");

try (KafkaProducer<String, String> producer = new KafkaProducer<>(properties)) {
    producer.initTransactions();

    try {
        producer.beginTransaction();
        producer.send(new ProducerRecord<>("orders-confirmed", "order-42", "confirmed"));
        producer.send(new ProducerRecord<>("billing-audit", "order-42", "approved"));
        producer.commitTransaction();
    } catch (ProducerFencedException | OutOfOrderSequenceException | AuthorizationException fatal) {
        producer.close();
        throw fatal;
    } catch (KafkaException abortRequired) {
        producer.abortTransaction();
    }
}

When commitTransaction() succeeds, both topic records commit together. If the transaction aborts, its records may remain physically in the log, but consumers using isolation.level=read_committed do not receive them. That consumer setting is therefore part of the guarantee, not an optional finishing touch.

The KafkaProducer API documentation explains that a transactional producer needs transactional.id, and a newer instance with the same ID can fence the previous one. This is a stable identifier for producer ownership, not a business order ID.

Common Use 2: Make Read, Process, and Write One Unit

In a pipeline that reads Kafka input, transforms it, and writes a different Kafka record, committing the output and input offset separately leaves a failure window.

Output publication succeeds.
The process stops before committing the offset.
The same input is fetched again.
The output can be published again.

sendOffsetsToTransaction(nextOffsets, consumer.groupMetadata()) replaces the consumer’s separate offset commit. It adds the next offset after the consumed records to the current producer transaction, so the Kafka output and consumer position commit or abort together.

This pattern needs auto commit disabled and the consumer configured not to read records from aborted transactions.

Properties consumerProperties = new Properties();
consumerProperties.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
consumerProperties.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");

consumer.subscribe(List.of("orders"));

while (running) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
    if (records.isEmpty()) {
        continue;
    }

    try {
        producer.beginTransaction();

        for (ConsumerRecord<String, String> record : records) {
            String result = transform(record.value());
            producer.send(new ProducerRecord<>("order-projections", record.key(), result));
        }

        // enable.auto.commit=false. Do not separately call commitSync() or commitAsync().
        producer.sendOffsetsToTransaction(records.nextOffsets(), consumer.groupMetadata());
        producer.commitTransaction();
    } catch (ProducerFencedException | OutOfOrderSequenceException | AuthorizationException fatal) {
        producer.close();
        throw fatal;
    } catch (KafkaException abortRequired) {
        producer.abortTransaction();
    }
}

The actor that executes the transaction here is the producer, not the consumer. The producer writes output records and commits the consumer group’s next position on its behalf. That is why this pattern needs enable.auto.commit=false and must not mix in commitSync() or commitAsync().

On abort, the same input can be processed again. What is visible once is the committed combination of Kafka output and offset, not one execution of the processing function. What Does Kafka EOS Actually Guarantee Once? covers that distinction in more detail.

Why the Database and External APIs Remain Outside

The following code looks like one unit, but it contains two independent commits.

database.commit();
producer.commitTransaction();

If the process stops between the two lines, only one can succeed. Reversing the order merely creates the mismatch in the other direction. The standard Kafka transaction coordinator does not know the state of an ordinary database or HTTP server.

That leaves these as the usual choices today:

Outbox: store the service DB change and the event-to-publish record in one database transaction.
Inbox: store a consumed event_id and the business result in one database transaction.
A stable idempotency key and outcome lookup when an external API provides them.
Business processing that remains meaningful after retries, with compensation when it is needed.

Why Kafka Fits Between Outbox and Inbox explains how to build those contracts at both ends of Kafka. Kafka transactions and Outbox are not competing alternatives; they protect different boundaries.

KIP-939: If Kafka Participates in External 2PC

KIP-939 does not aim to give Kafka XA support or turn Kafka into the coordinator of 2PC. Its goal is to let Kafka become a participant in a 2PC led by an external coordinator. The proposal keeps a Kafka transaction in a prepared state, then lets that coordinator decide whether it finally commits or aborts.

The following is pseudocode for understanding the sequence described in the KIP. prepareTransaction() and completePreparedTransaction() are not APIs you can call from the current public Kafka client, so this is not runnable code.

Begin an external coordinator transaction.
Begin a Kafka transaction.
Produce Kafka records.
Prepare the Kafka transaction.                  ← proposed by KIP-939
Persist DB changes and the prepared decision.
Commit the DB transaction.
Complete the prepared Kafka transaction.        ← decide commit or abort

Flow where an external 2PC coordinator prepares Kafka and a database, then completes the Kafka transaction from a persisted decision during normal operation or recovery — The core of 2PC is not a single commit call. It is keeping the prepared state and final decision recoverable after a failure.

Even if this becomes available, it will not make Kafka and every database automatically one transaction. The system on the other side must also support prepare, commit, and recovery. The external coordinator owns a durable decision record and the retry procedure. KIP-939 proposes Kafka’s participant contract, not a universal distributed-transaction shortcut.

As of 2026-07-21, KIP-939 is Accepted, while its implementation issue KAFKA-15370 is Open. Some subtasks are complete, but prepared-state retention and scenario handling are still in progress. Do not make it a product-design dependency or implement against imagined APIs yet.

The Operational Cost Added by 2PC

Current Kafka transactions time out so Kafka can clean up when a producer disappears. In externally coordinated 2PC, Kafka cannot independently abort only because time passed: that could conflict with the decision made by another participant.

KIP-939 therefore proposes no automatic timeout for a 2PC-participant transaction. The trade-off is that operators must handle transactions that remain prepared for too long.

When a prepared state stays too long	Operational response needed
A `read_committed` consumer can be unable to read later records in that partition	Observe time spent prepared together with consumer delay
Log compaction can be unable to advance	Diagnose retention and compaction pressure alongside prepared state
A coordinator failure can lose the final decision	Store the decision durably and define a restart procedure
A casual force-abort can create inconsistency	Use an authorized administrative procedure with an audit trail

Kafka needs to be read as an operational system, not only as code concepts such as brokers, topics, partitions, and consumer groups. Konduo connects Kafka-like clustered resources through plugins so teams can inspect status, metric evidence, diagnostics, and alert response in one operating flow. But the final 2PC decision and business recovery rule still belong to the application and its coordinator.

What Should You Choose Today?

Situation	First tool to choose
Results in several Kafka topics or partitions must appear together	Kafka transaction plus `read_committed`
Kafka input is transformed into Kafka output	Include offsets in the Kafka transaction
A service DB change and event publication must agree	Outbox and a retryable relay
Kafka consumption produces a DB result	Inbox, or atomic storage of result and processing position
A payment or notification API is called	Stable idempotency key, outcome lookup, and compensation policy
Several systems genuinely support prepare and recovery for 2PC	Check KIP-939 implementation and release status, then evaluate separately

Kafka transactions are already very useful. Their value is not a promise to bind every system at once; it is precise control over which Kafka results become visible together.

KIP-939 may expand the available choices when it is implemented and released. The most important questions will remain the same.

Who records the final decision?
After a failure, who resumes commit or abort, and on what evidence?