Why Kafka Fits Between Outbox and Inbox Patterns

In an MSA, connecting every service with synchronous HTTP calls is easy to understand. A request comes in, the service calls the next service, and it waits for a response.

Real systems rarely stay that simple.

When an order is created, payment, inventory, notification, settlement, recommendation, and analytics systems may all need to react. If all of that work is forced into one synchronous request, user-facing latency grows and one downstream failure can shake the whole request.

So many systems put a message queue or event streaming platform in the middle.

Order Service
-> event
-> Kafka
-> Payment Consumer
-> Inventory Consumer
-> Notification Consumer

That introduces two important questions.

How does Order Service safely combine a DB change and event publication?
How does a consumer safely handle the same message more than once?

Outbox and Inbox patterns live between those two questions.

The Problem Is Not Async. It Is Dual-Write.

Consider an order creation API.

The service needs to save the order in its database and publish an OrderCreated event.

A simple implementation might look like this:

1. Insert the order into the orders table
2. Publish OrderCreated to Kafka

Or the reverse:

1. Publish OrderCreated to Kafka
2. Insert the order into the orders table

Both are fragile.

If the database commit succeeds but Kafka publish fails, the order exists but other services never hear about it. If Kafka publish succeeds but the database commit fails, other services may react to an order that does not exist.

This is the dual-write problem. It appears when two different systems, such as a database and a message broker, are treated as if they were one atomic system.

The Outbox pattern approaches the problem differently.

Put the business state change and the event-to-publish record
inside the same local database transaction.

The service does not publish to Kafka immediately. It writes an outbox record to its own database first.

BEGIN
  INSERT INTO orders (...)
  INSERT INTO outbox_events (
    event_id,
    aggregate_type,
    aggregate_id,
    event_type,
    payload,
    created_at
  )
COMMIT

Now the order and the intent to publish the event commit together. A separate relay process later reads the outbox table and publishes the event to Kafka.

Outbox Is a Local Contract for Not Losing Messages

With Outbox, the flow becomes:

Application transaction
-> business table update
-> outbox table insert
-> commit
-> relay reads outbox
-> relay publishes to Kafka
-> relay marks outbox row as sent

The benefit is clear.

If the business change commits, the event that must be published is also stored. If Kafka is temporarily slow or unavailable, the outbox row remains. The relay can publish it later.

But Outbox does not magically make everything exactly-once.

For example, the relay may publish successfully to Kafka and then crash before marking the outbox row as sent. On the next run, the same outbox row may be published again.

So an Outbox relay is usually at-least-once by nature.

If you choose not to lose messages, you must tolerate duplicates.

That is why events need a stable event_id. If the same business event is published more than once, consumers need a way to identify it.

Inbox Is the Consumer-Side Duplicate Contract

If Outbox is the producer-side safety mechanism, Inbox is the consumer-side one.

A Kafka consumer can see the same record again. It may finish business processing and crash before committing the offset. Rebalancing can cause a record to be delivered again. The Outbox relay may also publish a duplicate.

So consumers should not assume that a message arrives only once.

The Inbox pattern records processed message IDs in the consumer’s own database. The inbox record and the business update are committed in the same local transaction.

BEGIN
  INSERT INTO inbox_messages (
    event_id,
    consumer_name,
    received_at
  )
  -- unique(event_id, consumer_name)

  UPDATE payment_requests ...
COMMIT

If the event_id has already been processed, the unique constraint catches it and the consumer can skip it.

One subtle point matters: Inbox should not be global for all consumers. The same Kafka topic may be consumed by multiple consumer groups for different purposes. Payment, notification, and analytics services may all read the same OrderCreated event and do different work.

So the inbox key is usually:

event_id + consumer identity

The consumer identity may be a consumer group, subscriber name, or handler name. It needs to represent “this consumer processed this event” in a stable way.

Why Kafka Fits Well

Any messaging system can sit between Outbox and Inbox. So why does Kafka fit especially well?

Not because Kafka is simply a fast MQ.

Kafka is a partitioned append-only log. That characteristic fits Outbox and Inbox patterns well.

First, Kafka keeps events and allows replay.

In many queue-like systems, a message disappears after consumption. Kafka records are not removed just because one consumer read them. They remain according to the topic retention policy, and a consumer group can reset offsets and read them again when needed.

That replayability is valuable when an Outbox relay, consumer, or downstream process has a problem.

Second, consumer groups naturally separate subscribers.

The same OrderCreated topic can be read by payment, inventory, notification, and analytics services, each using its own consumer group. Kafka tracks offsets independently for each group. One group’s delay or failure does not directly block another group’s progress.

That fits the common shape where one outbox event flows into several bounded contexts.

Third, Kafka can preserve key-based ordering.

If records with the same key go to the same partition, Kafka preserves their order within that partition. If order events use order_id as the key, consumers can observe events for the same order in a consistent order.

This is not global ordering across the whole topic. Kafka ordering is partition-scoped. But most business event flows care more about aggregate-level ordering than global ordering.

Fourth, Kafka exposes operational signals.

Kafka gives operators consumer lag, produce and fetch latency, broker throughput, topic and partition status, and consumer group state. You can see whether the Outbox relay is falling behind, whether a specific consumer group is stuck, or whether a topic is overloaded.

Asynchronous architecture becomes uncomfortable when the flow is invisible. Kafka makes the message flow operationally observable.

Fifth, Kafka features such as producer idempotence and transactions can help.

Producer idempotence reduces the chance of duplicate records caused by retries at the broker write path. Kafka transactions help when multiple partition writes and offset commits need to be grouped inside Kafka.

But those features do not replace Outbox and Inbox. A Kafka transaction does not automatically include a normal service database transaction. The dual-write boundary between service DB and Kafka publish is still handled by Outbox. The consumer-side DB update and duplicate handling still need Inbox or an idempotent handler.

Kafka features complement the patterns. They do not erase them.

The Outbox Relay Is a Real Component

Outbox relay is sometimes treated too lightly. It sounds like “read rows from a table and publish them to Kafka.”

But the relay is an operational component.

It needs answers to questions like these:

• In what order should outbox rows be read?
• How many rows should be published in one batch?
• What happens if Kafka publish succeeds but the database status update fails?
• How do multiple relay instances avoid claiming the same row?
• Should failed events retry forever or move to a separate state?
• When is the outbox table cleaned up?
• How are schema changes handled?

If the outbox table becomes the real queue, database pressure builds up. Outbox is a durable handoff record, not an infinite message broker. After successful publication to Kafka, it needs a cleanup policy.

Deleting too aggressively is also risky. You may lose evidence for tracing and replay decisions. In production, sent-state retention, error-state retention, and archival policy should be explicit.

Consumers Should Think Beyond Offsets

A common consumer-side mistake is treating offset commit as the only sign of processing success.

Kafka offset tells a consumer group how far it has read. It does not prove that the business database was updated, that an external API side effect was idempotent, or that duplicate delivery is safe.

Inbox reduces that gap.

The consumer reads a message, commits the inbox record and business state change in its local database transaction, and then commits the Kafka offset.

Failure points still remain.

DB commit succeeds
offset commit fails
consumer restarts
same record is processed again
inbox unique constraint skips it

That flow is normal. Inbox exists to tolerate it.

External API calls need extra care. If the consumer calls a payment API or notification API, it should usually send an idempotency key to that external system too. Inbox protects the local database. It does not automatically undo side effects that already left the service boundary.

Contracts to Decide Before Code

When combining Outbox, Inbox, and Kafka, define the contracts before writing the code.

How is event_id created?
How are event type and schema version managed?
What is the Kafka key?
What defines consumer identity?
Where do retry and DLQ happen?
What is the outbox cleanup policy?
How long are inbox records retained?
Where do trace context and correlation id live?

event_id is especially important. If a new UUID is generated every time the relay republishes the same business event, consumers may not be able to detect duplicates. Depending on the system, a deterministic ID based on aggregate ID, event sequence, and event type may be better.

Kafka key is also important. For order events, order_id is often natural. For account events, account_id may be better. If the key is wrong, events for the same aggregate may scatter across partitions and break ordering assumptions.

Trace context is worth planning too. If the service stores correlation ID or trace context in the outbox row and the relay forwards it into Kafka headers, it becomes easier to trace the path from producer service to relay to consumer service.

That connects directly with the perspective in Trace Should Not Break When Kafka Is in the Middle.

Kafka Is Not Always the Right Choice

Kafka fitting Outbox and Inbox well does not mean every asynchronous call needs Kafka.

Other choices may be simpler when:

• a short-lived work queue has only one consumer
• operational simplicity matters more than replay or fan-out
• there is no need for multiple subscribers
• response-oriented request/reply latency matters more than retention and ordering

Kafka is powerful, but it is still a system to operate. Topic design, partitioning, retention, schema, consumer lag, broker capacity, and incident response come with it. Kafka is a good choice when those costs buy something important.

Outbox, Inbox, and Kafka usually fit well when:

one business change needs to flow into several services.
events must not be lost.
each consumer can progress and fail independently.
events need to be readable again for a period of time.
aggregate-level ordering matters.
operators need to see where the flow is delayed or failing.

When those conditions are true, Kafka becomes a stronger foundation than a simple queue.

Closing From an Operations View

Outbox and Inbox are not problems Kafka solves automatically. They are application-level contracts that make Kafka safer to use in asynchronous business flows.

Outbox ties the producer service’s database transaction to the intent to publish an event. Inbox lets a consumer service tolerate duplicates and reprocessing. Kafka sits between them, retaining events, delivering them independently to multiple consumer groups, and providing key-based ordering and replayability.

A tool like Konduo is useful in this kind of asynchronous architecture because Outbox relay delay, Kafka topic throughput, consumer group lag, and broker resource pressure often move together. Even a good pattern becomes guesswork if operators cannot see where the flow is slowing down.

The goal of the pattern is not to declare magical exactly-once behavior.

Do not lose messages.
Tolerate duplicates.
Allow replay.
Make delay visible.

Kafka fits well between Outbox and Inbox because it helps achieve those goals in a practical way.

Further Reading

For related operational and architecture topics, these posts connect well with this one:

• Trace Should Not Break When Kafka Is in the Middle - Covers trace context continuity across Kafka producer and consumer boundaries.
• Consumer Lag Is Not a Health Score: Thinking in Kafka Consuming Pressure - Explains how to interpret consumer delay as processing pressure, not just lag.
• Not Every Consumer Group Consumes: How Kafka Nodes Find Each Other - Looks at consumer groups and coordination from a broader Kafka perspective.