Where Kafka's Disk Is Going: Object Storage and the New Streaming Layer

When Kafka grows in the cloud, storage stops being an implementation detail. It becomes one of the main drivers of cost and operability.

Traditional Kafka brokers own local logs for partition replicas, and replication provides durability. That model is powerful and familiar. But cloud block storage, cross-AZ replication, broker replacement, partition reassignment, and long retention can combine into a very expensive operating shape.

That is why the Kafka ecosystem keeps returning to one question:

Can we keep the Kafka protocol and ecosystem,
but move durable storage away from broker-local disks?

WarpStream, AutoMQ, Apache Fluss, and Apache Kafka’s KIP-1150 Diskless Topics all live near that question. They are not the same answer.

Calm watercolor-like landscape of many streams flowing into a broad lake — The stream can stay familiar while the place where data settles changes.

The short version

If you only keep a few points, keep these:

Using object storage does not mean every system has the same architecture.
Kafka Tiered Storage mostly moves older segments to a remote tier.
WarpStream and AutoMQ are closer to Kafka-compatible systems that redesign the storage layer around object storage.
Apache Fluss is closer to lakehouse-native streaming storage for Flink and analytics than to a simple Kafka replacement. Its Kafka protocol compatibility is documented as still in development.
KIP-1150 Diskless Topics is an important signal that the Apache Kafka community accepts the diskless direction, but it is not itself a finished production feature.

The core question is not simply “can Kafka store data in S3?” It is “how much durable state should a streaming broker own directly?”

Why this is happening now

This shift is not only about S3 being cheaper. Several pressures have grown at the same time.

The first pressure is cloud cost. Kafka stores data for long periods and replicates it across zones. At scale, block storage and network charges compound. Cross-AZ replication traffic is especially easy to ignore at first and very hard to ignore later.

The second pressure is elasticity. Teams are used to Kubernetes and cloud autoscaling, so waiting for partition reassignment before scaling brokers down feels increasingly awkward. Compute wants to move quickly, but broker-local data keeps it heavy.

The third pressure is the network effect of the Kafka protocol. Kafka clients, Connect, Streams, Schema Registry, operational tooling, and developer habits are hard to replace. That is why many new systems do not start with “throw Kafka away.” They start with “keep the Kafka protocol, but change the storage layer.”

The fourth pressure is the maturity of object storage. Services like S3 have become the default substrate for lakehouse, backup, analytics, and ML pipelines. Durability and cost are compelling, and throughput has improved. The hard work is adapting latency, ordering, metadata, caching, and failure recovery to Kafka’s append/fetch semantics.

The final pressure comes from lakehouse and AI workloads. Real-time events live in Kafka, while historical analysis and training data live in lakehouse systems such as Iceberg or Paimon. If those worlds are connected only through connectors and copy pipelines, cost, freshness, and governance suffer. Systems like Fluss are trying to reduce that gap between streams and tables.

So the movement is not happening because Kafka suddenly became obsolete. It is happening because the Kafka protocol became too successful to abandon, while the broker-local storage model became harder to fit into cloud economics and lakehouse-era workloads.

Diagram comparing Classic Kafka, Tiered Storage, Shared Storage, and Streamhouse storage models — The same phrase, object storage, can describe very different designs: active logs, remote history, shared durable layers, or lakehouse hot and cold tiers.

First, the facts

Recent events show that this is no longer a small side experiment. It matters to vendors and to upstream Kafka.

Confluent announced its acquisition of WarpStream on September 9, 2024. The official announcement described WarpStream as an Apache Kafka-compatible data streaming platform, with a BYOC architecture designed for large-scale workloads with relaxed latency requirements, such as logging, observability, and feeding data lakes.

IBM completed its acquisition of Confluent on March 17, 2026. As of June 2026, it is no longer just an announced intent to acquire; the acquisition is complete.

In Apache Kafka itself, KIP-1150: Diskless Topics is marked Accepted. This KIP sets the direction and requirements for Diskless Topics. It is an umbrella-style KIP; detailed public interfaces and implementation work are left to follow-up KIPs.

Apache Fluss is described on its official site as lakehouse-native streaming storage. Its stated goal is to collapse a message broker, online KV store, stream-processing state backend, and lakehouse cold store into one foundation. Fluss documentation includes Kafka protocol compatibility, but the configuration page says it is still in development and disabled by default.

Why local disk becomes painful

Kafka’s default storage model is broker-local log storage.

producer
  -> leader broker local log
  -> follower broker local logs
  -> consumer fetch

This made deep sense in the data center and commodity hardware world. A broker’s disk stores the partition replica, and ISR replication provides durability and availability.

The model still works in the cloud. The pressure comes from cost and elasticity.

Replication traffic can become expensive when it crosses availability zones.
Cloud block storage cost compounds with Kafka replication cost.
Scaling down or replacing brokers often requires data movement.
Long retention can force compute and storage to scale together.
Backfills and replay can consume serving broker bandwidth.

Object storage is attractive because services like S3 offer durability, capacity elasticity, and pay-as-you-go economics. But replacing a Kafka log with object files is not enough.

Kafka still has to preserve append order, fetch behavior, offsets, transactions, consumer groups, compaction, retention, and failure recovery. Object storage is durable, but its write latency and IOPS profile are different from local disk. The hard question is not whether data can be placed in S3. The hard question is how to preserve Kafka semantics while hiding the weak spots of object storage.

Four patterns to separate

Kafka and object storage meet in several different ways.

Pattern	Where object storage sits	What it means
export sink	downstream of Kafka	Copy or load Kafka records into a lake.
tiered storage	remote tier for older segments	The active write path still centers on broker-local logs.
shared or diskless storage	durable stream layer	Broker-local disk becomes much less important as a source of truth.
streaming lakehouse	combined stream and table storage	Analytics, tables, state, and lakehouse integration shape the design.

This distinction matters. “Kafka data also exists in S3” is not the same as “Kafka’s active durable storage is built around S3.”

Tiered Storage is a natural extension of classic Kafka. It moves older segments to a remote tier, reducing retention cost and broker disk pressure. But the active segment and ISR replication model remain mostly intact.

Systems like WarpStream and AutoMQ are more aggressive. They expose Kafka-compatible APIs while moving away from broker-local disks as the durable source of truth.

Fluss is different again. It targets streaming storage for Flink and the lakehouse, with Kafka compatibility as an ecosystem bridge rather than the whole product identity.

WarpStream: Kafka-compatible, object-storage-first BYOC

WarpStream became known as a brokerless or diskless Kafka-compatible platform. Operationally, the interesting idea is to keep compute agents close to stateless while using object storage in the customer’s cloud as the durable layer.

Confluent’s acquisition announcement is a strong market signal for this direction. Confluent already had fully managed Confluent Cloud and self-managed Confluent Platform. WarpStream added a BYOC option. The announcement specifically points to high-scale workloads with relaxed latency requirements, including logging, observability, and data lake feeding.

That phrase, relaxed latency, is important. Diskless and shared-storage architectures can be compelling for cost and elasticity, but they are not automatically better for every low-latency workload. A transactional event flow and a high-throughput observability pipeline should not be judged by the same storage criteria.

AutoMQ: replacing Kafka’s storage layer with S3Stream

AutoMQ describes itself as a Kafka-compatible shared storage architecture. Its official architecture documentation says it offloads the Kafka storage layer to cloud storage, makes brokers stateless, and uses both WAL storage and object storage.

The simplified shape is:

Kafka protocol-facing broker
  -> WAL / cache layer
  -> S3-compatible object storage

Object storage alone is not a complete answer for Kafka’s write path. Its latency and IOPS profile are not the same as local disk. AutoMQ addresses that by adding a WAL storage layer for write efficiency and storing data near real time into object storage.

The attraction is that scale-out, scale-in, and partition reassignment become less about moving large amounts of broker-local data and more about metadata and shared storage coordination. The trade-off is that you now need confidence in a new storage engine, WAL durability, metadata coordination, cache behavior, object layout, and recovery model.

The easiest place to overstate the comparison is Fluss.

If the question is “Does Apache Fluss support the Kafka protocol and use an AutoMQ-like storage model?” my answer is: there are related ideas, but grouping them too tightly creates confusion.

Fluss is positioned as lakehouse-native streaming storage, not primarily as an Apache Kafka replacement. The official site describes a design with Coordinator Servers, Tablet Servers, a hot tier, a Tiering Service, and lakehouse cold tiers such as Paimon, Iceberg, and Lance.

Also, Fluss documentation marks Kafka protocol compatibility as still in development, with kafka.enabled defaulting to false. So as of June 2026, it is too strong to say that Fluss can simply be used like an AutoMQ-style Kafka-compatible shared-storage cluster by pointing existing Kafka clients at it.

There are still similarities:

It rethinks broker and log systems as a storage substrate problem.
It includes remote object storage and lakehouse tiers in the design.
It tries to separate replay, backfill, and analytical reads from the serving path.
Kafka protocol compatibility appears as a migration or ecosystem bridge.

So I would frame the difference this way: AutoMQ is closer to “keep the Kafka protocol and replace the storage layer.” Fluss is closer to “build a streaming storage substrate for Flink and the lakehouse, then connect it to Kafka/Flink/lakehouse ecosystems.”

KIP-1150: upstream Kafka accepts the direction

The acceptance of KIP-1150 Diskless Topics matters because it shows that the Apache Kafka community now sees object storage as more than a place for inactive segments.

The KIP’s motivation is direct. The Kafka protocol has become a successful base for streaming applications, while the Apache Kafka implementation is built around low-durability block storage and direct replication. In hyperscaler clouds, high-reliability object storage is available and can be more cost-effective for equivalent workloads.

KIP-1150 does not propose removing existing topics. It is closer to a per-topic choice. Existing low-latency topics remain useful, and diskless topics introduce a different latency and cost profile.

Compatibility is also a central goal. The KIP says Diskless Topics should support existing Kafka APIs with the same external semantics, including ordering, idempotency, transactions, consumer groups, share groups, and tiered storage.

But Accepted is not GA. KIP-1150 itself does not propose new public interfaces, and the details are delegated to sub-KIPs. The practical meaning today is not “turn on Diskless Topics in Kafka 4.x.” It is “upstream Kafka has accepted diskless topics as a future direction.”

How to choose

Start with the workload, not the product name.

Workload	Model to examine first
low-latency event processing	classic Kafka with carefully tuned broker storage
long retention with occasional replay	Kafka Tiered Storage
high-volume observability, logging, or data lake feeding	shared-storage Kafka-compatible systems, WarpStream-like BYOC
Kafka compatibility plus cloud elasticity	AutoMQ-like shared storage architecture
Flink-centered real-time analytics and lakehouse integration	Apache Fluss-style streaming lakehouse storage
tracking the upstream Kafka standard path	KIP-1150 Diskless Topics and follow-up KIPs

Latency is the first question. Object-storage-centered systems can improve cost and elasticity, but hot-path latency and tail behavior need separate measurement.

Ownership is the second question. Are you ready to keep Kafka semantics while trusting a very different storage engine?

Observability is the third question. With broker-local logs, operators watch disk usage, under-replicated partitions, ISR, flush behavior, and page cache. With shared storage, they also need WAL pressure, object upload latency, cache hit ratio, remote fetch latency, metadata coordinator health, and object store error budgets.

Moving Kafka storage toward object storage is also a change in how teams read operational signals. This operational angle is central to Konduo. Konduo connects Kafka and other infrastructure resources through plugins so teams can read brokers, topics, consumer groups, metric evidence, and alert response in one operating flow.

Checklist

Before adopting a new storage model, ask these questions:

Is this workload sensitive to p99 latency, or are throughput and cost more important?
How much cross-AZ traffic do producers and consumers create?
Are you over-sizing broker disks mainly for retention?
Do replay and backfill workloads interfere with serving brokers?
Is partition reassignment time an operational bottleneck?
Do you only need Kafka protocol compatibility, or do you require Apache Kafka implementation behavior?
How will you observe object store outages, throttling, consistency behavior, and request cost?
If a WAL or cache layer is introduced, do you understand its durability and failure modes?
Have you verified compacted topics, transactions, exactly-once, ACLs, quotas, Connect, and Streams behavior?
Will you wait for upstream Kafka’s KIP-1150 path, or test Kafka-compatible alternatives first?

My rule of thumb is:

Object storage is one credible future for Kafka storage.
But "it uses S3" is not enough to understand the architecture.

WarpStream, AutoMQ, Fluss, and KIP-1150 all point toward a broader rethinking of streaming storage. Some change the storage layer behind a Kafka-compatible broker. Some build a lakehouse-native streaming substrate. Some define the upstream Kafka direction.

That distinction helps you ask the real questions:

Do we want Kafka to cost less?
Do we want it to scale in and out more easily?
Do we want streams and lakehouse tables to be closer together?
Or do we want to stay inside the upstream Kafka standard path?

The point is not that disks disappear. The point is that durable stream ownership moves somewhere else, and that move changes cost, latency, recovery, and observability.