Why Rednote Moved from Kafka to Fluss: Lessons from a Real-Time Indexing Migration

Rednote (Xiaohongshu) is moving a core real-time indexing path from Kafka to Fluss. This is not a decision to replace Kafka everywhere. It targets the cost created when search, recommendation, and advertising services share a wide table, transfer full rows, and repeatedly rebuild indexes from historical data.

The case is valuable because it is not a generic broker-speed comparison. Once data becomes a wide table, many consumers read different subsets, and historical indexes are rebuilt frequently, a delivery path starts behaving like repeatedly queried storage. Rednote addressed that mismatch by changing the data path rather than adding more Kafka capacity.

Rednote at a Glance

Area	What Rednote reported
Workload	A core real-time indexing path where search, recommendation, and advertising services read different columns
Previous problems	Read amplification from full-row transfer, slow historical replay, and contention between online reads and index rebuilds
Migration	Write Arrow data to Fluss, let online services read required columns, and replay historical data directly from object storage
Lake role	Write the same real-time data to Paimon for dual-run validation, exploration, and historical use
Published results	About 30% less CPU, 50% less write traffic, 50–80% faster index builds, 30–90% less online bandwidth, and roughly 3× peak throughput
Validation	Kafka-baseline load tests, an extended dual run, and phased traffic migration

These figures apply to Rednote’s wide real-time indexing workload. They are not performance guarantees for every Kafka deployment. Rednote’s Kafka-to-Fluss migration case study

Before: Kafka Carried Three Different Read Responsibilities

Rednote’s upstream data platform wrote wide tables to Kafka for several online services. Real-time services consumed fresh data from Kafka, while batch and incremental index-building jobs replayed data retained on local disks and object storage. One Kafka path carried online distribution, historical replay, and index reconstruction.

Problem 1: Consumers Needed a Few Columns but Read Full Rows

Search, recommendation, and advertising services cared about different columns. In Rednote’s Kafka-based wide-table path, however, a consumer transferred and processed the complete row even when it needed only a small subset. Every additional consumer repeated more irrelevant network and compute work.

Rednote reported fan-out of roughly 1,000× in some localized workloads. At that scale, another full-row read is not a small consumer inefficiency; it becomes structural read amplification across network and compute resources.

Problem 2: Historical Data Took a Detour Through the Broker

Index reconstruction needs more than the newest events. Builders must scan large amounts of history from object storage. In the previous design, clients could not take a direct path to those files; the data returned through a Kafka broker before reaching the builder.

That detour slowed historical reads and made brokers intended for real-time delivery carry large historical scans as well.

Problem 3: Historical Rebuilds Competed with Online Reads

Batch and incremental builds scan historical data with high concurrency. They consumed broker network bandwidth and polluted the page cache, directly competing with latency-sensitive online consumers. The central problem was not Kafka’s absolute performance. It was putting two workloads with different access patterns on the same read path.

After: Fluss Split the Path Around the Access Pattern

Both figures are reproduced from Apache Fluss’s Rednote migration case study, which is distributed under the Apache License 2.0. Open the before figure or after figure at full size if the labels are too small.

Change 1: Replace Full-Row Transfer with Projected Reads

The upstream data platform writes to Fluss through Apache Arrow. Online services read the columns they need instead of transferring every column in a wide row. The savings compound as tables become wider and more consumers read different subsets.

This does not mean Kafka inherently supports only row-formatted data. Rednote’s wide-table encoding and consumption path repeatedly moved full rows, while Fluss’s columnar streaming storage matched that particular access pattern more closely.

Change 2: Historical Replay No Longer Passes Through the Broker

Index builders can read Fluss log files directly from object storage. Large historical scans bypass the online broker path, so they no longer compete for the same broker network and cache resources. Online reads and parallel historical rebuilds now use paths aligned with their different workloads.

Change 3: Paimon Became a Validation and Historical Path

Rednote uses Fluss Lakestream to write the same real-time data to Apache Paimon. During migration, the Paimon copy supported dual-run comparison and data exploration. It also established a path for historical replay, offline analytics, and batch index construction.

This was more than adding another store. Rednote separated real-time and historical access while retaining a shared data foundation that could compare and validate both paths.

The Published Gains Came from the Whole Path Change

Rednote reported improvements across writes, online reads, and index construction.

Area	Published result	Directly related change
Write path	About 30% less CPU and 50% less write traffic	Arrow writes and a layout suited to the indexing workload
Index construction	About 50–80% faster batch and incremental builds	Parallel historical reads and removal of the broker detour
Online reads	About 30–90% less bandwidth and roughly 3× peak throughput	Projecting required columns instead of repeatedly transferring wide rows

These numbers should not be reduced to “Fluss is always several times faster than Kafka.” Rednote changed the data layout, online read path, historical access path, and lake integration together. The published results belong to that complete change in this specific indexing workload.

Migration Conditions That Generalize from Rednote

The closer a Kafka result path is to the following conditions, the stronger the case for evaluating Fluss.

Condition to check	How it appeared at Rednote
Is the result a wide table?	Search, recommendation, and advertising shared a wide indexing table
Do consumers need different columns?	Each service used a subset, but the previous path transferred full rows
Is fan-out high?	Many online services reread the same data, reaching roughly 1,000× in some localized workloads
Are large historical scans frequent?	Batch and incremental index builds scanned long histories from object storage
Do historical reads compete with online traffic?	Both consumed broker bandwidth and page cache
Does the same result serve real time and the lake?	Fluss served online reads while Paimon supported validation, exploration, and historical use

A Flink or Spark job may produce a similarly wide derived result, write it to Kafka, and have many services and index builders reread it. That can resemble Rednote’s access pattern. It does not mean Rednote published that exact topology; it is an application of the conditions demonstrated by the case.

Where Kafka Should Remain

Rednote’s case is not evidence for replacing Kafka everywhere. Kafka remains the natural choice when:

• Contracts with external services are organized around topics, partitions, and consumer groups.
• Kafka Connect or existing Kafka clients are central to ingestion and delivery.
• Consumer-specific retries, DLQs, and event ordering are primary operational semantics.
• The result is a command or notification for independent services, not a table that will be queried repeatedly.
• No online read path is required and analytics alone can write directly to a data lake without adding Fluss.

The useful question is not “Can Kafka be removed?” It is: Is this Kafka path still delivering messages, or has it become storage for distributing a wide table, repeated reads, and historical replay?

The Migration Discipline Rednote Demonstrated

Rednote did not move a critical path on the strength of one benchmark.

1. It ran load tests against important Kafka baselines.
2. It operated the old and Fluss paths together for an extended period to compare data correctness.
3. It shifted selected core workloads progressively after validation.
4. It prepared table management, multi-tenancy, monitoring and alerting, lineage, cluster operations, backup, and lake-ingestion management alongside the data path.

At publication time, one table was processing roughly one billion records and 10 TB per day. Rednote also stated a plan to expand the migration to more than 100 tables in the second half of 2026. The important point is not the scale alone: baseline tests, dual runs, and phased migration remained mandatory at that scale. Rednote’s validation approach and expansion plan

Other teams should also begin with one result path suffering from severe read amplification or replay contention. Validate duplication and loss, schema changes, recovery after failure, latency, throughput, and cost before moving consumers.

Kafka operations emphasize topics, partitions, and consumer lag. A Fluss migration adds table health, tiering and lake ingestion, and processing-job state to the operating picture. Konduo is an integrated management and operations platform designed to connect data platforms such as Kafka and Fluss through plugins, bringing status, metrics, and alert response into one flow. A phased migration needs comparable operating evidence from both the old and new paths.

Closing Thought: Rednote Replaced the Path That No Longer Fit, Not Kafka Everywhere

Rednote’s Kafka-to-Fluss migration is not a general broker benchmark. It changed a path where many consumers read subsets of a wide table, historical indexes were rebuilt frequently, and those rebuilds shared resources with online reads.

Fluss enabled column projection, moved historical replay to object storage, and connected Paimon for validation and historical use. Write cost, online bandwidth, index-build time, and peak throughput improved together as a result.

The practical lesson is clear. When a Kafka topic has grown beyond message delivery and now distributes a wide derived table, serves repeated reads, and carries historical replay, evaluate whether the access pattern itself fits Fluss better before scaling Kafka further.

Further Reading

• What Is Apache Fluss? A Streaming Lakehouse Between Kafka, Flink, and Iceberg — Compare the roles Kafka, Flink, Iceberg, and Fluss occupy in a data path.
• Your First Apache Fluss Run: What the Flink Quickstart Shows — Explore Fluss Log Tables and Primary Key Tables with Flink SQL.
• What Your Sensor Event Is Missing: Enrich It with fluss-go — A practical example of using Fluss tables for real-time enrichment.