The Large Payload Kafka Handles Well, and the One It Should Not Carry

Kafka handles large-scale traffic well. That is why this question comes up:

If Kafka is built for large-scale data, why not send large files through Kafka?

You can make it possible. You can raise producer max.request.size, topic max.message.bytes, consumer fetch.max.bytes, and max.partition.fetch.bytes so that Kafka accepts record batches larger than the defaults. The Apache Kafka documentation also treats producer request size, topic max message size, and consumer fetch size as separate limits.

But this is not mainly a configuration question. The more important difference is the nature of the payload.

The large payload Kafka handles well is usually a large volume of events flowing continuously. Sending a large file as a Kafka message is closer to copying data that is already a file through another file-like path. Both involve bytes, but they create different storage cost models across the system.

The conclusion is simple:

Do not send the file through Kafka.
Send the event that the file exists.
Store the file body in object storage and publish the reference and metadata to Kafka.

Illustration of small event signals flowing while a large file remains in separate storage — Kafka is a better fit for the small event that a file exists than for the file body itself.

Kafka’s Large Scale Is Event-Stream Scale

Kafka was designed for real-time data feeds, high-throughput event streams, and partitioned processing. Producers write events, brokers append them to partition logs, and multiple consumer groups read the same log independently.

A common real-time event flow looks like this:

A producer creates an event.
The producer publishes the event to Kafka.
The producer does not keep that same payload locally for long.
A consumer reads the event and processes it.
The consumer keeps only the result or state it needs.

In this model, Kafka storage pressure is relatively easy to reason about.

Kafka storage ~= incoming bytes/sec
              x retention window
              x replication factor
              x compression/index/segment overhead

Real operations add more variables: compaction, tiered storage, partition distribution, and broker hot spots. Still, the basic shape is clear. Kafka stores the payload in a log for a configured retention window, then removes it.

The important part is that producer and consumer are not usually preserving the same payload as a file source. Events are processing units: state changes, commands, measurements, log fragments, or facts that something happened. A consumer may write a result into another system, but it does not necessarily store the exact same payload again as a file.

File Payloads Move Differently

Large files move differently.

When a producer puts a file into Kafka, it is usually doing one of these things:

reading a file that already exists in object storage, NAS, local disk, or a database
converting a large binary result directly into a Kafka message
receiving an upload request and passing the file body to Kafka so that a later consumer can store it

In that flow, the producer is less like “create an event and discard it” and more like “read a file body and copy it through another path.”

The consumer behaves similarly. If a consumer receives a payload that is tens or hundreds of megabytes, it usually writes that payload somewhere: object storage, a filesystem, another database, or a transformed storage location. The consumer is not merely processing and discarding an event. It is treating the payload as a file again.

At that point, Kafka becomes an intermediate storage layer in a file transfer path.

The Broker Is Not the Only Problem

Large file payloads do put pressure on Kafka brokers.

Producer requests become larger.
Broker network I/O increases.
Partition log segments grow faster.
Replication traffic increases.
Page cache and disk write paths carry larger bursts.
Consumer fetch requests and processing latency may become unstable.

But this view keeps the problem inside the Kafka cluster. The more interesting part is outside Kafka.

Multiple consumer groups do not make Kafka store the same topic data multiple times. Kafka lets different consumer groups read the same log independently. That is one of its strengths.

But when the payload is a file, the story changes. Inside Kafka, the log is shared. Outside Kafka, each consumer group may store the file payload again for its own purpose.

Imagine a topic named uploaded-video.

consumer group	work	downstream storage
`thumbnail-generator`	creates thumbnails	thumbnail bucket
`transcoder`	creates renditions	video rendition bucket
`moderation-scanner`	stores review evidence	moderation evidence store
`archive-writer`	keeps long-term copy	archive bucket

Inside Kafka, these groups are just reading the same message. But if the message payload is the original file, each group may write that payload again in its own downstream storage. The original payload is amplified once by broker replication, then again by downstream storage copies created by consumer groups.

A simplified model looks like this:

Total file payload pressure
  = producer-side original or temporary file
  + Kafka log x replication factor x retention
  + downstream copies per consumer group
  + retry, reprocessing, failure storage, temporary files

The point is not that Kafka storage grows with the number of consumer groups. Kafka shares the log. The point is that a file payload continues to live as a file after it leaves Kafka. As consumer groups grow, system-wide storage amplification can grow with them.

Raising Limits Is Not the Solution

Kafka has settings that control large messages.

producer max.request.size
topic max.message.bytes
broker default message.max.bytes
consumer fetch.max.bytes
consumer max.partition.fetch.bytes

If these are not aligned, the producer may fail to send, the broker may reject the batch, or the consumer may fetch inefficiently. So the question “Which settings do I need to increase to send large files?” is natural.

But these settings are not a switch that turns large file transfer into a good design. They only change the limits that allow large record batches to pass.

Once the limits are raised, the next questions arrive immediately.

Is producer memory and buffering still healthy?
Can broker network threads and request queues absorb the larger requests?
Can replication keep up?
Does the consumer poll loop remain stable while processing large payloads?
How much data is resent when retry happens?
Will the DLQ or failure topic also store the same large file payload?
Are byte rate, request size, disk write, and consumer lag observed together?

Allowing larger messages is only the start. The operating cost model remains.

Better Pattern: Files in Storage, References in Kafka

In most cases, the better design is to separate the file body from the event.

The producer or upload service stores the file in object storage.
After the upload succeeds, it publishes file metadata and a reference to Kafka.
Consumers read the Kafka event and download the file only when needed.
Each consumer stores only the result it owns.

The Kafka message can look like this:

{
  "eventType": "FileUploaded",
  "fileId": "file_01JZ9M0E4A8R6Z7E3N2Q",
  "bucket": "raw-upload",
  "objectKey": "tenant-a/2026/06/19/file_01JZ9M0E4A8R6Z7E3N2Q.bin",
  "sizeBytes": 73400320,
  "contentType": "video/mp4",
  "sha256": "8d8f4c7e...",
  "createdAt": "2026-06-19T08:20:00Z"
}

This is not a workaround because Kafka is weak. It is a division of responsibility.

data	better place	why
file body	object storage	large binary data, lifecycle, versioning, range reads, access control
fact that a file exists	Kafka	fast fan-out, replay, independent consumer groups
processing result	consumer-owned storage	thumbnails, scan results, indexes, transformed files
operating state	metrics/logs/traces	throughput, failure rate, latency, storage cost

With this pattern, Kafka messages stay small and easy to replay. Multiple consumer groups no longer need to drag the same large payload through Kafka. Consumers that need the file can fetch it. Consumers that only need metadata can skip the download.

The Reference Pattern Still Has Edges

Publishing object storage references is not free. A few design questions remain.

Did the upload succeed before the Kafka event was published?
If Kafka publish succeeds but the producer crashes afterward, can orphan objects be cleaned up?
Can consumers see the object when they receive the event?
Should the event carry a presigned URL, or should consumers use bucket/key with identity-based access?
Is URL or token expiration longer than the retry window?
Do object lifecycle rules and Kafka retention agree with each other?
Is processing idempotent by fileId?
Does the consumer verify the downloaded object with a checksum?

These questions are annoying, but they are usually easier than pushing the file body through Kafka. The file lifecycle belongs to the storage layer. Kafka owns the order and distribution of events around that file.

When Is It Acceptable?

Not every binary payload needs to be banned. Small compressed documents, small thumbnails, protocol buffer payloads, and small encrypted blobs can be perfectly reasonable Kafka messages.

The decision is not only about size.

Will several consumer groups store the same payload again as a file?
Will retry and DLQ paths copy the same payload again?
Will retention make broker storage grow sharply?
Does the consumer need large memory buffers to process a single message?
Do some consumers only need metadata?
Does file lifecycle differ from Kafka retention?

If several answers are yes, send a reference rather than the file.

What to Watch Operationally

This is an architecture decision and an observability problem. When large payloads enter Kafka, message count may look ordinary while byte rate, request size, broker disk writes, replication traffic, and consumer fetch latency look very different.

Watch these signals together.

area	signal
producer	request size, record size, compression ratio, retry rate
broker	bytes in/out, request latency, disk write, replication lag, rejected bytes
topic	retention bytes, segment roll, max message size override
consumer	fetch size, processing time, lag, retry, downstream write latency
storage	object put/get latency, storage growth, lifecycle cleanup, failed object count

This operational angle is also central to Konduo. Konduo connects operating targets such as Kafka through plugins so teams can read resource status, metric evidence, and alert response in one operating flow. That connected view matters when the real issue spans Kafka, consumers, and storage rather than a single broker metric.

Conclusion

Kafka handling large-scale traffic does not mean every large payload belongs inside Kafka.

Kafka is good at receiving many events into ordered logs and distributing them to multiple consumers. A large file is different. It already has its own storage lifecycle. If you put it into a Kafka message, broker replication, retention, consumer storage, retry, and failure handling can overlap into system-wide storage amplification.

A useful rule is:

Do not send the file through Kafka.
Send the event that the file exists.

Let object storage own the file body. Let Kafka own the event and metadata around it. That boundary is simpler, cheaper, and easier to operate.