Catalog My learning Blog
Apache Kafka Tutorial: Event Streaming to Production · Phase 1 · Lesson 5 · 4 min read

Kafka Producers: Batching, Partitioning and acks

On this page
  1. What happens when you call produce()
  2. Producer batching and compression: linger.ms and batch.size
  3. Producer acks: 0, 1 or all
  4. A durable Kafka producer in Python and C#
  5. Why Java and librdkafka clients pick different partitions
  6. Check yourself

What happens when you call produce()

Calling produce() doesn't send anything over the network yet. The client:

  1. Serializes the key and value to bytes.
  2. Picks a partition: from the key's hash if there is a key, otherwise spreading records across partitions.
  3. Adds the record to a batch for that partition in memory.
  4. A background thread sends batches to each partition's leader broker when they're full or have waited long enough.
  5. The broker acknowledges, and your delivery callback reports success or failure.
Records are batched per partition in the producer, sent to each partition's leader, and acknowledged according to the acks setting.
Records are batched per partition in the producer, sent to each partition's leader, and acknowledged according to the acks setting.

Producer batching and compression: linger.ms and batch.size

SettingDefault (Kafka 4.x Java client)Effect
linger.ms5How long to wait for more records before sending a batch
batch.size16384 bytesMaximum batch size per partition
compression.typenoneCompress whole batches: lz4, zstd, snappy, gzip

Bigger batches mean fewer requests and better compression, at the cost of a few milliseconds of latency. For high-volume topics, linger.ms of 10 to 50 ms, larger batches and zstd or lz4 often multiply throughput and cut network and disk use substantially.

Producer acks: 0, 1 or all

`acks`The producer considers a write done when...Risk
0It's sentLost silently if the broker never got it
1The leader wrote itLost if the leader dies before followers copy it
all (default since 3.0)All in-sync replicas have itLost only if every in-sync replica is lost

acks=all is only as strong as the topic's min.insync.replicas (Day 9): with replication factor 3 and min.insync.replicas=2, a write needs at least two copies or it's rejected.

Idempotence (on by default since Kafka 3.0) lets the broker discard duplicates caused by the producer's own retries, so retries can't create duplicates or reorder records within a partition (Day 13).

A durable Kafka producer in Python and C#

Code
# producer.py: a durable, reasonably batched producer for order events
import json
from confluent_kafka import Producer

producer = Producer({
    "bootstrap.servers": "localhost:9092",
    "acks": "all",                      # wait for all in-sync replicas
    "enable.idempotence": True,         # no duplicates from internal retries (Day 13)
    "linger.ms": 20,                    # wait up to 20 ms to fill a batch
    "batch.size": 131072,               # up to 128 KB per partition batch
    "compression.type": "zstd",         # compress whole batches
})

def on_delivery(err, msg):
    if err is not None:
        print(f"FAILED {msg.key()}: {err}")   # alert or retry from your own outbox
    else:
        print(f"ok {msg.key().decode()} -> partition {msg.partition()} offset {msg.offset()}")

for order_id, status in [(991, "created"), (991, "paid"), (992, "created")]:
    producer.produce("orders", key=str(order_id),
                     value=json.dumps({"order_id": order_id, "status": status}),
                     on_delivery=on_delivery)
    producer.poll(0)                    # serve delivery callbacks

producer.flush(10)                      # block until everything is delivered or fails

The same producer in C# with Confluent.Kafka, the client most .NET services use (it wraps librdkafka, so the partitioner gotcha below applies):

C#
using Confluent.Kafka;

var config = new ProducerConfig
{
    BootstrapServers = "localhost:9092",
    Acks = Acks.All,
    EnableIdempotence = true,
    LingerMs = 20,
    CompressionType = CompressionType.Zstd,
    Partitioner = Partitioner.Murmur2Random,   // hash keys the way the Java client does
};

using var producer = new ProducerBuilder<string, string>(config).Build();

foreach (var (orderId, status) in new[] { (991, "created"), (991, "paid"), (992, "created") })
{
    producer.Produce("orders",
        new Message<string, string> { Key = orderId.ToString(), Value = $"{{\"order_id\":{orderId},\"status\":\"{status}\"}}" },
        report =>
        {
            if (report.Error.IsError) Console.WriteLine($"FAILED {report.Message.Key}: {report.Error.Reason}");
            else Console.WriteLine($"ok {report.Message.Key} -> partition {report.Partition.Value} offset {report.Offset.Value}");
        });
}

producer.Flush(TimeSpan.FromSeconds(10));

ProduceAsync is the awaitable alternative, but awaiting each message in turn sends one record per round trip and throws away batching. Use Produce with a delivery handler for throughput, or start many ProduceAsync calls and await them together.

Against a local broker this printed ok 991 -> partition 3 offset 0, then offset 1 for the second 991 event, and partition 1 for 992. Both events for order 991 share a partition, so they stay in order.

Always handle the delivery result. produce() returning doesn't mean the record is stored; a full buffer or an unreachable broker surfaces only in the callback or after delivery.timeout.ms (2 minutes by default). And call flush() before the process exits, or buffered records are lost.

Why Java and librdkafka clients pick different partitions

The Java client hashes keys with murmur2. librdkafka, the C library behind the Python, .NET, Go (confluent-kafka-go) and Node clients, defaults to a different hash. Producing the key 991 to the same 6-partition topic:

Code
Java client                             -> partition 5
librdkafka default (consistent_random)  -> partition 3
librdkafka partitioner=murmur2_random   -> partition 5

If a Java service and a Python service both produce events for the same keys, they land in different partitions, and per-key ordering silently breaks. Set "partitioner": "murmur2_random" in librdkafka clients when mixing languages.

Check yourself

The producer waits a few milliseconds before sending, so more records accumulate in each batch. Fewer, larger requests mean less per-request overhead on the network and brokers, and compression works better on bigger batches. The trade-off is a few milliseconds of added latency per record.

With acks=1 the leader acknowledges as soon as it has written the record locally. If it crashes before any follower has copied the record, a follower without it is elected leader, and the acknowledged record is gone. acks=all with a suitable min.insync.replicas closes that gap.

The clients use different default hash functions, so the same key maps to different partitions. Configure librdkafka with the murmur2_random partitioner.

lz4 or zstd usually give a good balance of speed and ratio, with zstd compressing better at slightly higher CPU cost. Compression happens per batch, so it pairs naturally with a bit of linger.ms.