How to measure number of send() request sent by Kafkaproducer in a batch of 2 mins

Question

I am playing around the Kafka producer performance. And i need to reduce the number of requests that Kafka producer sends it to Kafka cluster by tuning batch size . I am using batch processing of 2 mins

I tried printing metrics mentioned in https://kafka.apache.org/documentation/. But there is no specific metric to count number of send requests to Kafka cluster by Kafka producer.

Expected Results :

I need to measure the number of times Kafka producer performs call or talk to Kafka cluster to send the records . If Kafka is spending more time in taking to cluster , then i can tune in linger.ms and batch-size

Answer 1

You should be able to use some of the producer metrics mentioned in the docs to get that measurement.

You can use request-total. If you get the value before your 2 minute window start and after you know how many requests the producer sent to the cluster. A request can contain several records if the producer is batching.

There's also record-send-total for the actual record count and outgoing-byte-total for the byte count.

All of these metrics also exist as rate (average per second).

That should be more than enough to achieve what you want.