Kafka producer poor performance

Question

I have an issue with kafka producer. Actually I am using spring kafka, and sending messages through KafkaTemplate leke this:

DefaultKafkaProducerFactory<K, V> defaultKafkaProducerFactory = new DefaultKafkaProducerFactory<>(producerParams);
KafkaTemplate kafkaTemplate = new KafkaTemplate<>(defaultKafkaProducerFactory);
RecordMetadata recordMetadata = kafkaTemplate.send(record).get().getRecordMetadata();

Problem is that sometimes send message takes 4-20 sec. There a lot of messages that takes 100 ms to send. So I have few questions:

Is there any correlation between message size and throughput, what is the relation?
What should I check firstly, maybe I have't done tuning well, any direction?

Maybe your sender has batch.size set and and a high linger.ms? — daniu, Oct 13 '18 at 07:50
I see that sometimes CPU on machine with application reaches 70-85%, kafka on the same machine. Maybe it is because of this. How can I check that? — liotur, Oct 13 '18 at 07:57

score 2 · Accepted Answer · answered Oct 14 '18 at 09:38

Ok, actually problem was in machines, cpu was to high, there were logs in Cloudera manager

Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 4332ms: no GCs detected.

Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 10827ms: GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=11107ms

When I run the same on machine with 8 cores - problem is gone. one more thing that was recommended is increasing Java Heap Size of Broker

score 0 · Answer 2 · answered Apr 03 '19 at 17:56

If you have a huge amount of Data then Kafka producer requires a Performance Tunning. In this case, its better to configure batch.size, linger.ms & buffer.memory. Normally, in the kafka the batch.size default configuration is 16K bytes. To improve the Performance, just increase the batch.size.

props.put(ProducerConfig.BATCH_SIZE_CONFIG, 16_384 * 4);
    // Send with little bit buffering
    props.put(ProducerConfig.LINGER_MS_CONFIG, 200);    
  //Use Snappy compression for batch compression.
    props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "snappy");

With the above configurations the performance must be good. More details you will find in the below link.

Kafka Performance

Kafka producer poor performance

2 Answers2