6

I'm doing a test run with Kafka using the command line producer and consumer.

I'm running this in one Terminal window

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic tag7

and this in another 

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic tag7 --zookeeper localhost:2181

but it takes over 1 second for the data that I sent to be printed by the consumer. The data that I'm sending is what I type in to the producer, so basically 1 message every few seconds. Are there any configuration options I can change so that the Kafka broker expects very few messages per second and thus makes the messages move significantly quicker?

I'm using the default configurations of Zookeeper and Kafka, so I haven't configured much.

Thank you in advance!

kgman1234
  • 63
  • 1
  • 3

2 Answers2

8

As the answer from Ivan Georgiev didn't work for me (not only because it is about the consumer and not the producer), I opened another question here in StackOverFlow, and finally also in Apache's Jira and ppatierno (thanks) answered why it happened and how to solve it.

For changing the console producer's properties linger.ms (1000ms by default) and batch.size (16384 by default) you need to specify it in the command line with --timeout and --max-partition-memory-bytes respectively, for example, if you are in the Kafka's main folder:

./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic testConsole --timeout 100
froblesmartin
  • 1,527
  • 18
  • 25
  • 1
    Actually both answers are right. Ivan's answer is how you tune the consumer latency and this answer is how you tune the producer latency. It just happens that there is more room for improvement from the defaults on the producer side. If you do both you should get the best results for end to end latency. – Hans Jespersen Jun 12 '17 at 15:38
  • Yes I know (I am going to add it to my answer, thanks!), the thing is that it is not needed (at least in version 0.10.2.1) to do any tune in the consumer to have less than one second delay which is brought about by the producer. – froblesmartin Jun 12 '17 at 16:19
  • Also console-producer defaults to acks=1 which reduces latency but also can lead to message loss so don't get surprised if you see message loss when killing brokers and testing failover when using defaults. – Hans Jespersen Jun 12 '17 at 16:31
4

There are two configuration parameters for Kafka - one is setting the minimum amount of data received before answering consumer request and the other is setting the maximum amount of time to wait for this data to arrive before answering the request.

You can try to add following options:

--consumer-property fetch.max.wait.ms=0 --consumer-property fetch.min.bytes=0

For more info:

Ivan Georgiev
  • 1,115
  • 7
  • 11
  • not sure if it is changed. this does not work. As the other answer says, changing the producer side `--timeout 100` is what helped me. – RamPrakash Feb 18 '23 at 18:12