0

I am doing experimenting with kafka. I already start the 

kafka-console-producer and 
kafka-console-consumer.

I send messages with kafka-producer and successfully receive at the kafka-console-consumer. Now I want to produce and consume around 5000 messages at once. I look into documentation and get to know that there are two commands. 

kafka-verifiable-producer.sh

kafka-verifiable-consumer.sh

I tried to use these commands .

 kafka-verifiable-producer.sh --broker-list localhost:9092 --max-messages 5000 --topic data-sending

kafka-verifiable-consumer.sh  --group-instance-id 1 --group-id data-world --topic data-sending --broker-list localhost:9092

The result is as follow 

"timestamp":1581268289761,"name":"producer_send_success","key":null,"value":"4996","offset":44630,"topic":"try_1","partition":0}
{"timestamp":1581268289761,"name":"producer_send_success","key":null,"value":"4997","offset":44631,"topic":"try_1","partition":0}
{"timestamp":1581268289761,"name":"producer_send_success","key":null,"value":"4998","offset":44632,"topic":"try_1","partition":0}
{"timestamp":1581268289761,"name":"producer_send_success","key":null,"value":"4999","offset":44633,"topic":"try_1","partition":0}

{"timestamp":1581268289769,"name":"shutdown_complete"}
 {"timestamp":1581268289771,"name":"tool_data","sent":5000,"acked":5000,"target_throughput":-1,"avg_throughput":5285.412262156448}

On the consumer console the result is as follow

{"timestamp":1581268089357,"name":"records_consumed","count":352,"partitions":[{"topic":"try_1","partition":0,"count":352,"minOffset":32777,"maxOffset":33128}]}
{"timestamp":1581268089359,"name":"offsets_committed","offsets":[{"topic":"try_1","partition":0,"offset":33129}],"success":true}
{"timestamp":1581268089384,"name":"records_consumed","count":500,"partitions":[{"topic":"try_1","partition":0,"count":500,"minOffset":33129,"maxOffset":33628}]}
 {"timestamp":1581268089391,"name":"offsets_committed","offsets":[{"topic":"try_1","partition":0,"offset":33629}],"success":true}
 {"timestamp":1581268089392,"name":"records_consumed","count":270,"partitions":[{"topic":"try_1","partition":0,"count":270,"minOffset":33629,"maxOffset":33898}]}
 {"timestamp":1581268089394,"name":"offsets_committed","offsets":[{"topic":"try_1","partition":0,"offset":33899}],"success":true}
 {"timestamp":1581268089415,"name":"records_consumed","count":500,"partitions":[{"topic":"try_1","partition":0,"count":500,"minOffset":33899,"maxOffset":34398}]}
 {"timestamp":1581268089416,"name":"offsets_committed","offsets":[{"topic":"try_1","partition":0,"offset":34399}],"success":true}
 {"timestamp":1581268089417,"name":"records_consumed","count":235,"partitions":[{"topic":"try_1","partition":0,"count":235,"minOffset":34399,"maxOffset":34633}]}
{"timestamp":1581268089419,"name":"offsets_committed","offsets":[{"topic":"try_1","partition":0,"offset":34634}],"success":true}

In above results, the key is null. How i can send a bulk of messages with this command ? I tried to look into one example how to use them but didn't found any. It produces integer number like values but where i can insert the messages?. Is there any way i can use this command to produce messages in bulk? Also is it possible to implement such commands in windows or it is just for linux? Any link to the examples would be greatly appreciated.

H.Ç.T
  • 3,335
  • 1
  • 18
  • 37
Rio
  • 347
  • 3
  • 6
  • 20
  • Why don't you just write a Producer in Java that will generate as many messages as you wish? – Giorgos Myrianthous Feb 09 '20 at 17:39
  • @GiorgosMyrianthous I can, but I am curious to know how i can use these commands. Kafka-verifiable-producer and consumer. Why these commands are available and in which scenarios i or other people can use them . – Rio Feb 09 '20 at 17:42
  • @GiorgosMyrianthous I am new to kafka so maybe my question is basic. But I would be really grateful if any solution is there. – Rio Feb 09 '20 at 17:43

1 Answers1

1

The script kafka-verifiable-producer.sh executes the classorg.apache.kafka.tools.VerifiableProducer. (https://github.com/apache/kafka/blob/trunk/tools/src/main/java/org/apache/kafka/tools/VerifiableProducer.java)

Its program arguments --throughput, --repeating-keys and --value-prefix may fulfil your needs.

For example, the following produces messages with prefix value, 111 and with an incremental key which rotates for every 5 messages. You can also configure the throughput of the messages with the --throughput option. Int this example, it produces an average of 5 messages per second.

./kafka-verifiable-producer.sh --broker-list localhost:9092 --max-messages 10 --repeating-keys 5 --value-prefix 100 --throughput 5 --topic test
{"timestamp":1581271492652,"name":"startup_complete"}
{"timestamp":1581271492860,"name":"producer_send_success","key":"0","value":"100.0","offset":45,"topic":"test","partition":0}
{"timestamp":1581271492862,"name":"producer_send_success","key":"1","value":"100.1","offset":46,"topic":"test","partition":0}
{"timestamp":1581271493048,"name":"producer_send_success","key":"2","value":"100.2","offset":47,"topic":"test","partition":0}
{"timestamp":1581271493254,"name":"producer_send_success","key":"3","value":"100.3","offset":48,"topic":"test","partition":0}
{"timestamp":1581271493256,"name":"producer_send_success","key":"4","value":"100.4","offset":49,"topic":"test","partition":0}
{"timestamp":1581271493457,"name":"producer_send_success","key":"0","value":"100.5","offset":50,"topic":"test","partition":0}
{"timestamp":1581271493659,"name":"producer_send_success","key":"1","value":"100.6","offset":51,"topic":"test","partition":0}
{"timestamp":1581271493860,"name":"producer_send_success","key":"2","value":"100.7","offset":52,"topic":"test","partition":0}
{"timestamp":1581271494063,"name":"producer_send_success","key":"3","value":"100.8","offset":53,"topic":"test","partition":0}
{"timestamp":1581271494268,"name":"producer_send_success","key":"4","value":"100.9","offset":54,"topic":"test","partition":0}
{"timestamp":1581271494483,"name":"shutdown_complete"}
{"timestamp":1581271494484,"name":"tool_data","sent":10,"acked":10,"target_throughput":5,"avg_throughput":5.452562704471101}

The easiest is to modify/extend the above class If you are looking for more customized message keys and values.

wpnpeiris
  • 766
  • 4
  • 14
  • Thanks for your answer. I will try this and will come back to you. If i am not mistaken, this functionality is only available in linux. right? I cannot use the same command in windows while using kafka. – Rio Feb 09 '20 at 18:39
  • @Rio, As I see Kafka distribution has a bunch of BAT files. (e.g kafka_2.12-2.4.0/bin/windows). The above class can be executed simply with your own bat file. e.g: ``` @echo off SetLocal set KAFKA_HEAP_OPTS=-Xmx512M "%~dp0kafka-run-class.bat" org.apache.kafka.tools.VerifiableProducer %* EndLocal ``` – wpnpeiris Feb 09 '20 at 19:13
  • Can you give it as an example that how I can implement these two classes as my own bat file in windows?. I would really appreciate your help in this regard . Cox I saw that these two classes are not available as a bat files in windows. – Rio Feb 09 '20 at 19:17
  • @Rio, I cannot provide a working bat file, as I don't use windows. You can take a copy one of bat file, in kafka_2.XX-2.X.X/bin/windows, and modify the class name that it executes. e.g take a copy of kafka-console-producer.bat file. and replace `kafka.tools.ConsoleProducer` with `org.apache.kafka.tools.VerifiableProducer` – wpnpeiris Feb 09 '20 at 19:41
  • Periris I got the following error when I copy one file and change it as you mention above. "kafka-run-class.bat"' is not recognized as an internal or external command, operable program or batch file" – Rio Feb 09 '20 at 19:53
  • 1
    @Rio, don't you have `kafka-run-class.bat` in `kafka_2.XX-2.X.X/bin/windows` folder? You could even directly run the `kafka-run-class.bat`. `kafka-run-class.bat org.apache.kafka.tools.VerifiableProducer --broker-list localhost:9092 --max-messages 10 --repeating-keys 5 --value-prefix 100 --throughput 5 --topic test` – wpnpeiris Feb 09 '20 at 20:04