How to seed 10 million messages into Kafka?

Question

As of my requirement on real time analytics I have to insert 10 million records into Kafka and on top of that I have to write custom consumer logic.But to fulfill this requirement how can I seed 10 million records into Kafka within 3-5 minutes time?

Preferably we want to use Spring boot as per the project requirements. Please suggest

Possible duplicate of [Is there any simulator/tool to generate messages for streaming?](https://stackoverflow.com/questions/40131580/is-there-any-simulator-tool-to-generate-messages-for-streaming) — Bushuev, May 03 '19 at 21:39
https://www.confluent.io/blog/easy-ways-generate-test-data-kafka — OneCricketeer, May 03 '19 at 21:45

score 3 · Accepted Answer · answered May 05 '19 at 09:49

3

Several options:

kafka-connect-datagen, detailed in this article.
A command line producer like kafkacat:
```
seq 1 10000000 | kafkacat -b localhost:9092 -t testtopic -P
```
- For more realistic test data use this in conjunction with Mockaroo

answered May 05 '19 at 09:49

Robin Moffatt

30,382
3
65
92

score 0 · Answer 2 · answered May 04 '19 at 08:09

0

You can try streamsets tool, it has a data generator functionality. It generates data rapidly. Also this will help you with connector and consumer/producer, in-fact you can do some processing & filtering also here so ideally almost no coding and you are done.

answered May 04 '19 at 08:09

Mayank J

71
3

Nifi also has GenerateFlowFile – OneCricketeer May 04 '19 at 18:03

score 0 · Answer 3 · edited May 05 '19 at 15:42

0

You can create a simple producer and produce in a for loop. I did it, it works like a charm. Don't forget to NOT use the .flush() if you dont want to produce 1 msg/sec...

edited May 05 '19 at 15:42

oguz ismail

1
16
47
69

answered May 05 '19 at 15:38

Gremi64

1,534
12
19

How to seed 10 million messages into Kafka?

3 Answers3