Produce 1 million message/seconds with kafka

Question

I have a requirement to prove kafka producer can produce 1 million message/second to Kafka cluster and then evaluate its performance.

How can I achieve produce 1 million message/sec?

public static void main(String args[]){

    Random rnd = new Random();

    Properties props = new Properties();
    props.put("bootstrap.servers", "localhost:9092");
    props.put("acks", "all");
    props.put("retries", 0);
    props.put("batch.size", 16384);
    props.put("linger.ms", 1);
    props.put("buffer.memory", 33554432);
    props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
    props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
    Producer<String, String> producer = new KafkaProducer<String, String>(props);
    int counter=0;
    int i = 0;
    while ( true ){
        TimeZone tz = TimeZone.getTimeZone("UTC");
        DateFormat df = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm.sss'Z'"); // Quoted "Z" to indicate UTC, no timezone offset
        df.setTimeZone(tz);
        String nowAsISO = df.format(new Date());
        ++counter;

            final String message = "sensor" + i + ":" + Integer.toString(rnd.nextInt(10000)) + " " +  String.valueOf(rnd.nextDouble())+ " " +
                    "MyDevice" + " " +
                    "Sensor" + " " + "Sensing" + " " + "Property" + " " + "Unit" + " " + "9845A" + " " + nowAsISO ;


        try {
            producer.send(new ProducerRecord<String, String>("test", message), new Callback() {
                        public void onCompletion(RecordMetadata recordMetadata, Exception e) {
                            if ( e == null ){
                                System.out.println("Partition: "+recordMetadata.partition()
                                        +", Offset" + recordMetadata.offset()
                                        + ", timestamp: " + recordMetadata.timestamp());
                                System.out.println(message);
                            }
                            else {
                                e.printStackTrace();
                            }
                        }
                    }
            );

            i++;
            TimeUnit.SECONDS.sleep(1000);
        }
        catch ( InterruptedException e   ){
            System.out.println("I was interrupted.");
        }
    }

}
}

Thanks !!

you could publish a constant string inside a while loop a million times, flush the producer after the loop and time it. — Vikas Tikoo, Jul 21 '17 at 09:25
having a TimeUnit.SECONDS.sleep(1000) is self defeating. I'd say flush after the while loop and predicate the loop on i++ < 1mil and do time difference of the post and pre loop times. — Vikas Tikoo, Jul 21 '17 at 09:41

score 0 · Answer 1 · answered Jul 21 '17 at 10:21

There are some instructions to produce 850,000 msg/sec with just 2 brokers here on the librdkafka github page

https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md

You should be able to get over 1 million msg/sec with 3 brokers provided you have enough network bandwidth and disk I/O capacity.

Produce 1 million message/seconds with kafka

1 Answers1