I'm following a tutorial here from MapR in making a basic Kafka producer/consumer:
https://github.com/mapr-demos/kafka-sample-programs
I'm using the Hortonworks Sandbox VM, version 2.4, with Kafka 0.9.2 Here is my Producer code:
package com.whatever.kafka.basic;
...
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.ProducerConfig;
public class BasicProducer {
public static void main(String[] args) throws IOException {
/* comment-out snippet from MapR example; hard-code properties instead
KafkaProducer<String, String> producer;
try (InputStream props = Resources.getResource("producer.props").openStream()) {
Properties properties = new Properties();
properties.load(props);
producer = new KafkaProducer<String, String>(properties);
}
*/
Properties props = new Properties();
props.put("bootstrap.servers", "sandbox.hortonworks.com:6667");
props.put("acks", "all");
props.put("retries", 0);
props.put("batch.size", 16384);
props.put("auto.commit.interval.ms", 1000);
props.put("linger.ms", 0);
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("block.on.buffer.full", true);
final KafkaProducer<String, String> producer = new KafkaProducer<String, String>(props);
try {
for (int i = 0; i < 1000000; i++) {
System.out.println("Test 1"); // prints to console and stops on the next line
producer.send(new ProducerRecord<String, String>(
"fast-messages", String.format("{\"type\":\"marker\", \"t\":%.3f, \"k\":%d}", System.nanoTime() * 1e-9, i)
));
System.out.println("Test 2");
if (i % 250 == 0) {
producer.send(new ProducerRecord<String, String>(
"fast-messages", String.format("{\"type\":\"marker\", \"t\":%.3f, \"k\":%d}", System.nanoTime() * 1e-9, i)
));
producer.send(new ProducerRecord<String, String>(
"summary-markers", String.format("{\"type\":\"other\", \"t\":%.3f, \"k\":%d}", System.nanoTime() * 1e-9, i)
));
producer.flush();
System.out.println("Sent msg number " + i);
}
}
producer.close();
} catch (Throwable throwable) {
System.out.printf("%s", throwable.getStackTrace());
} finally {
producer.close();
}
}
}
I have started Zookeeper and Kafka and created topics as indicated in the instruction:
$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic fast-messages
$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic summary-markers
but when I run the compiled application
$ ./kafka-example producer
I get this output:
log4j:WARN No appenders could be found for logger (org.apache.kafka.clients.producer.ProducerConfig)
log4j:WARN Please initialize the log4j system properly
Test 1
It gets stuck on the first producer.send
line, after the Test 1
sysout log. No exception gets thrown either, it just stops at printing out that Test 1
line. When I use Eclipse to debug it using line-breaks, it stops on that line as well, apparently with nothing happening.
The fast-messages
and summary-markers
directories in Kafka logs get created, but nothing written.
Any idea?