0

I'm following a tutorial here from MapR in making a basic Kafka producer/consumer:

https://github.com/mapr-demos/kafka-sample-programs

I'm using the Hortonworks Sandbox VM, version 2.4, with Kafka 0.9.2 Here is my Producer code:

package com.whatever.kafka.basic;

...

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.ProducerConfig;

public class BasicProducer {

    public static void main(String[] args) throws IOException {

        /* comment-out snippet from MapR example; hard-code properties instead
        KafkaProducer<String, String> producer;
        try (InputStream props = Resources.getResource("producer.props").openStream()) {
            Properties properties = new Properties();
            properties.load(props);
            producer = new KafkaProducer<String, String>(properties);
        }
        */

        Properties props = new Properties();
        props.put("bootstrap.servers", "sandbox.hortonworks.com:6667");
        props.put("acks", "all");
        props.put("retries", 0);
        props.put("batch.size", 16384);
        props.put("auto.commit.interval.ms", 1000);
        props.put("linger.ms", 0);
        props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("block.on.buffer.full", true);

        final KafkaProducer<String, String> producer = new KafkaProducer<String, String>(props);

        try {
            for (int i = 0; i < 1000000; i++) {

                System.out.println("Test 1"); // prints to console and stops on the next line

                producer.send(new ProducerRecord<String, String>(
                    "fast-messages", String.format("{\"type\":\"marker\", \"t\":%.3f, \"k\":%d}", System.nanoTime() * 1e-9, i)  
                ));

                System.out.println("Test 2");

                if (i % 250 == 0) {
                    producer.send(new ProducerRecord<String, String>(
                            "fast-messages", String.format("{\"type\":\"marker\", \"t\":%.3f, \"k\":%d}", System.nanoTime() * 1e-9, i)
                    ));
                    producer.send(new ProducerRecord<String, String>(
                            "summary-markers", String.format("{\"type\":\"other\", \"t\":%.3f, \"k\":%d}", System.nanoTime() * 1e-9, i)
                    )); 
                    producer.flush();
                    System.out.println("Sent msg number " + i);
                }
            }
            producer.close();
        } catch (Throwable throwable) {
            System.out.printf("%s", throwable.getStackTrace());
        } finally {
            producer.close();
        }
    } 
}

I have started Zookeeper and Kafka and created topics as indicated in the instruction:

$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic fast-messages
$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic summary-markers

but when I run the compiled application

$ ./kafka-example producer

I get this output:

log4j:WARN No appenders could be found for logger (org.apache.kafka.clients.producer.ProducerConfig)
log4j:WARN Please initialize the log4j system properly
Test 1

It gets stuck on the first producer.send line, after the Test 1 sysout log. No exception gets thrown either, it just stops at printing out that Test 1 line. When I use Eclipse to debug it using line-breaks, it stops on that line as well, apparently with nothing happening.

The fast-messages and summary-markers directories in Kafka logs get created, but nothing written.

Any idea?

oikonomiyaki
  • 7,691
  • 15
  • 62
  • 101
  • I assume that you are running the code from within the Hortonworks sandbox and "sandbox.hortonworks.com" gets resolved correctly? Have you tried producing to the topics with the console producer to ensure that everything is set up correctly? Your code looks good to me.. – Sönke Liebau Mar 08 '17 at 08:03
  • The address of the broker that I should use is the `listener` field in the `server.properties`, right? – oikonomiyaki Mar 08 '17 at 08:26
  • When I try `bin/kafka-console-producer.sh --broker-list :6667 --topic fast-messages` and type some message, I get this never ending stream of: `WARN Error while fetching metadata with correlation id 128 : {fast-messages=TOPIC_AUTHORIZATION_FAILED (org.apache.kafka.clients.NetworkClient)` – oikonomiyaki Mar 08 '17 at 08:30
  • Apparently you have Kafka configured to perform authorization against requests, check your config for this line please: _authorizer.class.name=kafka.security.auth.SimpleAclAuthorize‌​r_ I have recently answered this for someone else [here](http://stackoverflow.com/questions/42635682/kafka-on-cloudera-test-topic-authorization-failed) perhaps that answer helps? – Sönke Liebau Mar 08 '17 at 08:34

0 Answers0