On KafkaProducer.send(message), I am getting "exception Error serializing Avro message"

Question

I am using Avro to generate class. Here is my code in producer looks like

TweetInfo tweetInfo = TweetInfo.newBuilder()
                    .setTweetId(status.getId())
                    .setTweetCreatedAt(status.getCreatedAt().toString())
                    .setTweetMessage(status.getText())
                    .setUserId(user.getId())
                    .setUserCreatedAt(user.getCreatedAt().toString())
                    .setUserName(user.getName())
                    .setUserScreenName(user.getScreenName())
                    .build();

            ProducerRecord<String, TweetInfo> data = new ProducerRecord(KafkaConstants.TOPIC, tweetInfo);
            producer.send(data);

TweetInfo is class generated by Avro schema. When I run the program I see a stacktrace as following

    2018-12-11 01:51:58.138  WARN 16244 --- [c Dispatcher[0]] o.i.service.kafka.TweetKafkaProducer     : exception Error serializing Avro message
2018-12-11 01:51:59.162 ERROR 16244 --- [c Dispatcher[0]] i.c.k.s.client.rest.RestService          : Failed to send HTTP request to endpoint: http://localhost:8081/subjects/twitterData-value/versions

java.net.ConnectException: Connection refused: connect
    at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method) ~[na:1.8.0_152]
    at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85) ~[na:1.8.0_152]
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_152]
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_152]
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_152]
    at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172) ~[na:1.8.0_152]
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_152]
    at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_152]
    at sun.net.NetworkClient.doConnect(NetworkClient.java:175) ~[na:1.8.0_152]
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:463) ~[na:1.8.0_152]
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:558) ~[na:1.8.0_152]
    at sun.net.www.http.HttpClient.<init>(HttpClient.java:242) ~[na:1.8.0_152]
    at sun.net.www.http.HttpClient.New(HttpClient.java:339) ~[na:1.8.0_152]
    at sun.net.www.http.HttpClient.New(HttpClient.java:357) ~[na:1.8.0_152]
    at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220) ~[na:1.8.0_152]
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156) ~[na:1.8.0_152]
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050) ~[na:1.8.0_152]
    at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984) ~[na:1.8.0_152]
    at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1334) ~[na:1.8.0_152]
    at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1309) ~[na:1.8.0_152]
    at io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:178) [kafka-schema-registry-client-5.0.1.jar:na]
    at io.confluent.kafka.schemaregistry.client.rest.RestService.httpRequest(RestService.java:235) [kafka-schema-registry-client-5.0.1.jar:na]
    at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:326) [kafka-schema-registry-client-5.0.1.jar:na]
    at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:318) [kafka-schema-registry-client-5.0.1.jar:na]
    at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:313) [kafka-schema-registry-client-5.0.1.jar:na]
    at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.registerAndGetId(CachedSchemaRegistryClient.java:114) [kafka-schema-registry-client-5.0.1.jar:na]
    at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.register(CachedSchemaRegistryClient.java:153) [kafka-schema-registry-client-5.0.1.jar:na]
    at io.confluent.kafka.serializers.AbstractKafkaAvroSerializer.serializeImpl(AbstractKafkaAvroSerializer.java:79) [kafka-avro-serializer-5.0.1.jar:na]
    at io.confluent.kafka.serializers.KafkaAvroSerializer.serialize(KafkaAvroSerializer.java:53) [kafka-avro-serializer-5.0.1.jar:na]
    at org.apache.kafka.common.serialization.Serializer.serialize(Serializer.java:60) [kafka-clients-2.1.0.jar:na]
    at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:879) [kafka-clients-2.1.0.jar:na]
    at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:841) [kafka-clients-2.1.0.jar:na]
    at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:728) [kafka-clients-2.1.0.jar:na]
    at org.interview.service.kafka.TweetKafkaProducer$1.onStatus(TweetKafkaProducer.java:95) [classes/:na]
    at twitter4j.StatusStreamImpl.onStatus(StatusStreamImpl.java:75) [twitter4j-stream-4.0.6.jar:4.0.6]
    at twitter4j.StatusStreamBase$1.run(StatusStreamBase.java:105) [twitter4j-stream-4.0.6.jar:4.0.6]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_152]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_152]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_152]

I have zookeeper and kafka running. Do I also need to run Schema Registry? If yes then is there a guide to do that? I can't find any.

`Properties props = new Properties(); props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, KafkaConstants.BOOTSTRAP_SERVERS); props.put(ProducerConfig.BATCH_SIZE_CONFIG, 1000); props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName()); props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, KafkaAvroSerializer.class.getName()); props.put("schema.registry.url", "http://localhost:8081"); props.put("producer.type", "sync"); props.put(ProducerConfig.CLIENT_ID_CONFIG, KafkaConstants.CLIENT_ID);` — BountyHunter, Dec 11 '18 at 01:58
The error suggests that your schema registry is not running. Have you checked that bit? — Sid Malani, Dec 11 '18 at 02:33
check this article out - starting the registry bit.... https://aseigneurin.github.io/2018/08/02/kafka-tutorial-4-avro-and-schema-registry.html — Sid Malani, Dec 11 '18 at 02:35
@SidMalani The article didn't really help. I am not able to start the registry. Should I add anything in env variables? I just downloaded the confluent project. When I try and run the commands in article it gives error that bin is not valid command — BountyHunter, Dec 11 '18 at 03:01
How are you running kafka on local? Are you using docker? Also can you post the log trail including command you tried to execute and the error messages? — Sid Malani, Dec 11 '18 at 03:25

OneCricketeer · Answer 1 · 2018-12-12T17:23:23.260

0

Failed to send HTTP request to endpoint

The Confluent Schema Registry Server needs to be running. And you might want to try hitting the HTTP endpoints yourself (refer docs below).

Not sure how you started it, but you can download Confluent OSS, extract it somewhere, then in the terminal, you will want to navigate to the extracted folder's bin location and run confluent start schema-registry. Note: This only works for Linux.

Or if you want a "production deployment" configuration, will need to edit the property files in the etc folder there first and run each of Zookeeper, Kafka, and the Registry using their respective scripts.

Docs: Running Schema Registry

Regarding comments

When I try and run the commands in article it gives error that bin is not valid command

$ bin/... first assumes that you had cd'd into the confluent-x.x.x folder that was extracted

By the way there are existing Kafka Connect projects that interact with the Twitter API.

edited Dec 12 '18 at 17:23

answered Dec 11 '18 at 03:25

OneCricketeer

179,855
19
132
245

Thanks for the answer. I am trying to run the command `confluent start schema-registry` but it shows 'confluent' is not recognized as an internal or external command'. Do I need to make any changes in env variables? – BountyHunter Dec 11 '18 at 09:54
You can edit your PATH, or you need to `cd` into the `bin` folder, as mentioned. Note: Confluent command is not available in Windows – OneCricketeer Dec 11 '18 at 18:37
I am already in bin folder. I am using windows. I updated PATH too and still no luck with running schema-registry. I am still seeing the same message `confluent' is not recognized as an internal or external command` – BountyHunter Dec 12 '18 at 14:57
Again, `confluent` Command is not available on Windows. You'll need to switch to the `bin\windows` folder (and add that to your PATH instead) and run the `schema-registry-start.bat` file there – OneCricketeer Dec 12 '18 at 15:53
Actually, seems like `schema-registry` commands aren't made for Windows either, so you'll have to run in Docker or a VM, as mentioned on Confluent Slack – OneCricketeer Dec 12 '18 at 17:22

Saïd Bouras · Accepted Answer · 2018-12-13T14:41:58.783

0

Like @cricket_007 said if you are on windows try to use docker.

Below the link to a docker compose that will run kafka, zookeeper, schema registry and kafka rest, with that you can test your producer easily. https://github.com/confluentinc/docker-images/blob/master/examples/fullstack/docker-compose.yml

EDIT: Sorry my bad, it's the link of the old repo, check the one below, you have all confluent platform (you can remove what you don't need) !

https://github.com/confluentinc/cp-docker-images/blob/5.0.1-post/examples/cp-all-in-one/docker-compose.yml

edited Dec 13 '18 at 14:41

answered Dec 12 '18 at 22:12

Saïd Bouras

286
1
4

Thanks for the reply. I tried using the docker-compose.yml but not able to connect. I posted the stacktrace here https://github.com/wurstmeister/kafka-docker/issues/440 – BountyHunter Dec 13 '18 at 14:12
Sorry I gave you the link to the old repo, check the answer, I juste add an edit. – Saïd Bouras Dec 13 '18 at 14:42

On KafkaProducer.send(message), I am getting "exception Error serializing Avro message"

2 Answers2