Kafka | Unable to publish data to broker - ClosedChannelException

Question

I am trying to run simple kafka producer consumer example on HDP but facing below exception.

[2016-03-03 18:26:38,683] WARN Fetching topic metadata with correlation id 0 for topics [Set(page_visits)] from broker [BrokerEndPoint(0,sandbox.hortonworks.com,9092)] failed (kafka.client.ClientUtils$)
java.nio.channels.ClosedChannelException
        at kafka.network.BlockingChannel.send(BlockingChannel.scala:120)
        at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:75)
        at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:74)
        at kafka.producer.SyncProducer.send(SyncProducer.scala:115)
        at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59)
        at kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)
        at kafka.producer.async.DefaultEventHandler$$anonfun$handle$1.apply$mcV$sp(DefaultEventHandler.scala:68)
        at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:89)
        at kafka.utils.Logging$class.swallowError(Logging.scala:106)
        at kafka.utils.CoreUtils$.swallowError(CoreUtils.scala:51)
        at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:68)
        at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:105)
        at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:88)
        at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:68)
        at scala.collection.immutable.Stream.foreach(Stream.scala:547)
        at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:67)
        at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:45)
[2016-03-03 18:26:38,688] ERROR fetching topic metadata for topics [Set(page_visits)] from broker [ArrayBuffer(BrokerEndPoint(0,sandbox.hortonworks.com,9092))] failed (kafka.utils.CoreUtils$)
kafka.common.KafkaException: fetching topic metadata for topics [Set(page_visits)] from broker [ArrayBuffer(BrokerEndPoint(0,sandbox.hortonworks.com,9092))] failed
        at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:73)
        at kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)
        at kafka.producer.async.DefaultEventHandler$$anonfun$handle$1.apply$mcV$sp(DefaultEventHandler.scala:68)
        at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:89)
        at kafka.utils.Logging$class.swallowError(Logging.scala:106)
        at kafka.utils.CoreUtils$.swallowError(CoreUtils.scala:51)
        at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:68)
        at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:105)
        at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:88)
        at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:68)
        at scala.collection.immutable.Stream.foreach(Stream.scala:547)
        at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:67)
        at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:45)
Caused by: java.nio.channels.ClosedChannelException
        at kafka.network.BlockingChannel.send(BlockingChannel.scala:120)
        at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:75)
        at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:74)
        at kafka.producer.SyncProducer.send(SyncProducer.scala:115)
        at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59)
        ... 12 more
[2016-03-03 18:26:38,693] WARN Fetching topic metadata with correlation id 1 for topics [Set(page_visits)] from broker [BrokerEndPoint(0,sandbox.hortonworks.com,9092)] failed (kafka.client.ClientUtils$)
java.nio.channels.ClosedChannelException

Here is command that I am using for producer.

./kafka-console-producer.sh --broker-list sandbox.hortonworks.com:9092  --topic page_visits

After doing bit of googling , I found that I need to add advertised.host.name property in server.properties file . Here is my server.properties file.

# Generated by Apache Ambari. Thu Mar  3 18:12:50 2016

advertised.host.name=sandbox.hortonworks.com
auto.create.topics.enable=true
auto.leader.rebalance.enable=true
broker.id=0
compression.type=producer
controlled.shutdown.enable=true
controlled.shutdown.max.retries=3
controlled.shutdown.retry.backoff.ms=5000
controller.message.queue.size=10
controller.socket.timeout.ms=30000
default.replication.factor=1
delete.topic.enable=false
fetch.purgatory.purge.interval.requests=10000
host.name=sandbox.hortonworks.com
kafka.ganglia.metrics.group=kafka
kafka.ganglia.metrics.host=localhost
kafka.ganglia.metrics.port=8671
kafka.ganglia.metrics.reporter.enabled=true
kafka.metrics.reporters=org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter
kafka.timeline.metrics.host=sandbox.hortonworks.com
kafka.timeline.metrics.maxRowCacheSize=10000
kafka.timeline.metrics.port=6188
kafka.timeline.metrics.reporter.enabled=true
kafka.timeline.metrics.reporter.sendInterval=5900
leader.imbalance.check.interval.seconds=300
leader.imbalance.per.broker.percentage=10
listeners=PLAINTEXT://sandbox.hortonworks.com:6667
log.cleanup.interval.mins=10
log.dirs=/kafka-logs
log.index.interval.bytes=4096
log.index.size.max.bytes=10485760
log.retention.bytes=-1
log.retention.hours=168
log.roll.hours=168
log.segment.bytes=1073741824
message.max.bytes=1000000
min.insync.replicas=1
num.io.threads=8
num.network.threads=3
num.partitions=1
num.recovery.threads.per.data.dir=1
num.replica.fetchers=1
offset.metadata.max.bytes=4096
offsets.commit.required.acks=-1
offsets.commit.timeout.ms=5000
offsets.load.buffer.size=5242880
offsets.retention.check.interval.ms=600000
offsets.retention.minutes=86400000
offsets.topic.compression.codec=0
offsets.topic.num.partitions=50
offsets.topic.replication.factor=3
offsets.topic.segment.bytes=104857600
producer.purgatory.purge.interval.requests=10000
queued.max.requests=500
replica.fetch.max.bytes=1048576
replica.fetch.min.bytes=1
replica.fetch.wait.max.ms=500
replica.high.watermark.checkpoint.interval.ms=5000
replica.lag.max.messages=4000
replica.lag.time.max.ms=10000
replica.socket.receive.buffer.bytes=65536
replica.socket.timeout.ms=30000
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
socket.send.buffer.bytes=102400
zookeeper.connect=sandbox.hortonworks.com:2181
zookeeper.connection.timeout.ms=15000
zookeeper.session.timeout.ms=30000
zookeeper.sync.time.ms=2000

After adding property i am getting same exception.

Any suggestion.

score 0 · Answer 1 · answered Mar 04 '16 at 02:17

As per the log it seems the kafka server(broker) is not running. The broker server should run first.

Producers and consumers are client programs that will interact with the broker servers and zookeeper also.

Before running the producer or consumer please check whether broker and zookeeper are running successfully or not.

Run the server ./kafka-server-start.sh ../config/server.properties check the logs for any errors, if no errors then start producing the messages to the server.

Check the zookeeper service also.

score 0 · Answer 2 · edited Apr 22 '16 at 07:19

modified the file /usr/hdp/current/kafka-broker/config/server.properties with the following 2 lines

advertised.host.name=sandbox.hortonworks.com
listeners=PLAINTEXT://sandbox.hortonworks.com:6667,PLAINTEXT://0.0.0.0:6667

run the following execution commands

 ./kafka-console-producer.sh --broker-list sandbox.hortonworks.com:6667 --topic tst2
 ./kafka-console-consumer.sh --zookeeper localhost:2181 --topic tst2 --from-beginning

with this its working fine

Viktor · Accepted Answer · 2016-04-23T23:46:59.423

I had similar problem. First I have checked listeners property for Kafka broker in the Ambari

Also possible to check with:

[root@sandbox bin]# cat /usr/hdp/current/kafka-broker/conf/server.properties  | grep listeners
listeners=PLAINTEXT://sandbox.hortonworks.com:6667

Ambari replaces localhost with hostname as you can see and the port is same - 6667.

Then I checked that broker really listens on that port:

[root@sandbox bin]# netstat -tulpn | grep 6667
tcp        0      0 10.0.2.15:6667              0.0.0.0:*                   LISTEN      11137/java

Next step was to launch producer:

./kafka-console-producer.sh --broker-list 10.0.2.15:6667 --topic test

At last I have launched consumer:

 ./kafka-console-consumer.sh --zookeeper 10.0.2.15:2181 --topic test --from-beginning

After typing few words with hitting Enter on producer side, consumer received messages.

Kafka | Unable to publish data to broker - ClosedChannelException

3 Answers3