Connecting to Kafka server hosted on EC2 from local machine

Question

I have been battling this issue for a couple of days now. I am able to set up a simple single-node Kafka instance on an EC2 Ubuntu instance and create topics and post/consume to them. But that is possible because everything is bound to localhost. Yet, when I try to assign advertised.listeners to the public IP address of the EC2 instance (e.g. advertised.listeners=PLAINTEXT://ec2_public_ip:9092) and set listeners to the local host (e.g. listeners=PLAINTEXT://ec2_internal_ip:9092), neither local nor remote consumers and producers can connect.

I have made sure the port is open on the instance's firewall, and I can telnet to the zookeeper port (2181) remotely without issue.

This feels like a simple configuration issue, but I've not found any examples or documentation to make this clear.

EDIT #1 Here is the relevant info from the server.properties file:

broker.id=0
listeners=PLAINTEXT://<internal_ip>:9092
advertised.listeners=PLAINTEXT://<external_ip>:9092
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000

And here's the log output when I run it.

bin/kafka-server-start.sh config/server.properties 
[2017-05-22 15:41:58,909] INFO KafkaConfig values: 
    advertised.host.name = null
    advertised.listeners = PLAINTEXT://<external_ip>:9092
    advertised.port = null
    authorizer.class.name = 
    auto.create.topics.enable = true
    auto.leader.rebalance.enable = true
    background.threads = 10
    broker.id = 0
    broker.id.generation.enable = true
    broker.rack = null
    compression.type = producer
    connections.max.idle.ms = 600000
    controlled.shutdown.enable = true
    controlled.shutdown.max.retries = 3
    controlled.shutdown.retry.backoff.ms = 5000
    controller.socket.timeout.ms = 30000
    create.topic.policy.class.name = null
    default.replication.factor = 1
    delete.topic.enable = false
    fetch.purgatory.purge.interval.requests = 1000
    group.max.session.timeout.ms = 300000
    group.min.session.timeout.ms = 6000
    host.name = 
    inter.broker.listener.name = null
    inter.broker.protocol.version = 0.10.2-IV0
    leader.imbalance.check.interval.seconds = 300
    leader.imbalance.per.broker.percentage = 10
    listener.security.protocol.map = SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,TRACE:TRACE,SASL_SSL:SASL_SSL,PLAINTEXT:PLAINTEXT
    listeners = PLAINTEXT://<internal_ip>:9092
    log.cleaner.backoff.ms = 15000
    log.cleaner.dedupe.buffer.size = 134217728
    log.cleaner.delete.retention.ms = 86400000
    log.cleaner.enable = true
    log.cleaner.io.buffer.load.factor = 0.9
    log.cleaner.io.buffer.size = 524288
    log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308
    log.cleaner.min.cleanable.ratio = 0.5
    log.cleaner.min.compaction.lag.ms = 0
    log.cleaner.threads = 1
    log.cleanup.policy = [delete]
    log.dir = /tmp/kafka-logs
    log.dirs = /tmp/kafka-logs
    log.flush.interval.messages = 9223372036854775807
    log.flush.interval.ms = null
    log.flush.offset.checkpoint.interval.ms = 60000
    log.flush.scheduler.interval.ms = 9223372036854775807
    log.index.interval.bytes = 4096
    log.index.size.max.bytes = 10485760
    log.message.format.version = 0.10.2-IV0
    log.message.timestamp.difference.max.ms = 9223372036854775807
    log.message.timestamp.type = CreateTime
    log.preallocate = false
    log.retention.bytes = -1
    log.retention.check.interval.ms = 300000
    log.retention.hours = 168
    log.retention.minutes = null
    log.retention.ms = null
    log.roll.hours = 168
    log.roll.jitter.hours = 0
    log.roll.jitter.ms = null
    log.roll.ms = null
    log.segment.bytes = 1073741824
    log.segment.delete.delay.ms = 60000
    max.connections.per.ip = 2147483647
    max.connections.per.ip.overrides = 
    message.max.bytes = 1000012
    metric.reporters = []
    metrics.num.samples = 2
    metrics.recording.level = INFO
    metrics.sample.window.ms = 30000
    min.insync.replicas = 1
    num.io.threads = 8
    num.network.threads = 3
    num.partitions = 1
    num.recovery.threads.per.data.dir = 1
    num.replica.fetchers = 1
    offset.metadata.max.bytes = 4096
    offsets.commit.required.acks = -1
    offsets.commit.timeout.ms = 5000
    offsets.load.buffer.size = 5242880
    offsets.retention.check.interval.ms = 600000
    offsets.retention.minutes = 1440
    offsets.topic.compression.codec = 0
    offsets.topic.num.partitions = 50
    offsets.topic.replication.factor = 3
    offsets.topic.segment.bytes = 104857600
    port = 9092
    principal.builder.class = class org.apache.kafka.common.security.auth.DefaultPrincipalBuilder
    producer.purgatory.purge.interval.requests = 1000
    queued.max.requests = 500
    quota.consumer.default = 9223372036854775807
    quota.producer.default = 9223372036854775807
    quota.window.num = 11
    quota.window.size.seconds = 1
    replica.fetch.backoff.ms = 1000
    replica.fetch.max.bytes = 1048576
    replica.fetch.min.bytes = 1
    replica.fetch.response.max.bytes = 10485760
    replica.fetch.wait.max.ms = 500
    replica.high.watermark.checkpoint.interval.ms = 5000
    replica.lag.time.max.ms = 10000
    replica.socket.receive.buffer.bytes = 65536
    replica.socket.timeout.ms = 30000
    replication.quota.window.num = 11
    replication.quota.window.size.seconds = 1
    request.timeout.ms = 30000
    reserved.broker.max.id = 1000
    sasl.enabled.mechanisms = [GSSAPI]
    sasl.kerberos.kinit.cmd = /usr/bin/kinit
    sasl.kerberos.min.time.before.relogin = 60000
    sasl.kerberos.principal.to.local.rules = [DEFAULT]
    sasl.kerberos.service.name = null
    sasl.kerberos.ticket.renew.jitter = 0.05
    sasl.kerberos.ticket.renew.window.factor = 0.8
    sasl.mechanism.inter.broker.protocol = GSSAPI
    security.inter.broker.protocol = PLAINTEXT
    socket.receive.buffer.bytes = 102400
    socket.request.max.bytes = 104857600
    socket.send.buffer.bytes = 102400
    ssl.cipher.suites = null
    ssl.client.auth = none
    ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
    ssl.endpoint.identification.algorithm = null
    ssl.key.password = null
    ssl.keymanager.algorithm = SunX509
    ssl.keystore.location = null
    ssl.keystore.password = null
    ssl.keystore.type = JKS
    ssl.protocol = TLS
    ssl.provider = null
    ssl.secure.random.implementation = null
    ssl.trustmanager.algorithm = PKIX
    ssl.truststore.location = null
    ssl.truststore.password = null
    ssl.truststore.type = JKS
    unclean.leader.election.enable = true
    zookeeper.connect = localhost:2181
    zookeeper.connection.timeout.ms = 6000
    zookeeper.session.timeout.ms = 6000
    zookeeper.set.acl = false
    zookeeper.sync.time.ms = 2000
 (kafka.server.KafkaConfig)
[2017-05-22 15:41:58,951] INFO starting (kafka.server.KafkaServer)
[2017-05-22 15:41:58,952] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
[2017-05-22 15:41:58,965] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2017-05-22 15:41:58,971] INFO Client environment:zookeeper.version=3.4.9-1757313, built on 08/23/2016 06:50 GMT (org.apache.zookeeper.ZooKeeper)
[2017-05-22 15:41:58,971] INFO Client environment:host.name=<internal_ip>.ec2.internal (org.apache.zookeeper.ZooKeeper)
[2017-05-22 15:41:58,971] INFO Client environment:java.version=1.8.0_121 (org.apache.zookeeper.ZooKeeper)
[2017-05-22 15:41:58,971] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)
[2017-05-22 15:41:58,971] INFO Client environment:java.home=/usr/lib/jvm/java-8-oracle/jre (org.apache.zookeeper.ZooKeeper)
[2017-05-22 15:41:58,971] INFO Client environment:java.class.path=:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/aopalliance-repackaged-2.5.0-b05.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/argparse4j-0.7.0.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/connect-api-0.10.2.0.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/connect-file-0.10.2.0.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/connect-json-0.10.2.0.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/connect-runtime-0.10.2.0.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/connect-transforms-0.10.2.0.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/guava-18.0.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/hk2-api-2.5.0-b05.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/hk2-locator-2.5.0-b05.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/hk2-utils-2.5.0-b05.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/jackson-annotations-2.8.0.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/jackson-annotations-2.8.5.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/jackson-core-2.8.5.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/jackson-databind-2.8.5.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/jackson-jaxrs-base-2.8.5.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/jackson-jaxrs-json-provider-2.8.5.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/jackson-module-jaxb-annotations-2.8.5.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/javassist-3.20.0-GA.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/javax.annotation-api-1.2.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/javax.inject-1.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/javax.inject-2.5.0-b05.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/javax.servlet-api-3.1.0.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/javax.ws.rs-api-2.0.1.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/jersey-client-2.24.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/jersey-common-2.24.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/jersey-container-servlet-2.24.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/jersey-container-servlet-core-2.24.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/jersey-guava-2.24.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/jersey-media-jaxb-2.24.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/jersey-server-2.24.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/jetty-continuation-9.2.15.v20160210.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/jetty-http-9.2.15.v20160210.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/jetty-io-9.2.15.v20160210.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/jetty-security-9.2.15.v20160210.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/jetty-server-9.2.15.v20160210.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/jetty-servlet-9.2.15.v20160210.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/jetty-servlets-9.2.15.v20160210.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/jetty-util-9.2.15.v20160210.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/jopt-simple-5.0.3.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/kafka_2.11-0.10.2.0.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/kafka_2.11-0.10.2.0-sources.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/kafka_2.11-0.10.2.0-test-sources.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/kafka-clients-0.10.2.0.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/kafka-log4j-appender-0.10.2.0.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/kafka-streams-0.10.2.0.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/kafka-streams-examples-0.10.2.0.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/kafka-tools-0.10.2.0.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/log4j-1.2.17.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/lz4-1.3.0.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/metrics-core-2.2.0.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/osgi-resource-locator-1.0.1.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/reflections-0.9.10.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/rocksdbjni-5.0.1.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/scala-library-2.11.8.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/scala-parser-combinators_2.11-1.0.4.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/slf4j-api-1.7.21.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/slf4j-log4j12-1.7.21.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/snappy-java-1.1.2.6.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/validation-api-1.1.0.Final.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/zkclient-0.10.jar:/home/ubuntu/kafka_2.11-0.10.2.0/bin/../libs/zookeeper-3.4.9.jar (org.apache.zookeeper.ZooKeeper)
[2017-05-22 15:41:58,971] INFO Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)
[2017-05-22 15:41:58,971] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2017-05-22 15:41:58,971] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2017-05-22 15:41:58,971] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2017-05-22 15:41:58,971] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2017-05-22 15:41:58,971] INFO Client environment:os.version=4.4.0-36-generic (org.apache.zookeeper.ZooKeeper)
[2017-05-22 15:41:58,971] INFO Client environment:user.name=ubuntu (org.apache.zookeeper.ZooKeeper)
[2017-05-22 15:41:58,971] INFO Client environment:user.home=/home/ubuntu (org.apache.zookeeper.ZooKeeper)
[2017-05-22 15:41:58,971] INFO Client environment:user.dir=/home/ubuntu/kafka_2.11-0.10.2.0 (org.apache.zookeeper.ZooKeeper)
[2017-05-22 15:41:58,972] INFO Initiating client connection, connectString=localhost:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@333291e3 (org.apache.zookeeper.ZooKeeper)
[2017-05-22 15:41:58,983] INFO Waiting for keeper state SyncConnected (org.I0Itec.zkclient.ZkClient)
[2017-05-22 15:41:58,986] INFO Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2017-05-22 15:41:58,990] INFO Socket connection established to localhost/127.0.0.1:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2017-05-22 15:41:59,010] INFO Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x15c30cd33890000, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2017-05-22 15:41:59,011] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2017-05-22 15:41:59,114] INFO Cluster ID = l-wScqxkQ9OIoKkySSuuLQ (kafka.server.KafkaServer)
[2017-05-22 15:41:59,145] INFO [ThrottledRequestReaper-Fetch], Starting  (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2017-05-22 15:41:59,146] INFO [ThrottledRequestReaper-Produce], Starting  (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2017-05-22 15:41:59,205] INFO Loading logs. (kafka.log.LogManager)
[2017-05-22 15:41:59,240] INFO Completed load of log __consumer_offsets-39 with 1 log segments and log end offset 0 in 17 ms (kafka.log.Log)
[2017-05-22 15:41:59,245] INFO Completed load of log __consumer_offsets-1 with 1 log segments and log end offset 0 in 1 ms (kafka.log.Log)
[2017-05-22 15:41:59,248] INFO Completed load of log __consumer_offsets-40 with 1 log segments and log end offset 0 in 0 ms (kafka.log.Log)
[2017-05-22 15:41:59,251] INFO Completed load of log __consumer_offsets-34 with 1 log segments and log end offset 0 in 2 ms (kafka.log.Log)
[2017-05-22 15:41:59,253] INFO Completed load of log __consumer_offsets-35 with 1 log segments and log end offset 0 in 0 ms (kafka.log.Log)
[2017-05-22 15:41:59,261] INFO Completed load of log __consumer_offsets-15 with 1 log segments and log end offset 0 in 1 ms (kafka.log.Log)
[2017-05-22 15:41:59,266] INFO Completed load of log __consumer_offsets-20 with 1 log segments and log end offset 91 in 3 ms (kafka.log.Log)
[2017-05-22 15:41:59,269] INFO Completed load of log __consumer_offsets-48 with 1 log segments and log end offset 0 in 1 ms (kafka.log.Log)
[2017-05-22 15:41:59,271] INFO Completed load of log __consumer_offsets-32 with 1 log segments and log end offset 0 in 1 ms (kafka.log.Log)
[2017-05-22 15:41:59,274] INFO Completed load of log __consumer_offsets-46 with 1 log segments and log end offset 144 in 1 ms (kafka.log.Log)
[2017-05-22 15:41:59,279] INFO Completed load of log __consumer_offsets-22 with 1 log segments and log end offset 0 in 1 ms (kafka.log.Log)
[2017-05-22 15:41:59,281] INFO Completed load of log __consumer_offsets-8 with 1 log segments and log end offset 0 in 1 ms (kafka.log.Log)
[2017-05-22 15:41:59,283] INFO Completed load of log __consumer_offsets-12 with 1 log segments and log end offset 0 in 1 ms (kafka.log.Log)
[2017-05-22 15:41:59,285] INFO Completed load of log __consumer_offsets-19 with 1 log segments and log end offset 0 in 1 ms (kafka.log.Log)
[2017-05-22 15:41:59,287] INFO Completed load of log __consumer_offsets-21 with 1 log segments and log end offset 0 in 1 ms (kafka.log.Log)
[2017-05-22 15:41:59,290] INFO Completed load of log __consumer_offsets-4 with 1 log segments and log end offset 0 in 0 ms (kafka.log.Log)
[2017-05-22 15:41:59,292] INFO Completed load of log __consumer_offsets-5 with 1 log segments and log end offset 0 in 0 ms (kafka.log.Log)
[2017-05-22 15:41:59,295] INFO Completed load of log __consumer_offsets-41 with 1 log segments and log end offset 852 in 2 ms (kafka.log.Log)
[2017-05-22 15:41:59,298] INFO Completed load of log __consumer_offsets-23 with 1 log segments and log end offset 0 in 1 ms (kafka.log.Log)
[2017-05-22 15:41:59,301] INFO Completed load of log __consumer_offsets-25 with 1 log segments and log end offset 49 in 1 ms (kafka.log.Log)
[2017-05-22 15:41:59,304] INFO Completed load of log __consumer_offsets-2 with 1 log segments and log end offset 0 in 1 ms (kafka.log.Log)
[2017-05-22 15:41:59,307] INFO Completed load of log __consumer_offsets-26 with 1 log segments and log end offset 0 in 1 ms (kafka.log.Log)
[2017-05-22 15:41:59,310] INFO Completed load of log __consumer_offsets-24 with 1 log segments and log end offset 0 in 0 ms (kafka.log.Log)
[2017-05-22 15:41:59,312] INFO Completed load of log __consumer_offsets-49 with 1 log segments and log end offset 0 in 1 ms (kafka.log.Log)
[2017-05-22 15:41:59,316] INFO Completed load of log matt-0 with 1 log segments and log end offset 13 in 2 ms (kafka.log.Log)
[2017-05-22 15:41:59,319] INFO Completed load of log __consumer_offsets-14 with 1 log segments and log end offset 0 in 1 ms (kafka.log.Log)
[2017-05-22 15:41:59,322] INFO Completed load of log __consumer_offsets-16 with 1 log segments and log end offset 6 in 2 ms (kafka.log.Log)
[2017-05-22 15:41:59,324] INFO Completed load of log __consumer_offsets-31 with 1 log segments and log end offset 0 in 1 ms (kafka.log.Log)
[2017-05-22 15:41:59,327] INFO Completed load of log __consumer_offsets-29 with 1 log segments and log end offset 0 in 1 ms (kafka.log.Log)
[2017-05-22 15:41:59,329] INFO Completed load of log __consumer_offsets-10 with 1 log segments and log end offset 0 in 1 ms (kafka.log.Log)
[2017-05-22 15:41:59,333] INFO Completed load of log __consumer_offsets-7 with 1 log segments and log end offset 19 in 2 ms (kafka.log.Log)
[2017-05-22 15:41:59,337] INFO Completed load of log __consumer_offsets-47 with 1 log segments and log end offset 288 in 3 ms (kafka.log.Log)
[2017-05-22 15:41:59,340] INFO Completed load of log __consumer_offsets-36 with 1 log segments and log end offset 0 in 1 ms (kafka.log.Log)
[2017-05-22 15:41:59,343] INFO Completed load of log __consumer_offsets-28 with 1 log segments and log end offset 0 in 0 ms (kafka.log.Log)
[2017-05-22 15:41:59,345] INFO Completed load of log __consumer_offsets-27 with 1 log segments and log end offset 0 in 1 ms (kafka.log.Log)
[2017-05-22 15:41:59,348] INFO Completed load of log __consumer_offsets-9 with 1 log segments and log end offset 268 in 1 ms (kafka.log.Log)
[2017-05-22 15:41:59,350] INFO Completed load of log __consumer_offsets-37 with 1 log segments and log end offset 0 in 1 ms (kafka.log.Log)
[2017-05-22 15:41:59,353] INFO Completed load of log __consumer_offsets-6 with 1 log segments and log end offset 3 in 1 ms (kafka.log.Log)
[2017-05-22 15:41:59,357] INFO Completed load of log __consumer_offsets-0 with 1 log segments and log end offset 129 in 3 ms (kafka.log.Log)
[2017-05-22 15:41:59,359] INFO Completed load of log __consumer_offsets-42 with 1 log segments and log end offset 0 in 1 ms (kafka.log.Log)
[2017-05-22 15:41:59,362] INFO Completed load of log __consumer_offsets-33 with 1 log segments and log end offset 0 in 0 ms (kafka.log.Log)
[2017-05-22 15:41:59,364] INFO Completed load of log __consumer_offsets-13 with 1 log segments and log end offset 0 in 1 ms (kafka.log.Log)
[2017-05-22 15:41:59,366] INFO Completed load of log __consumer_offsets-3 with 1 log segments and log end offset 0 in 1 ms (kafka.log.Log)
[2017-05-22 15:41:59,369] INFO Completed load of log __consumer_offsets-11 with 1 log segments and log end offset 0 in 1 ms (kafka.log.Log)
[2017-05-22 15:41:59,371] INFO Completed load of log __consumer_offsets-45 with 1 log segments and log end offset 0 in 1 ms (kafka.log.Log)
[2017-05-22 15:41:59,374] INFO Completed load of log __consumer_offsets-44 with 1 log segments and log end offset 38 in 2 ms (kafka.log.Log)
[2017-05-22 15:41:59,376] INFO Completed load of log __consumer_offsets-43 with 1 log segments and log end offset 0 in 1 ms (kafka.log.Log)
[2017-05-22 15:41:59,379] INFO Completed load of log __consumer_offsets-18 with 1 log segments and log end offset 24 in 2 ms (kafka.log.Log)
[2017-05-22 15:41:59,382] INFO Completed load of log __consumer_offsets-38 with 1 log segments and log end offset 11 in 1 ms (kafka.log.Log)
[2017-05-22 15:41:59,384] INFO Completed load of log __consumer_offsets-17 with 1 log segments and log end offset 0 in 1 ms (kafka.log.Log)
[2017-05-22 15:41:59,387] INFO Completed load of log test-0 with 1 log segments and log end offset 6 in 2 ms (kafka.log.Log)
[2017-05-22 15:41:59,388] INFO Completed load of log __consumer_offsets-30 with 1 log segments and log end offset 0 in 0 ms (kafka.log.Log)
[2017-05-22 15:41:59,390] INFO Completed load of log remote_matt-0 with 1 log segments and log end offset 0 in 1 ms (kafka.log.Log)
[2017-05-22 15:41:59,392] INFO Logs loading complete in 186 ms. (kafka.log.LogManager)
[2017-05-22 15:41:59,442] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)
[2017-05-22 15:41:59,443] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
[2017-05-22 15:41:59,477] FATAL [Kafka Server 0], Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.common.KafkaException: Socket server failed to bind to <internal_ip>:9092: Cannot assign requested address.
    at kafka.network.Acceptor.openServerSocket(SocketServer.scala:330)
    at kafka.network.Acceptor.<init>(SocketServer.scala:255)
    at kafka.network.SocketServer$$anonfun$startup$1.apply(SocketServer.scala:98)
    at kafka.network.SocketServer$$anonfun$startup$1.apply(SocketServer.scala:90)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
    at kafka.network.SocketServer.startup(SocketServer.scala:90)
    at kafka.server.KafkaServer.startup(KafkaServer.scala:215)
    at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:39)
    at kafka.Kafka$.main(Kafka.scala:67)
    at kafka.Kafka.main(Kafka.scala)
Caused by: java.net.BindException: Cannot assign requested address
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:433)
    at sun.nio.ch.Net.bind(Net.java:425)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67)
    at kafka.network.Acceptor.openServerSocket(SocketServer.scala:326)
    ... 10 more
[2017-05-22 15:41:59,480] INFO [Kafka Server 0], shutting down (kafka.server.KafkaServer)
[2017-05-22 15:41:59,482] INFO [Socket Server on Broker 0], Shutting down (kafka.network.SocketServer)
[2017-05-22 15:41:59,486] INFO [Socket Server on Broker 0], Shutdown completed (kafka.network.SocketServer)
[2017-05-22 15:41:59,489] INFO Shutting down. (kafka.log.LogManager)
[2017-05-22 15:41:59,520] INFO Shutdown complete. (kafka.log.LogManager)
[2017-05-22 15:41:59,521] INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2017-05-22 15:41:59,525] INFO Session: 0x15c30cd33890000 closed (org.apache.zookeeper.ZooKeeper)
[2017-05-22 15:41:59,526] INFO EventThread shut down for session: 0x15c30cd33890000 (org.apache.zookeeper.ClientCnxn)
[2017-05-22 15:41:59,527] INFO [Kafka Server 0], shut down completed (kafka.server.KafkaServer)
[2017-05-22 15:41:59,528] FATAL Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
kafka.common.KafkaException: Socket server failed to bind to <internal_ip>:9092: Cannot assign requested address.
    at kafka.network.Acceptor.openServerSocket(SocketServer.scala:330)
    at kafka.network.Acceptor.<init>(SocketServer.scala:255)
    at kafka.network.SocketServer$$anonfun$startup$1.apply(SocketServer.scala:98)
    at kafka.network.SocketServer$$anonfun$startup$1.apply(SocketServer.scala:90)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
    at kafka.network.SocketServer.startup(SocketServer.scala:90)
    at kafka.server.KafkaServer.startup(KafkaServer.scala:215)
    at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:39)
    at kafka.Kafka$.main(Kafka.scala:67)
    at kafka.Kafka.main(Kafka.scala)
Caused by: java.net.BindException: Cannot assign requested address
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:433)
    at sun.nio.ch.Net.bind(Net.java:425)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67)
    at kafka.network.Acceptor.openServerSocket(SocketServer.scala:326)
    ... 10 more
[2017-05-22 15:41:59,528] INFO [Kafka Server 0], shutting down (kafka.server.KafkaServer)

EDIT #2 Wound up moving zookeeper to another node and that cleared everything up. I'm still not sure why the zookeeper and the Kafka services can't run on the same instance.

I am experiencing the same issue - did you manage to find a fix for the issue? In my case, I am already running zookeeper on a separate instance. — fuzzi, Nov 05 '18 at 21:28

Amm Sokun · Answer 1 · 2017-08-31T07:07:21.040

2

I had the same issue and it was fixed by setting zookeeper.connect=[public-ip]:2181

edited Aug 31 '17 at 07:07

answered Aug 31 '17 at 06:57

Amm Sokun

1,298
4
20
35

score 0 · Answer 2 · answered May 21 '17 at 07:34

0

the listeners and advertised.listeners setting seem correct. Have you also exposed port 9092 (for your Kafka broker) ?

answered May 21 '17 at 07:34

Abhishek

1,175
1
11
21

Yeah, that's what I thought but I checked the console and the port is open for all IPs. I updated my post with some log output, if that helps. – duanemat May 22 '17 at 15:45

score 0 · Answer 3 · answered May 23 '17 at 13:39

I am basing my solution on a previous similar issue posted here:

Why I cannot connect to Kafka from outside?

The Kafka FAQ says:

When a broker starts up, it registers its ip/port in ZK. You need to make sure the registered ip is consistent with what's listed in metadata.broker.list in the producer config. By default, the registered ip is given by InetAddress.getLocalHost.getHostAddress. Typically, this should return the real ip of the host. However, sometimes (e.g., in EC2), the returned ip is an internal one and can't be connected to from outside. The solution is to explicitly set the host ip to be registered in ZK by setting the "hostname" property in server.properties. In another rare case where the binding host/port is different from the host/port for client connection, you can set advertised.host.name and advertised.port for client connection.

To explicitly set the host in the server.properties file you need to modify the variable host.name (as indicated in one of the comments from the question I am referencing).

In your logs I can see:

advertised.host.name = null

host.name = 

[2017-05-22 15:41:58,971] INFO Client environment:host.name=<internal_ip>.ec2.internal (org.apache.zookeeper.ZooKeeper)

I hope this can help you.

Those changes didn't seem to help, but I ultimately moved zookeeper to another server and that seemed to remedy the issues. I assume the issue was that I didn't configure some port properly such that the Kafka server couldn't connect to the open port. Still, it seems weird that the two couldn't co-exist on the same server, since it seemed fine with localhost. — duanemat, May 31 '17 at 03:00

Connecting to Kafka server hosted on EC2 from local machine

3 Answers3