0

Trying to launch Apache Kafka and zookeeper in AWS ECS environment with an ALB. Upon startup the docker instance gets killed after few seconds . The Zookeeper instance has been linked to the kafka instance .

Env variables used while launching the Kafka container: 

KAFKA_ADVERTISED_HOST_NAME  <AWS EC2 Instance Private IP Address > ( used to modify the advertised.hostname in server.properties file )
KAFKA_ZOOKEEPER_CONNECT zookeeper:2181 

Host Port = 9092 
Container port =9092

Docker logs for kafka container 

2018-03-05 05:58:08,218] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2018-03-05 05:58:08,218] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2018-03-05 05:58:08,218] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2018-03-05 05:58:08,218] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2018-03-05 05:58:08,218] INFO Client environment:os.version=4.9.77-31.58.amzn1.x86_64 (org.apache.zookeeper.ZooKeeper)
[2018-03-05 05:58:08,218] INFO Client environment:user.name=root (org.apache.zookeeper.ZooKeeper)
[2018-03-05 05:58:08,218] INFO Client environment:user.home=/root (org.apache.zookeeper.ZooKeeper)
[2018-03-05 05:58:08,218] INFO Client environment:user.dir=/ (org.apache.zookeeper.ZooKeeper)
[2018-03-05 05:58:08,219] INFO Initiating client connection, connectString=zookeeper:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@363ee3a2 (org.apache.zookeeper.ZooKeeper)
[2018-03-05 05:58:08,251] INFO Waiting for keeper state SyncConnected (org.I0Itec.zkclient.ZkClient)
[2018-03-05 05:58:08,253] INFO Opening socket connection to server zookeeper/172.17.0.2:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2018-03-05 05:58:14,252] INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2018-03-05 05:58:14,256] WARN Client session timed out, have not heard from server in 6005ms for sessionid 0x0 (org.apache.zookeeper.ClientCnxn)
[2018-03-05 05:58:14,368] INFO EventThread shut down for session: 0x0 (org.apache.zookeeper.ClientCnxn)
[2018-03-05 05:58:14,368] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper)
[2018-03-05 05:58:14,371] FATAL Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.I0Itec.zkclient.exception.ZkTimeoutException: **Unable to connect to zookeeper server 'zookeeper:2181' with timeout of 6000 ms**
    at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1233)
    at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:157)
    at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:131)
    at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:115)
    at kafka.utils.ZkUtils$.withMetrics(ZkUtils.scala:92)
    at kafka.server.KafkaServer.initZk(KafkaServer.scala:350)
    at kafka.server.KafkaServer.startup(KafkaServer.scala:194)
    at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
    at kafka.Kafka$.main(Kafka.scala:92)
    at kafka.Kafka.main(Kafka.scala)
[2018-03-05 05:58:14,381] INFO shutting down (kafka.server.KafkaServer)
[2018-03-05 05:58:14,410] INFO shut down completed (kafka.server.KafkaServer)
[2018-03-05 05:58:14,411] FATAL Exiting Kafka. (kafka.server.KafkaServerStartable)
[2018-03-05 05:58:14,418] INFO shutting down (kafka.server.KafkaServer)

Docker logs for Zookeeper container: 

ZooKeeper JMX enabled by default
Using config: /opt/zookeeper-3.4.9/bin/../conf/zoo.cfg
2018-03-05 05:58:05,694 [myid:] - INFO  [main:QuorumPeerConfig@124] - Reading configuration from: /opt/zookeeper-3.4.9/bin/../conf/zoo.cfg
2018-03-05 05:58:05,703 [myid:] - INFO  [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2018-03-05 05:58:05,703 [myid:] - INFO  [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 1
2018-03-05 05:58:05,704 [myid:] - WARN  [main:QuorumPeerMain@113] - Either no config or no quorum defined in config, running  in standalone mode
2018-03-05 05:58:05,704 [myid:] - INFO  [PurgeTask:DatadirCleanupManager$PurgeTask@138] - Purge task started.
2018-03-05 05:58:05,729 [myid:] - INFO  [PurgeTask:DatadirCleanupManager$PurgeTask@144] - Purge task completed.
2018-03-05 05:58:05,732 [myid:] - INFO  [main:QuorumPeerConfig@124] - Reading configuration from: /opt/zookeeper-3.4.9/bin/../conf/zoo.cfg
2018-03-05 05:58:05,732 [myid:] - INFO  [main:ZooKeeperServerMain@96] - Starting server
2018-03-05 05:58:05,751 [myid:] - INFO  [main:Environment@100] - Server environment:zookeeper.version=3.4.9-1757313, built on 08/23/2016 06:50 GMT
2018-03-05 05:58:05,751 [myid:] - INFO  [main:Environment@100] - Server environment:host.name=5944c205a288
2018-03-05 05:58:05,751 [myid:] - INFO  [main:Environment@100] - Server environment:java.version=1.7.0_65
2018-03-05 05:58:05,751 [myid:] - INFO  [main:Environment@100] - Server environment:java.vendor=Oracle Corporation
2018-03-05 05:58:05,751 [myid:] - INFO  [main:Environment@100] - Server environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre
2018-03-05 05:58:05,751 [myid:] - INFO  [main:Environment@100] - Server environment:java.class.path=/opt/zookeeper-3.4.9/bin/../build/classes:/opt/zookeeper-3.4.9/bin/../build/lib/*.jar:/opt/zookeeper-3.4.9/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/zookeeper-3.4.9/bin/../lib/slf4j-api-1.6.1.jar:/opt/zookeeper-3.4.9/bin/../lib/netty-3.10.5.Final.jar:/opt/zookeeper-3.4.9/bin/../lib/log4j-1.2.16.jar:/opt/zookeeper-3.4.9/bin/../lib/jline-0.9.94.jar:/opt/zookeeper-3.4.9/bin/../zookeeper-3.4.9.jar:/opt/zookeeper-3.4.9/bin/../src/java/lib/*.jar:/opt/zookeeper-3.4.9/bin/../conf:
2018-03-05 05:58:05,751 [myid:] - INFO  [main:Environment@100] - Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
2018-03-05 05:58:05,752 [myid:] - INFO  [main:Environment@100] - Server environment:java.io.tmpdir=/tmp
2018-03-05 05:58:05,754 [myid:] - INFO  [main:Environment@100] - Server environment:java.compiler=<NA>
2018-03-05 05:58:05,754 [myid:] - INFO  [main:Environment@100] - Server environment:os.name=Linux
2018-03-05 05:58:05,754 [myid:] - INFO  [main:Environment@100] - Server environment:os.arch=amd64
2018-03-05 05:58:05,754 [myid:] - INFO  [main:Environment@100] - Server environment:os.version=4.9.77-31.58.amzn1.x86_64
2018-03-05 05:58:05,754 [myid:] - INFO  [main:Environment@100] - Server environment:user.name=root
2018-03-05 05:58:05,754 [myid:] - INFO  [main:Environment@100] - Server environment:user.home=/root
2018-03-05 05:58:05,754 [myid:] - INFO  [main:Environment@100] - Server environment:user.dir=/opt/zookeeper-3.4.9
2018-03-05 05:58:05,757 [myid:] - INFO  [main:ZooKeeperServer@815] - tickTime set to 2000
2018-03-05 05:58:05,757 [myid:] - INFO  [main:ZooKeeperServer@824] - minSessionTimeout set to -1
2018-03-05 05:58:05,757 [myid:] - INFO  [main:ZooKeeperServer@833] - maxSessionTimeout set to -1
2018-03-05 05:58:05,785 [myid:] - INFO  [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:2181
perror
  • 7,071
  • 16
  • 58
  • 85
sourabh vs
  • 1
  • 1
  • Zookeeper <-> Kafka is a two way street; have you opened port access on both containers? Can you post some more details for how you're linking the containers (screenshots would be sufficient)? Do you have security groups that may be blocking traffic? – MrDuk Mar 07 '18 at 15:12
  • yes all the necessary port are kept opened . Also i am trying to load balance the hosts using a network load balancer now . The target groups have listeners ( 9092 , 9093 , 2181) that are being used to connect to the instances (Able to telnet to the host machine on port 9092 using the Load balancer's DNS ) . – sourabh vs Mar 09 '18 at 05:44
  • I have a task defination which has few env variables that can identify the kafka container with the zookeeper container KAFKA_ADVERTISED_HOST_NAME KAFKA_ADVERTISED_PORT 9092 or 9093 ( port used by advertised.listerners ) KAFKA_AUTO_CREATE_TOPICS_ENABLE false KAFKA_BROKER_ID 1 or 2 or any other number but each container must have unique ID KAFKA_MESSAGE_MAX_BYTES 1295725856 KAFKA_PORT 9092 < container 1 > 9093 < container 2 > so on KAFKA_ZOOKEEPER_CONNECT :2181 – sourabh vs Mar 09 '18 at 05:48
  • In the above mentioned env variables the "KAFKA_ADVERTISED_HOST_NAME" is the one where i add the NLB DNS name. The only parameter that is used to uniquely identify the brokers are the PORT numbers , one is running on 9092 and the other on 9093 . – sourabh vs Mar 09 '18 at 05:51
  • Containers are existing in few seconds, so the issue would be health check failed or CMD command problem in docker image – Ashok Reddy Aug 21 '18 at 09:24

0 Answers0