1

I am working on deploying Kafka/Zookeeper in Kubernetes using MINIKUBE. below is my YAML file:

##################################
#   Setup Zookeeper Deployment
##################################

---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: zookeeper
  name: zookeeper
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: zookeeper
    spec:
      containers:
      - image: wurstmeister/zookeeper
       # imagePullPolicy: Always
        name: zookeeper
        ports:
        - containerPort: 2181


##################################
#   Setup Zookeeper Service
##################################

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: zookeeper-service
  name: zookeeper-service
spec:
  type: NodePort
  ports:
  - name: zookeeper-port
    port: 2181
    nodePort: 30181
    targetPort: 2181
  selector:
    app: zookeeper

---
##################################
  #   Setup Kafka service
##################################

apiVersion: v1
kind: Service
metadata:
  labels:
    app: kafka-service
  name: kafka-service
spec:
  type: NodePort
  ports:
  - name: kafka-port
    port: 9092
    nodePort: 30092
    targetPort: 9092
  selector:
    app: kafka

---
##################################
  #   Setup Kafka Broker Deployment
##################################

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: kafka
  name: kafka
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: kafka
    spec:
      containers:
      - env:
        - name: KAFKA_ADVERTISED_HOST_NAME
          value: 192.168.99.100
        - name: KAFKA_ADVERTISED_PORT
          value: "30092"
        - name: KAFKA_BROKER_ID
          value: "1"
        - name: KAFKA_ZOOKEEPER_CONNECT
          value: 192.168.99.100:30181    
        - name: KAFKA_ADVERTISED_LISTENERS
          value: "PLAINTEXT://192.168.99.100:30092"
     #   - name: KAFKA_LISTENERS
     #     value: "PLAINTEXT://192.168.99.100:9092"
        - name: KAFKA_CREATE_TOPICS
          value: "vignesh-topic:1:1"
        - name: LOG4J_LOGGER_KAFKA_AUTHORIZER_LOGGER
          value: "DEBUG"
        image: wurstmeister/kafka
        #imagePullPolicy: Always
        name: kafka
        ports:
        - containerPort: 9092

I have successfully created the Deployment/Services in local machine Kubernetes using MINIKUBE using below command.

kubectl create -f kafka.yml

I have navigated inside Kafka pods and I am able to create a topic using below command,

./bin/kafka-topics.sh --create --zookeeper 192.168.99.100:30181 --replication-factor 1 --partitions 1 --topic test-topic

But, When I try to send a message to the topic (test-topic), the system throws the below error.

error

Note

when I run netstat -tunap , both port 30092 and 30181 is showing established.

netstat

I don't know what I am missing here. Please help me to move forward.

Thanks and Appreciate your help.

Vignesh
  • 814
  • 7
  • 29
  • consume messages in the topic `test-topic` and check if your message has been produced successfully. – Soheil Pourbafrani Oct 24 '18 at 13:14
  • Hi @SoheilPourbafrani, thank you for quick reply.I have checked it. In consumer side, system does not produce message to consumer. – Vignesh Oct 24 '18 at 13:26
  • try `netstat -tan` in the cmd and check if port `30092` is active? Share us the output for TCP active ports. Note: the Kafka default port is `9092` – Soheil Pourbafrani Oct 24 '18 at 13:32
  • Hi @SoheilPourbafrani, i run netstat -tan and i updated the question with images for your reference. it show that port is established. – Vignesh Oct 24 '18 at 13:39
  • Is these two line commented in yml file: # - name: KAFKA_LISTENERS # value: "PLAINTEXT://192.168.99.100:9092" – Soheil Pourbafrani Oct 24 '18 at 13:50
  • I have checked it. yes. it was commented in server.properties. – Vignesh Oct 24 '18 at 13:52
  • uncomment it, restart Kafka server and try to produce message again. – Soheil Pourbafrani Oct 24 '18 at 13:56
  • Ok sure, just to confirm, What port should be used there? 9092 or 30092 in KAFKA_LISTENERS value #. – Vignesh Oct 24 '18 at 13:58
  • use port 9092 and in produce command try `--broker-list YourIP:9092`, too – Soheil Pourbafrani Oct 24 '18 at 14:02
  • i have used 9092. when i restart kafka, it went to shut down. following is the error i am receiving. "Socket server failed to bind to 192.168.99.100:9092: Address not available". – Vignesh Oct 24 '18 at 14:06
  • @SoheilPourbafrani even i have tried with port 30092 and i restarted the kafka. it went to shut down and same error i am receiving here. "Socket server failed to bind to 192.168.99.100:30092: Address not available". – Vignesh Oct 24 '18 at 14:29
  • The advertised listeners need to be external addresses, possibly only made available as an Ingress kubernetes service because the returned addresses actually come from Zookeeper, not just the initial connection to the open Kafka port – OneCricketeer Oct 24 '18 at 23:58
  • Hi @cricket_007, thank you for your thoughts. here i have used `NodePort` to expose the `kafka-broker port to 30092` and `zookeeper port to 30181`. Just to confirm, to externalize the ports do i need to add ingress here? not sure how to achieve what you saying. please help me with some reference to move forward. thanks! – Vignesh Oct 25 '18 at 11:33
  • I don't have experience with these wurstmeister images or with minikube, but I know that Confluent has helm charts and the Strimzi project also has K8s templates – OneCricketeer Oct 25 '18 at 14:31

1 Answers1

0

Thank you @SoheilPourbafrani and @cricket_007 for your help! I have found the workaround for the question I asked above.

Once I run the below command in the window PowerShell, Kafka started properly and able to communicate with it from Node Application and Kafka Tool as well.

minikube ssh
sudo ip link set docker0 promisc on

References: Newer versions of Minikube don't allow Pods to use their own Services

Vignesh
  • 814
  • 7
  • 29