0

I am able to set up Kafka with docker but when I try to access it with python, I am unable to do that. I am able to do that if I install python inside kafka shell, but outside kafka shell and inside docker python, I am unable to use kafka.

My Producer.py file:

import time
import random
from kafka import KafkaProducer

# give broker IP from docker
producer = KafkaProducer(bootstrap_servers='kafka:9092')

# continuous loop
var = 1
while var == 1:

    # generate a random integer
    num = random.randint(0, 10)

    # message value and key must be raw bytes
    num_bytes = bytes(str(num), encoding='utf-8')

    # send to topic on broker
    producer.send('test', value=num_bytes, key=num_bytes)

    # wait 1 second
    time.sleep(1)

My consumer.py file:

from kafka import KafkaConsumer

# continuous loop
var = 1
while var == 1:

    # initialize consumer to given topic and broker
    consumer = KafkaConsumer('test',
                            group_id='consumer-1',
                            bootstrap_servers='kafka:9092')

    # loop and print messages
    for msg in consumer:
        print (msg)

My docker-compose file:


version: '3'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    container_name: "zoo1"
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka
    container_name: "kafka1"
    ports:
     - "9092:9092"
    expose:
     - "9093"
    depends_on:
     - zookeeper
    environment:
      KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:9093,OUTSIDE://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_LISTENERS: INSIDE://0.0.0.0:9093,OUTSIDE://0.0.0.0:9092
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_CREATE_TOPICS: "test:1:1"
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
Adrian Mole
  • 49,934
  • 160
  • 51
  • 83
Divyanshu
  • 11
  • 5

1 Answers1

0

Try this one in your docker.compose (copied from actual config)

environment:
      KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka:9093,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
Vasyl Moskalov
  • 4,242
  • 3
  • 20
  • 28
  • Caused by: java.lang.IllegalArgumentException: No security protocol defined for listener LISTENER_DOCKER_EXTERNAL://127.0.0.1 – Divyanshu Feb 06 '20 at 10:51
  • Did you try to run your scripts inside docker or on a host computer? In second case try to repalce `kafka:9092` in your scripts with `localhost:9092` – Vasyl Moskalov Feb 06 '20 at 11:49
  • i have run my all my services inside docker..I have not run producer...i have run only consumer..the issue is i am able send data via the producer but consumer is not able to recieve it... – Divyanshu Feb 06 '20 at 13:15