0

I try to consume some messages created with a spring-boot kafka producer on localhost.

For the consumer I have the following python code (consumer.py):

from kafka import KafkaConsumer

# To consume latest messages and auto-commit offsets
print("consume...")
consumer = KafkaConsumer('upload_media',
                         group_id='groupId',
                         bootstrap_servers=['host.docker.internal:9092'])
for message in consumer:
    # message value and key are raw bytes -- decode if necessary!
    # e.g., for unicode: `message.value.decode('utf-8')`
    print("message")
    print ("%s:%d:%d: key=%s value=%s" % (message.topic, message.partition,
                                          message.offset, message.key,
                                          message.value))

And I have the dockerfile:

# Set python version
FROM python:3.6-stretch

# Make a directory for app
WORKDIR /consumer

# Install dependencies
COPY requirements.txt .
RUN pip install -r requirements.txt
# RUN pip install --no-cache-dir --user -r /req.txt

# Copy source code
COPY ./app/consumer.py .

# Run the application
#CMD ["python", "-m", "app"]
ENTRYPOINT [ "python3", "-u", "consumer.py" ]
CMD [ "" ]

When I build and run it, i get:

$ docker run --expose=9092 consumer
consume...

But it never gets some messages. When I start the script just with python consumer.py it works fine. So something seems to be wrong with the docker image?

Edit: I started the ZooKeeper and a broker locally as it is described here: https://kafka.apache.org/quickstart

Edit 2: I don't know why the question is a duplicate. I don't use ksqlDB, don't use docker compose. I am not on a VM and I do not get any error.

simplesystems
  • 839
  • 2
  • 14
  • 28
  • But you're not getting any sort of exception, and the consumer process stays running? If you use a tool like `kafka-console-producer` or `kcat` to manually feed messages into the topic, do they appear? I checked on `localhost` and nothing is running on port 9092; is the Kafka broker running somewhere else? – David Maze Dec 29 '22 at 12:19
  • Yes the consumer process stays running, no exception. I started the zookeeper and broker locally as it is described here: https://kafka.apache.org/quickstart – simplesystems Dec 29 '22 at 12:21
  • Your consumer doesn't need to expose any ports. You need to reconfigure your broker to not advertise localhost to clients – OneCricketeer Dec 29 '22 at 22:59
  • what do you mean by "not advertise localhost to clients" ? I want to test it all on localhost – simplesystems Dec 30 '22 at 09:20
  • _" I am not on a VM"_, you're using Docker, which means your application runs on what amounts to a VM. – Mark Rotteveel Dec 30 '22 at 15:25
  • Ok what I wanted to say is that I am not on some remote host / vm but on my local machine and I try to run that "simple" dockerized consumer, its not the same issue as it was on the linked question. And i still dont know how to get it this setup running on my local machine, any ideas? – simplesystems Dec 30 '22 at 16:48

0 Answers0