Can´t consume messages from topic

Question

I have the following code:

from confluent_kafka.admin import AdminClient, NewTopic

a = AdminClient({'bootstrap.servers': 'localhost:9092'})

new_topics = [NewTopic(topic, num_partitions=3, replication_factor=1) for topic in ["topic1", "topic2"]]
fs = a.create_topics(new_topics)

for topic, f in fs.items():
    try:
        f.result()
        print("Topic {} created".format(topic))
    except Exception as e:
        print("Failed to create topic {}: {}".format(topic, e))

Creating the topics worked fine.

This is my producer:

from confluent_kafka import Producer

p = Producer({'bootstrap.servers': 'localhost:9092'})

some_data_source = ["hello", "wuff"]

def delivery_report(err, msg):
    if err is not None:
        print('Message delivery failed: {}'.format(err))
    else:
        print('Message delivered to {} [{}]'.format(msg.topic(), msg.partition()))

for data in some_data_source:
    p.poll(0)
    p.produce('mytopic', data.encode('utf-8'), callback=delivery_report)

p.flush()

Message delivered to mytopic [0]
Message delivered to mytopic [0]

Consumer:

from confluent_kafka import Consumer

c = Consumer({
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'mygroup',
    'auto.offset.reset': 'earliest'
})

c.subscribe(['topic1'])

while True:
    msg = c.poll(1.0)

    print(msg)
    if msg is None:
        continue
    if msg.error():
        print("Consumer error: {}".format(msg.error()))
        continue
    print('Received message: {}'.format(msg.value().decode('utf-8')))
c.close()

When I subscribe to the topic (which works), I only get None every second. Am I doing something wrong here? Does it has to do something with 'group.id': 'mygroup'? Can anyone help me?

Two things you can try: - Try producing messages outside of this and see if the consumer is able to get all the messages. You can use [kcat] (https://docs.confluent.io/platform/current/app-development/kafkacat-usage.html#kcat-formerly-kafkacat-utility) formerly kafkacat to produce messages. - Try changing the group id( with the existing code for producer and see if the `"hello", "wuff"` show up in the output. — teedak8s, Oct 31 '22 at 01:07

score 1 · Accepted Answer · answered Oct 31 '22 at 02:21

Your producer code is writing to mytopic topic. Which doesn't match your create script or what your consumer has subscribed to.

Also, if you don't want it to print None, then move the print statement inside the if statement since poll function can return None

As commented, you may also want to try further debugging with CLI tools

Can´t consume messages from topic

1 Answers1