0

I created a AWS MSK instance with two brokers. I can use that to produce and consume messages by using kafka#s shell commands (kafka-console-producer.sh and kafka-console-consumer.sh) but the problem is, it is not working with kafka python library.

consumer = KafkaConsumer(
      bootstrap_servers=["address_of_broker_1:9092","address_of_broker_1:9092"],
      auto_offset_reset='latest',
      enable_auto_commit=True,
      api_version=(2,0,2),
      group_id='group8',
 fetch_max_wait_ms=5)
consumer.subscribe(['test1'])

for msg in consumer:
    print(msg)

This does not work because,

sudo sh kafka-consumer-groups.sh --bootstrap-server address_of_broker1:9092,address_of_broker2:9092 --list --all-groups

this only show 2 default consumer

amazon.msk.canary.group.broker-1
amazon.msk.canary.group.broker-2

whereas, when I use kafka-console-consumer.sh is show a newly added consumer in the list.

Problem is, the python code does not show any error message. Which is why I am completely clueless.

Can anybody give me a clue what might be happening or where can I see the error log in AWS MSK

Pashyant Srivastava
  • 635
  • 3
  • 17
Mainul
  • 13
  • 5
  • Typo? Your Python code says 9022, not 9092 like your shell commands – OneCricketeer May 18 '21 at 13:21
  • ah, sorry, but i does not working either – Mainul May 18 '21 at 14:28
  • Does `test1` topic actually have data being produced to it? You're going to get no group until offsets are actually committed. Since you're starting at the end of a topic, then if there's no data, nothing needs to be committed – OneCricketeer May 18 '21 at 14:31
  • 1
    found the problem. I was running the python code from my local network and to be connected the code need to be run within the same vpc. It works if I run form my EC2 which is in the same vpc of msk – Mainul May 18 '21 at 16:24
  • Or you could open up the VPC firewall to allow traffic from your address – OneCricketeer May 18 '21 at 23:30
  • You can turn on debugging to get details on the msk connection. Configure logging via import logging logger = logging.getLogger(__name__) (surround name with two "_") SO is removing them.) logger.setLevel(logging.DEBUG) – Jason Heithoff Apr 21 '23 at 11:26

0 Answers0