I've tried to create a Python Consumer for kafka running on Docker. I've read about kafka listeners but still not getting the right setup. I'm new at Docker and also to kafka.
Here's my docker-compose file:
version: '2'
services:
zookeeper:
image: debezium/zookeeper:1.1
ports:
- 2181:2181
- 2888:2888
- 3888:3888
kafka:
image: debezium/kafka:1.1
ports:
- 9092:9092
links:
- zookeeper
environment:
- ZOOKEEPER_CONNECT=zookeeper:2181
mysql:
image: debezium/example-mysql:1.1
ports:
- 3306:3306
environment:
- MYSQL_ROOT_PASSWORD=debezium
- MYSQL_USER=mysqluser
- MYSQL_PASSWORD=mysqlpw
connect:
image: debezium/connect:1.1
ports:
- 8083:8083
links:
- kafka
- mysql
environment:
- BOOTSTRAP_SERVERS=kafka:9092
- GROUP_ID=1
- CONFIG_STORAGE_TOPIC=my_connect_configs
- OFFSET_STORAGE_TOPIC=my_connect_offsets
- STATUS_STORAGE_TOPIC=my_connect_statuses
Following the steps the debezium tutorial I can get the topics but not the messages using this code on Python.
consumer = KafkaConsumer(
'dbserver1.inventory.customers',
bootstrap_servers=['192.168.99.100:9092'],
auto_offset_reset='earliest',
enable_auto_commit=True,
group_id='1',
value_deserializer=lambda x: loads(x.decode('utf-8')))
print(consumer.topics())
for m in consumer:
print(m)