0

I have a project that using kafka, zookeeper, debezium to capture data changes from mysql database. Actually, I quite new to kafka and I just follow debezium tutorial,so I face some troubles when trying to use kafka-python client to print message value. Here is my docker-compose file and config file:

My docker-compose file

version: '2'
services:
  zookeeper:
    image: quay.io/debezium/zookeeper:1.8
    ports:
     - 2181:2181
     - 2888:2888
     - 3888:3888
  kafka:
    image: quay.io/debezium/kafka:1.8
    ports:
     - 9092:9092
    links:
     - zookeeper
    environment:
     - ZOOKEEPER_CONNECT=zookeeper:2181
     - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092
  mysql:
    image: quay.io/debezium/example-mysql:1.8
    ports:
     - 3306:3306
    environment:
     - MYSQL_ROOT_PASSWORD=debezium
     - MYSQL_USER=mysqluser
     - MYSQL_PASSWORD=mysqlpw
  connect:
    image: quay.io/debezium/connect:1.8
    ports:
     - 8083:8083
    links:
     - kafka
     - mysql
    environment:
     - BOOTSTRAP_SERVERS=kafka:9092
     - GROUP_ID=1
     - CONFIG_STORAGE_TOPIC=my_connect_configs
     - OFFSET_STORAGE_TOPIC=my_connect_offsets
     - STATUS_STORAGE_TOPIC=my_connect_statuses

Config file

{
    "name": "inventory-connector",
    "config": {
        "connector.class": "io.debezium.connector.mysql.MySqlConnector",
        "tasks.max": "1",
        "database.hostname": "mysql",
        "database.port": "3306",
        "database.user": "debezium",
        "database.password": "dbz",
        "database.server.id": "184054",
        "database.server.name": "dbserver1",
        "database.include.list": "inventory",
        "database.history.kafka.bootstrap.servers": "kafka:9092",
        "database.history.kafka.topic": "schema-changes.inventory",
        "value.converter": "org.apache.kafka.connect.json.JsonConverter",
        "value.converter.schemas.enable": false,
        "key.converter":"org.apache.kafka.connect.json.JsonConverter",
        "key.converter.schemas.enable": false
    }
}

Python code

from kafka-python import KafkaConsumer
import json
import os
import logging
import sys
from google.cloud import bigquery

consumer = KafkaConsumer(
    "dbserver1.inventory.customers",
    auto_offset_reset="from-beginning",
    group_id='console-consumer-47348',
    bootstrap_servers=["kafka:9092"],
    api_version = (2,0,2),
    value_deserializer=lambda x: json.loads(x.decode('utf-8')),
    max_poll_interval_ms=5000,
    max_poll_records=1,
)
for message in consumer:
    msg = message.value
    print('{}'.format(msg))

Thank you very much!

Hiếu Mai
  • 1
  • What exactly is the error? Make sure you run the Python code from a container in the same compose file – OneCricketeer Mar 04 '22 at 18:46
  • @OneCricketeer Did you mean I have to run the python code inside the Kafka shell ? – Hiếu Mai Mar 05 '22 at 05:44
  • 1) Is there actually an error in the code 2) You could, but no, I'm saying to make a new Python app container if you want to connect to `kafka:9092`. Otherwise, please read answer at https://stackoverflow.com/questions/51630260/connect-to-kafka-running-in-docker – OneCricketeer Mar 06 '22 at 15:14

0 Answers0