0

I am trying out Kafka by running it on Windows 10 machines (JVM). Kafka is working fine along with zookeeper. I am also trying to run Kafka connect to ingest data from Postgres running inside docker. After much configuration, I was able to establish a connection with Postgres (without any errors) however I do not get the data ingested from the table into Kafka topic, yet also not see any errors. I know it is not recommended to run kafka directly on JVM but I am just testing it out and wanted to make sure that I can ingest data from postgres into a kafka topic. Am I missing something in the configuration of the jdbc connector properties file. I have also placed the jar for kafka-connect-jdbc and postgres-jdbc.

name=postgress-to-topic
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
connection.url=jdbc:postgresql://host.docker.internal:49153/StudentDB
connection.user=<username>
connection.password=<password>
topic.prefix=test-events
mode=incrementing
incrementing.column.name=id
errors.log.enable=true
errors.log.include.messages=true
tasks.max=1
poll.interval.ms =100
schema.pattern=public
table.whitelist=Students
key.converter.schemas.enable=false
value.converter.schemas.enable=false
abstract33
  • 1
  • 1

1 Answers1

1

You appear to be using Docker to run Connect. If this is true, you could run connect-standalone.bat instead, and use localhost in connection.url setting.

I'd also recommend using Debezium instead of JDBC source to snapshot and "tail" the database changes.

But if you are having connection issues to Kafka, which you didnt see the error logs for, you need to modify Kafka to be reachable from the container.

Related - Connect to Kafka on host from Docker (ksqlDB)

You could also use WSL2, which is more recommended than directly in Windows, or start everything in Docker.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245