1

I am a beginner to kafka. here i am trying to build real time data pipeline in kafka. I have database which is located remotely. I really don't understand how to access database and database continuously updating like real time. I read about debezium [https://debezium.io/docs/tutorial/] but entire tutorial they didn't show any example wherein they have pulled the data from real time remote database. I have MySQL database.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
P Singh
  • 51
  • 7
  • In the tutorial, the chapter "Monitor the MySQL database" (https://debezium.io/docs/tutorial/#monitor-mysql) describes hwo it connects to MySQL. You configure the address of your database and the credentials in the plugin configuration along with the table(s) you want to mirror. Once you create the connector, it will be automatically pulling the data from the database and pushing them into Kafka. – Jakub Sep 02 '18 at 20:09
  • It's not quite clear what you mean by "remote" here. For sure Debezium can connect to a MySQL database on any other host, i.e. Kafka Connect and with it Debezium doesn't have to run on the same host. As cricket_007 it makes sense of course to locate the connector closely to the DB, to reduce lags (as you'd host your regular application close to the DB). – Gunnar Sep 04 '18 at 13:20

1 Answers1

3

Debezium would be ideally be running as close to the database server as possible

But if you look at the tutorial, if specifies a remote docker container, and you can replace the hostname with whatever matches your environment 

"connector.class":  "io.debezium.connector.mysql.MySqlConnector", 
"tasks.max": "1", 
"database.hostname": "... Your server here... ", 
"database.port": "3306",

Another option is the Kafka Connect JDBC connectors and you can see a few blog posts by Confluent about how you can configure them

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • Hey forgive me for chiming in a bit late here - so is it possible to run debezium on the same host as the database and send the CDC info to a remote kafka cluster? – Funsaized Jun 12 '19 at 23:19
  • It's possible, sure. Just be aware of the load that you would be adding to that server. – OneCricketeer Jun 13 '19 at 15:22