10

I'm trying to find examples of kafka connect with springboot. It looks like there is no spring boot integration for kafka connect. Can some one point me in the right direction to be able to listen to changes on mysql db?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
user3310115
  • 1,372
  • 2
  • 18
  • 48

1 Answers1

8

Kafka Connect doesn't really need Spring Boot because there is nothing for you to code for it, and it really works best when ran in distributed mode, as a cluster, not embedded within other (single-instance) applications. I suppose if you did want to do it, then you could copy relevent portions of the source code, but that of course isn't using Spring Boot, and you'd have to wire it all yourself

The framework itself consists of a few core Java dependencies that have already been written (Debezium or Confluent JDBC Connector, for your mysql example), and two config files. One for Kafka Connect to know the bootstrap servers, serializers, etc. and another for the actual MySQL connector. So, if you want to use Kafka Connect, run it by itself, then just write the consumer in the Spring app.

The alternatives to Kafka Connect itself would be to use Apache Camel within a Spring application (Spring Integration) or Spring Cloud Dataflow and interfacing with those Kafka "components" (which aren't using the Connect API, AFAIK)

Another option, specific for listening to MySQL, is to use Debezium Engine within your code.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • So then how can we integrate with a java application. Can you point out to some examples – user3310115 Dec 30 '18 at 10:29
  • So its something like download docker image, and change config and run right? And one more question connector need to be install on database to stream the changes? Or its just a config changes? – Ryuzaki L Dec 30 '18 at 15:36
  • 1
    @user331 I suggest searching for "Debezium Embedded" if you want to do this. As I said, out of the box, Kafka Connect isn't exactly designed to be ran as part of another application – OneCricketeer Dec 31 '18 at 02:03
  • @Deadpool Docker is just one option, not required, but yes, just configs. And depends on the database, like Mongo/MySql/Postgres use the replication oplog. Couchbase has its own CDC protocols, though, for example. These act over the network, though, not running on the database server itself – OneCricketeer Dec 31 '18 at 02:06
  • i'm using postgres database, so it will use the replication oplog, but i have a question some of the organizations does not allow access to log file, then any other options? @cricket_007 – Ryuzaki L Dec 31 '18 at 02:09
  • @Deadpool other than JDBC, not that I'm aware of... But that adds increased load because it's constantly querying for data. If you can convince DB admin that you'll be sending database logs over TLS, that might make them feel better about it? – OneCricketeer Dec 31 '18 at 02:11