0

What is the difference between using Apache Kafka Connect API and Confluent Kafka Connect API ?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Abhinav Kumar
  • 210
  • 3
  • 13
  • 2
    This [post](https://stackoverflow.com/questions/61532385/kafka-dependencies-ccs-vs-ce/61573809#61573809) might help to understand that there are no differences in connect-api. – Michael Heil Sep 28 '20 at 07:39
  • "Confluent Kafka" does not exist. Confluent Platform simply provides a bundle. Like "Anaconda" is not a different Python altogether. – OneCricketeer Sep 28 '20 at 18:48

1 Answers1

1

As mike mentioned, there is no core difference between Apache kafka connect and confluent kafka connect.

As an example of using JDBC Connector plugin from Confluent with MySQL database to read data from MySQL and send it to kafka topic.

For a quick demo on your laptop, follow below main steps:

  1. Download Apache Kafka (from either Apache site, or you can download Confluent Platform)
  2. Run single Kafka broker using kafka-server-start script.
  3. Download kafka-connect-jdbc from Confluent Hub
  4. Edit plugin.path in connect-standalone.properties to include the full path to extracted kafka-connect-jdbc files.
  5. Download and copy mysql driver into kafka-connect-jdbc folder with other JARs (you should see sqlite JAR is already there)
  6. create jdbc source connector configuration file
  7. run Kafka connect in standalone mode with jdbc source connector configuration

$KAFKA_HOME/bin/connect-standalone.sh ../config/connect-standalone.properties ../config/jdbc-connector-config.properties

Useful links

https://www.confluent.io/blog/kafka-connect-deep-dive-jdbc-source-connector/ https://docs.confluent.io/current/connect/kafka-connect-jdbc/index.html

If you want to write code, then learn kafka producer api usage. https://docs.confluent.io/current/clients/producer.html https://kafka.apache.org/documentation/#producerapi

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
sramu
  • 1,001
  • 9
  • 9
  • Could you please provide any reference where I can find scala/java code to connect MYSQL to KAFKA for real time streaming ? I am unable to find any scala/java code. – Abhinav Kumar Oct 01 '20 at 15:05
  • @OneCricketeer Confluent Connectors only uses Configuration files. I need to implement Kafka and MYSQL connectivity using Scala code. – Abhinav Kumar Oct 06 '20 at 16:03
  • 1
    @AbhinavKumar Kafka Connect is not specific to Confluent, and they are all written and can be called from JVM code (refer `ConnectStandalone.java` in Kafka source code) – OneCricketeer Oct 06 '20 at 16:54