0

Steps followed
Installed/extracted confluent 5.0.0.-2.11

Modified kafka-connect-jdbc/source-quickstart.properties to point to my local db 

name=test-source-sqlite-jdbc-autoincrement
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
tasks.max=1
connection.url=jdbc:mysql://127.0.0.1:3306/test?user=root&password=root
mode=incrementing
incrementing.column.name=id 
topic.prefix=sqlite-jdbc
query=select * from customer
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter=org.apache.kafka.connect.json.JsonConverter
poll.interval.ms=5000
table.poll.interval.ms=120000
mode=bulk

Modified below properties in connect-avro-standalone.properties plugin.path=extracted folder of confluent

bootstrap.servers=domain:9092
key.converter=io.confluent.connect.avro.AvroConverter
key.converter.schemas.enable=true
value.converter=io.confluent.connect.avro.AvroConverter
value.converter.schemas.enable=true
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false 
plugin.path=/home/confluent-5.0.0/

After modifying above properties I started zookeeper, kafka, schema registry & kafka connect jdbc and also avro console consumer with below command. Table referred: Customer

sh kafka-avro-console-consumer --bootstrap-server ip:9092 --topic sqlite-jdbc-customer --from-beginning

i do not see any messages in the avro console consumer

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
suseelkumar annamraju
  • 33
  • 4
  • Are you sure that your source connector is up and running? – Giorgos Myrianthous Oct 30 '18 at 09:28
  • I started schema registry and kafka connect jdbc and i see below messages. `Schema registry INFO Started @4834ms (org.eclipse.jetty.server.Server:411) INFO Server started, listening for requests... (io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain:45)` ------------- kafka connect jdbc `INFO Advertised URI: http://domain:8083/ (org.apache.kafka.connect.runtime.rest.RestServer:267) INFO REST server listening at http://domain:8083/, INFO Kafka Connect started (org.apache.kafka.connect.runtime.Connect:55)` – suseelkumar annamraju Oct 30 '18 at 09:39
  • How did you load/run your source connector? – Giorgos Myrianthous Oct 30 '18 at 09:39
  • @GiorgosMyrianthous what do you mean by source connector. Is it Kafka Connect JDBC or schema registry or database which has tables? – suseelkumar annamraju Oct 30 '18 at 09:43
  • `kafka-connect-jdbc/source-quickstart.properties` is the configuration of your source connector (i.e. the connector that replicates data form your MySQL table to Kafka). You need to run this connector in order to achieve this replication. – Giorgos Myrianthous Oct 30 '18 at 09:45
  • How did you start confluent platform? By running `confluent start`? – Giorgos Myrianthous Oct 30 '18 at 09:46
  • 1
    yes @GiorgosMyrianthous. I have started source connector using below command ./bin/connect-standalone ./etc/schema-registry/connect-avro-standalone.properties ./etc/kafka-connect-jdbc/source-quickstart-sqlite.properties – suseelkumar annamraju Oct 30 '18 at 09:47
  • @GiorgosMyrianthous No we didnt start confluent platform with 'confluent start'. We are starting zk, kafka and schema, connect with individual commands like: --- to start zookeeper: zookeeper-server-start /etc/kafka/zookeeper.properties --------------- to start kafka: kafka-server-start /etc/kafka/server.properties to start schema registry: bin/schema-registry-start etc/schema-registry/schema-registry.properties =========================== ./bin/connect-standalone ./etc/schema-registry/connect-avro-standalone.properties ./etc/kafka-connect-jdbc/source-quickstart-mysql.properties – suseelkumar annamraju Oct 30 '18 at 10:03
  • 1
    Stop adding lines of code and commands in the comments as they are not readable. If you want to do so, just edit your answer. Can you try to start Kafka Connect and source connector by running `./bin/connect-standalone /etc/schema-registry/connect-avro-standalone.properties /etc/kafka-connect-jdbc/source-quickstart-mysql.properties` ? The command you've used seems to be incorrect. Also share the output of the logs when starting Kafka Connect. – Giorgos Myrianthous Oct 30 '18 at 10:07
  • If you are using `value.converter=org.apache.kafka.connect.json.JsonConverter`, then you shouldn't be using `kafka-avro-console-consumer`. In other words, your data wouldn't be Avro, so you shouldn't try to consume in that format. – OneCricketeer Oct 30 '18 at 19:52

0 Answers0