1

I'm trying to connect replicate a table at realtime using Kafka connect. The database used is MySQLv5.7. On working with insert and update mode separately, the columns are behaving as expected. However, when I use the upsert mode, no change is observed in the database.

Configuration File filled via UI

Sink

topic = custom-p2p
Connector Class = JdbcSinkConnector  
name = sink
tasks-max = 1
Key-converter-class=org.apache.kafka.connect.storage.StringConverter
Value-converter-class=org.apache.kafka.connect.json.JsonConverter
jdbc_url=jdbc:mysql://127.0.0.1:3306/p2p_service_db4?user=root&password=root&useSSL=false
insert mode = upsert
auto create = true
auto evolve = true

Source

Connector Class = JdbcSourceConnector
name = source-new
task max = 1
key converter class = org.apache.kafka.connect.storage.StringConverter
value converter class = org.apache.kafka.connect.json.JsonConverter
jdbc url = jdbc:mysql://127.0.0.1:3306/p2p_service_db3?user=root&password=root&useSSL=false
table loading mode = timestamp+incrementing
incrementing column name = auto_id
timestamp column name = last_updated_at
topic prefix = custom-

ver The issue that I'm having is that when the sink insert mode is changed to insert, the insertion takes place properly when changed to update, this also happens perfectly as expected, however when the value is changed to upsert, neither insertion nor update takes place. Please let me know if something done is wrong? Why this mode is not working? Is there some alternative to this if this inserts and updates both need to be replicated in the backup DB.

Thank you in advance. Let me know if some other information is needed

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Tushar Gandhi
  • 237
  • 4
  • 17
  • It's usually recommended to use Debezium if you want to capture all changes of the source database in your destination. But if you're going from MySQL to MySQL anyway, there's likely better options assuming you're not pulling messages elsewhere off the Kafka topic. – OneCricketeer Mar 16 '19 at 20:00
  • @cricket_007 better options as in? Could you please suggest some. I will explore them too. Thanks – Tushar Gandhi Mar 17 '19 at 02:15
  • MySQL replication? https://dev.mysql.com/doc/refman/5.7/en/replication.html – OneCricketeer Mar 17 '19 at 20:40

0 Answers0