3

I am using JDBC source and kafka-connect-mongodb sink connector to stream data from JDBC to MongoDB.

So far, I am able to get all inserts and updates as a new document (Yes, updates are inserted as new documents) on MongoDB side with following configs;

repo: https://github.com/hpgrahsl/kafka-connect-mongodb

configs:

"mongodb.document.id.strategy":"at.grahsl.kafka.connect.mongodb.processor.id.strategy.BsonOidStrategy",
"mongodb.post.processor.chain":"at.grahsl.kafka.connect.mongodb.processor.DocumentIdAdder",
"mongodb.delete.on.null.values":false,
"mongodb.writemodel.strategy":"at.grahsl.kafka.connect.mongodb.writemodel.strategy.ReplaceOneDefaultStrategy"

What I would like to achieve now is to update a document in case a row is updated on Oracle side.

With below configurations, I am able to get only the last insert or update done in Oracle side into MongoDB collection as a single document. I mean what I found in MongoDB is a collection with one document which is the last operation. 

"mongodb.document.id.strategy":"at.grahsl.kafka.connect.mongodb.processor.id.strategy.PartialValueStrategy",
"mongodb.key.projection.list": "name,surname",
"mongodb.key.projection.type":"whitelist",
"mongodb.post.processor.chain":"at.grahsl.kafka.connect.mongodb.processor.DocumentIdAdder",
"mongodb.writemodel.strategy":"at.grahsl.kafka.connect.mongodb.writemodel.strategy.ReplaceOneBusinessKeyStrategy"

Appriciate any help in advance. Many thanks, Soner

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Soner Guzeloglu
  • 103
  • 1
  • 11
  • 1
    I figured out the necessary configurations. I created a column on table called _id. It is simply a copy of primary key "id". I used ProvidedInValueStrategy and viola! – Soner Guzeloglu Jun 12 '19 at 10:37
  • See here -https://stackoverflow.com/questions/59245208/can-we-update-upsert-a-record-in-mongodb-data-source-is-kafka/60705264#60705264 – Shubham Dixit Mar 16 '20 at 12:19

0 Answers0