8

We can update/upsert the record in mongodb BUT is there is any method or function from which we can update or upsert the document directly in mongodb and the source system is kafka and destination is mongodb.

nixxo_raa
  • 391
  • 8
  • 21
  • Sorry not following. Update the document directly as in after reading from kafka process as update/upsert ? Can you expand that please and may be add some code ? – s7vr Mar 11 '20 at 13:08

2 Answers2

8

Yes we can update/upsert the data. For update you have to define a parameter in Kafka connector. and whitelist the column on which bases you want to update the record. The property is as followed: 

document.id.strategy=com.mongodb.kafka.connect.sink.processor.id.strategy.PartialValueStrategy
value.projection.list=tokenNumber
value.projection.type=whitelist
writemodel.strategy=com.mongodb.kafka.connect.sink.writemodel.strategy.UpdateOneTimestampsStrategy
nixxo_raa
  • 391
  • 8
  • 21
  • It works! Too bad the documentation is so hard to understand (not enough examples or just takes too much time to understand properly)... the names of the properties are just confusing in my opinion and counter intuitive I see a lot of developers have hard time setting this up. – solujic Feb 28 '21 at 08:44
  • Yes bro, you are right the documentation is too hard to understand in one go. – nixxo_raa Mar 31 '21 at 17:02
  • @nixxo_raa what's the *parameter* and *column* here in your shown configuration? – y_159 Jul 20 '21 at 08:19
0

I was struggling through it finally I got the answer. I used following Mongodb sink connector

And after bugging my head on their document for sometime ,I finally figured out the solution .

This is the exact mongodb sink connector configuration I am using

{
  "name": "mongodbsync",
  "connector.class": "at.grahsl.kafka.connect.mongodb.MongoDbSinkConnector",
  "topics": "alpha-foobar",
  "mongodb.connection.uri": "mongodb://localhost:27017/kafkaconnect?w=1&journal=true",
  "mongodb.document.id.strategy": "at.grahsl.kafka.connect.mongodb.processor.id.strategy.ProvidedInValueStrategy"

}

I left mongodb.writemodel.strategy blank in my configuration so it is taking the default one

I used the use case 2 of following docs from github of the same connector

I was dealing with this scenario ,transferring mysql table data with kafka-jdbc-source connect to mongodb sink.

Also the above strategies can be found in the official docs as well Please free to ask any doubts if you have .Thanks

Shubham Dixit
  • 9,242
  • 4
  • 27
  • 46
  • 1
    this method you have suggested doesn't work completely I'm getting an error called org.apache.kafka.connect.errors.DataException: Could not convert key `456` into a BsonDocument.\n\tat com.mongodb.kafka.connect.sink.converter.LazyBsonDocument.getUnwrapped(LazyBsonDocument.java:157)\n\tat com.mongodb.kafka.connect.sink.converter.LazyBsonDocument.clone(LazyBsonDocument.java:146)\n\tat com.mongodb.kafka.connect.sink.converter.SinkDocument.clone(SinkDocument.java:45)\n\tat – Harshith Yadav Nov 18 '20 at 04:40
  • you have to make sure that you have serializers active in kafka – Shubham Dixit Nov 18 '20 at 04:42
  • org.apache.kafka.connect.errors.DataException: Could not convert key `456` into a BsonDocument.\n\tat com.mongodb.kafka.connect.sink.converter.LazyBsonDocument.getUnwrapped(LazyBsonDocument.java:157)\n\tat com.mongodb.kafka.connect.sink.converter.LazyBsonDocument.clone(LazyBsonDocument.java:146)\n\tat com.mongodb.kafka.connect.sink.converter.SinkDocument.clone(SinkDocument.java:45)\n\tat this is the error im getting @Shubh – Harshith Yadav Nov 18 '20 at 04:42
  • let me share my connector so you can understand better – Harshith Yadav Nov 18 '20 at 04:43
  • "config":{ "connector.class" : "com.mongodb.kafka.connect.MongoSinkConnector", "tasks.max":"1", "connection.uri":"mongodb://xx.xx.xx:27017", "database":"topic1", "collection":"tag", "topics":"tag_update_new1", "key.converter": "org.apache.kafka.connect.storage.StringConverter", "value.converter":"io.confluent.connect.json.JsonSchemaConverter", "value.converter.schema.registry.url": "http://xx.xxx.xx.xx:8081", – Harshith Yadav Nov 18 '20 at 04:46
  • "document.id.strategy":"com.mongodb.kafka.connect.sink.processor.id.strategy.PartialValueStrategy", "document.id.strategy.partial.value.projection.type":"client_id", "document.id.strategy.partial.value.projection.type":"AllowList", "writemodel.strategy":"com.mongodb.kafka.connect.sink.writemodel.strategy.ReplaceOneBusinessKeyStrategy" } this is my full connector config have i missed out smthn?????? – Harshith Yadav Nov 18 '20 at 04:47