MongoSinkConnector: how to write data having same schema as the source

Question

I am trying to synchronous data from the MongoDBSourceConnector and write to the destination MongoDBSinkConnector. The problem is when written data has the schema as the topic message in the Kafka, not the same schema as the source database.

MongoSinkConnector

connector.class=com.mongodb.kafka.connect.MongoSinkConnector
key.projection.type=none
writemodel.strategy=com.mongodb.kafka.connect.sink.writemodel.strategy.ReplaceOneDefaultStrategy
tasks.max=1
field.renamer.regex=[]
field.renamer.mapping=[]
max.batch.size = 0
database=abc
document.id.strategy=com.mongodb.kafka.connect.sink.processor.id.strategy.ProvidedInValueStrategy
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter=org.apache.kafka.connect.json.JsonConverter
value.projection.list=_id
document.id.strategy.overwrite.existing=true
topic.override.mongo.abc.collection1.collection=collection1
topic.override.mongo.abc.collection2.collection=collection2
topics=mongo.arcusairdb.patientvisits,mongo.arcusairdb.patients
key.projection.list=_id
max.num.retries=10
delete.on.null.values=false
key.converter.schemas.enable=true
value.converter.schemas.enable=true
connection.uri=mongodb://xxxxx_test:xxxxx@xxxxxxx:27017/arcusairdb?authSource=admin
post.processor.chain=com.mongodb.kafka.connect.sink.processor.DocumentIdAdder
retries.defer.timeout=5000
rate.limiting.timeout=0
value.projection.type=none
rate.limiting.every.n=0

Actual Result

{
    "_id" : {
        "_data" : "8263CF9C73000000712B022C0100296E5A1004E67A90CFC7F34297BEB9316ACC15797446645F696400645F2965BF8FAC2EE61B71467F0004"
    },
    "operationType" : "update",
    "clusterTime" : Timestamp(1674550387, 113),
    "fullDocument" : {
        "_id" : ObjectId("5f2965bf8fac2ee61b71467f"),
        "data1": "test1",
        "data2": "test2",
        "modifiedat" : ISODate("2023-01-24T08:53:07.432+0000"),
        "modifiedby" : ObjectId("60619cdde9d05060d6f7db87"),
    },
    "ns" : {
        "db" : "abc",
        "coll" : "collection1"
    },
    "documentKey" : {
        "_id" : ObjectId("5f2965bf8fac2ee61b71467f")
    },
    "updateDescription" : {
        "updatedFields" : {
            "__v" : NumberInt(2),
            
            "modifiedat" : ISODate("2023-01-24T08:53:07.432+0000"),
            "modifiedby" : ObjectId("60619cdde9d05060d6f7db87"),
            
        },
        "removedFields" : [

        ]
    }
}

Expected Data

{
"_id" : ObjectId("5f2965bf8fac2ee61b71467f"),
        "data1": "test1",
        "data2": "test2",
        "modifiedat" : ISODate("2023-01-24T08:53:07.432+0000"),
        "modifiedby" : ObjectId("60619cdde9d05060d6f7db87"),
}

MongoSinkConnector: how to write data having same schema as the source

0 Answers0