Stack Overflow
Stack Overflow

Questions tagged [mongodb-kafka-connector]

The official MongoDB Kafka Connector.

The official MongoDB Kafka Connector.

  • The sink connector can store data from Kafka topics into MongoDB.
  • The source connector can watch data changes from MongoDB and publish them onto Kafka topics.
118 questions
8
votes
2 answers

Can we update/Upsert a record in mongodb? data source is kafka

We can update/upsert the record in mongodb BUT is there is any method or function from which we can update or upsert the document directly in mongodb and the source system is kafka and destination is mongodb.
nixxo_raa
  • 391
  • 8
  • 21
7
votes
2 answers

How to change the name of the topic generated by Kafka Connect Source Connector

I have an already running production deployed Kafka-Cluster and having Topic "existing-topic". I am using MongoDB-Source-Connector from Debezium. Here all what I want is to push the CDC events directly to the topic "existing-topic" so that my…
Abhinavece
  • 119
  • 1
  • 8
6
votes
2 answers

Can't start Kafka Connect with MongoDb plugin with Apache Kafka

I am new to Kafka and I want to see if I can sync MongoDb data with another system using Kafka. My set up: I am running AWS MSK Cluster and I have created an EC2 instance with Kafka client manually. I have added MongoDB Kafka Connect Plugin to…
Semant1ka
  • 647
  • 10
  • 26
6
votes
1 answer

How to get kafka message's headers in Kafka Connect Sink connector with MongoDB

How do I retrieve incoming headers from the kafka message with Kafka Connect to store them as additional data fields with MongoDB Sink Connector to mongodb. I have a kafka topic "PROJECT_EXAMPLE_TOPIC". As you see I am already able to save msg…
linuxidefix
  • 61
  • 1
  • 3
6
votes
1 answer

Automatically reconnect failed tasks in Kafka-Connect

I'm using a mongo-source plugin with Kafka-connect. I checked the source task state, and it was running and listening on a mongo collection. I manually stopped mongod service and waited about 1 minute, then I start it back again. I checked the…
toto
  • 1,197
  • 2
  • 15
  • 26
5
votes
3 answers

Getting ConfluentHubClient exception while installing kafka connectors

I have set up Kafka cluster and installed confluent-hub on a EC2 instance. I have downloaded the confluent-hub tar file, extracted and placed the binary inside /usr/local/bin. When I am now trying to install any connectors, I am getting the…
zilcuanu
  • 3,451
  • 8
  • 52
  • 105
4
votes
0 answers

Kafka Mongodb Sink Connector - unable to find valid certification path to requested target

I am trying to set up a mongodb sink connector but I am getting an error when I start it error: {"error_code":400,"message":"Connector configuration is invalid and contains the following 1 error(s):\nConfiguration is not defined: topic\nUnable to…
Kingindanord
  • 1,754
  • 2
  • 19
  • 48
4
votes
1 answer

Distributed Official Mongodb Kafka Source Connector with Multiple tasks Not working

I am running Apache Kafka on my Windows machine with two Kafka-Connect-Workers(Port 8083, 8084) and one topic with three partitions(replication of one). My issue is that I am able to see the fail-over to other Kafka-Connect worker whenever I…
Hamid
  • 717
  • 7
  • 15
3
votes
2 answers

In Kafka, the "Key" does not match the "Id" when updating the document in MongoDB

We are trying to take all the records from MongoDB to Kafka using the com.mongodb.kafka.connect.MongoSourceConnector. The settings are used for connector as follows: { "name": "mongo-source", "config": { "connector.class":…
Vitaly Motov
  • 35
  • 4
3
votes
1 answer

Connect to MongoDB atlas within AWS VPC using AWS MSK connector

I am trying to use MongoDB change stream using Kafka. I chose AWS MSK since my entire infrastructure is within AWS and it will be easy to integrate with other AWS services. I created an AWS MSK cluster within the VPC and created an AWS MSK connect…
Jerin A Mathews
  • 8,572
  • 4
  • 26
  • 49
3
votes
2 answers

Debezium Kafka connector mongodb : Error connecting kafka connector to mongodb

Below are my MongoDB config in /etc/kafka/connect-mongodb-source.properties name=mongodb-source-connector connector.class=io.debezium.connector.mongodb.MongoDbConnector mongodb.hosts=/remoteserveraddress:27017 mongodb.name=mongo_conn…
user837593
  • 337
  • 1
  • 5
  • 25
3
votes
1 answer

Kafka-MongoDB Debezium Connector : distributed mode

I am working on debezium mongodb source connector. Can I run connector in local machine in distributed mode by giving kafka bootstrap server address as remote machine (deployed in Kubernetes) and remote MongoDB url? I tried this and I see connector…
user837593
  • 337
  • 1
  • 5
  • 25
3
votes
2 answers

How to tell MongoSource (using Kafka Connect) what Key to serialize

I'm using mongo source to listen to mongo change stream and put all events into kafka, but I'm strangling to find a way to extract the "Real" key from the event. I tried transformation, but it didn't work, giving me error: Caused by:…
Winter Ha
  • 31
  • 2
2
votes
0 answers

What is the proper way of adding trust certificates to confluent kafka connect docker image

I have a kafka connect cluster (cp_kafka_connect_base) on docker, and I need to include a .pem file in order to connect to a source over TLS. It seems there are already a number of trusted certificates included in connect, so how would I add a new…
Fraznist
  • 71
  • 4
2
votes
2 answers

Outbox pattern with kafka and mongoDB connector

I want to implement Outbox Pattern in our microservices using Mongo-Kafka connector, in my outbox(a MongoDB collection) I store topic data using these fields: kafka_topic, kafka_key, kafka_value, kafka_header_keys, kafka_header_values. But how…
Mehran Prs
  • 509
  • 4
  • 18
1
2 3 4 5 6 7 8