Stack Overflow
Stack Overflow

Questions tagged [apache-kafka-connect]

Apache Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other data systems.

Apache Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other data systems.

It was first released with Kafka 0.9. It allows to import data from external systems (e.g., databases) into Kafka and it allows to export data from Kafka into external system (e.g., Hadoop). Apache Kafka Connect is a framework that supports a plug-in mechanism allowing to provide custom connectors for your system of choice.

Documentation

3693 questions
7
votes
2 answers

Use Avro in KafkaConnect without Confluent Schema Registry

We have vanilla apache Kafka setup in current infrastructure and we started logging some data that we want to process using Kafka Connect. Currently we use Avro for our message format, but there's no Schema Registry in our infrastructure. In future,…
Martin Macak
  • 3,507
  • 2
  • 30
  • 54
7
votes
2 answers

How to set kafka-connect connector and task's JVM heap size?

Does kafka connect start new connector and its tasks within kafka connect process? or a new JVM process will be forked. If it starts plugin within kafka connect process, then I need set kafka connect JVM heap size via KAFKA_CONNECT_JVM_HEAP_OPT…
Xiang Zhang
  • 2,831
  • 20
  • 40
7
votes
0 answers

Kafka Source Connect does not seem to do anything

I tried creating this connector from Confluent Control Center and clicking Continue on the first screen does not do anything, does not prompt for an error. When I ran it from the bash it won't stop, I don't see any extraordinary errors but it…
Gokhan
  • 279
  • 3
  • 13
7
votes
1 answer

Read data from KSQL tables

maybe this is a beginner question but what is the recommended way to read data produced in KSQL? Let's assume I do some stream processing and write the data to a KSQL table. Now I want to access this data via a Spring application (e.g. fan-out some…
DerM
  • 1,497
  • 1
  • 14
  • 27
7
votes
2 answers

Kafka Connect failed to start

I installed kafka confluent oss 4.0 on a fresh linux centos 7 but kafka connect failed to start. Steps to reproduce : - Install Oracle JDK 8 - Copy confluent-4.0.0 folder on opt/confluent-4.0.0 - Run /opt/confluent-4.0.0/confluent start Result…
Dev
  • 115
  • 1
  • 9
7
votes
2 answers

What should I use: Kafka Stream or Kafka consumer api or Kafka connect

I would like to know what would be best for me: Kafka stream or Kafka consumer api or Kafka connect? I want to read data from topic then do some processing and write to database. So I have written consumers but I feel I can write Kafka stream…
Nandish Kotadia
  • 441
  • 1
  • 6
  • 21
7
votes
2 answers

Kafka Connect JDBC sink connector not working

I am trying to use Kafka Connect JDBC sink connector to insert data into Oracle but it is throwing an error . I have tried with all the possible configurations of the schema. Below is the examples . Please suggest if I am missing anything below are…
Digvijay Waghela
  • 227
  • 1
  • 5
  • 15
7
votes
2 answers

Kafka JDBC connector load all data, then incremental

I am trying to figure out how to fetch all data from a query initially, then incrementally only changes using kafka connector. The reason for this is i want to load all data into elastic search, then keep es in sync with my kafka…
mike01010
  • 5,226
  • 6
  • 44
  • 77
7
votes
2 answers

Reset the JDBC Kafka Connector to start pulling rows from the beginning of time?

The Kafka Connector can make use of a primary key and a timestamp to determine which rows need to be processed. I'm looking for a way to reset the Connector so that it will process from the beginning of time.
user2122031
  • 581
  • 5
  • 11
7
votes
2 answers

Kafka-Connect: Creating a new connector in distributed mode is creating new group

I am currently working with confluent 3.0.1 platform. I am trying to create 2 connectors on two different workers but trying to create a new connector is creating a new group for it. Two connectors were created using below details: 1) POST…
Renukaradhya
  • 812
  • 2
  • 19
  • 31
7
votes
2 answers

Kafka connector- cannot stop rebalancing

I am using kafka connector confluent 3.0.1 version.I create a new group named new-group and there are about 20 topics on it.most of these topics is busy.But It is a pity that when I start the connector framework , the system cannot stop rebalanceing…
wuchang
  • 3,003
  • 8
  • 42
  • 66
6
votes
2 answers

Can't start Kafka Connect with MongoDb plugin with Apache Kafka

I am new to Kafka and I want to see if I can sync MongoDb data with another system using Kafka. My set up: I am running AWS MSK Cluster and I have created an EC2 instance with Kafka client manually. I have added MongoDB Kafka Connect Plugin to…
Semant1ka
  • 647
  • 10
  • 26
6
votes
1 answer

How to get kafka message's headers in Kafka Connect Sink connector with MongoDB

How do I retrieve incoming headers from the kafka message with Kafka Connect to store them as additional data fields with MongoDB Sink Connector to mongodb. I have a kafka topic "PROJECT_EXAMPLE_TOPIC". As you see I am already able to save msg…
linuxidefix
  • 61
  • 1
  • 3
6
votes
5 answers

Kafka.JS refuses to connect <<[BrokerPool] Failed to connect to seed broker, trying another broker from the list>>

Consider the Kafka Producer : const { Kafka, logLevel } = require('kafkajs'); const kafka = new Kafka({ clientId: 'my-app', brokers: ['kafka1:9092', 'kafka2:9092'], logLevel: logLevel.ERROR, }); const run = async () => { const producer =…
JAN
  • 21,236
  • 66
  • 181
  • 318
6
votes
1 answer

Kafka Connector - Tolerance exceeded in error handler

I have above 50 source connectors over sql server but two of them are going in error, please tell me what could be the reason as we have limited access to kafka server. { "name": "xxxxxxxxxxxxx", "connector": { "state": "RUNNING", …
Owais Ajaz
  • 244
  • 5
  • 20
1 2 3
…
99 100