Stack Overflow
Stack Overflow

Questions tagged [apache-kafka-connect]

Apache Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other data systems.

Apache Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other data systems.

It was first released with Kafka 0.9. It allows to import data from external systems (e.g., databases) into Kafka and it allows to export data from Kafka into external system (e.g., Hadoop). Apache Kafka Connect is a framework that supports a plug-in mechanism allowing to provide custom connectors for your system of choice.

Documentation

3693 questions
11
votes
1 answer

When does Kafka Leader Election happen?

When and how often does Kafka High Level Producer elect a leader? Does it do before sending each message or only once at the time of creating connection?
Marut Singh
  • 345
  • 1
  • 3
  • 10
11
votes
2 answers

Second and Third Distributed Kafka Connector workers failing to work correctly

With a Kafka cluster of 3 and a Zookeeper cluster of the same I brought up one distributed connector node. This node ran successfully with a single task. I then brought up a second connector, this seemed to run as some of the code in the task…
LaserJesus
  • 8,230
  • 7
  • 47
  • 65
10
votes
1 answer

Confluent platform Kafka Connect crashed with Exit 137

On Mac, I pulled latest docker images. As I run stack, everything seems to be okay but "Connect" crashed with Exit 137. When I looked at the command center, cluster health seems to be fine. What is the impact of this? How to correct the problem…
Sahas
  • 3,046
  • 6
  • 32
  • 53
10
votes
0 answers

Use RegexRouter to replace multiple dots in topics with underscores

It seems like it's not possible to use the RegexRoute to replace all the dots in a topic name with underscores since RegexRouter calls replaceFirst not replaceAll. Is there a way around this? One thought I had was to take multiple passes with…
moku
  • 4,099
  • 5
  • 30
  • 52
10
votes
1 answer

Apache Kafka Connect With Springboot

I'm trying to find examples of kafka connect with springboot. It looks like there is no spring boot integration for kafka connect. Can some one point me in the right direction to be able to listen to changes on mysql db?
user3310115
  • 1,372
  • 2
  • 18
  • 48
10
votes
3 answers

Updating a Debezium MySQL connector with table whitelist option

I'm using the Debezium (0.7.5) MySQL connector and I'm trying to understand what is the best approach if I want to update this configuration with the option table.whitelist. Let's say I create a connector, something like this: curl -i -X POST -H…
japoneizo
  • 508
  • 1
  • 5
  • 15
10
votes
1 answer

replication slot already exists

Whenever I restart the debezium kafka-connect container, or deploy another instance, I get the following error: io.debezium.jdbc.JdbcConnectionException: ERROR: replication slot "debezium" already exists at…
Zach Mays
  • 101
  • 1
  • 1
  • 6
10
votes
1 answer

Using Kafka Connect HOWTO "commit offsets" as soon as a "put" is completed in SinkTask

I am using Kafka Connect to get messages from a Kafka Broker (v0.10.2) and then sync it to a downstream service. Currently, I have code in SinkTask#put that will process the SinkRecord & then persist it to the downstream service. A couple of key…
Chantz
  • 5,883
  • 10
  • 56
  • 79
10
votes
1 answer

Kafka-Connect vs Filebeat & Logstash

I'm looking to consume from Kafka and save data into Hadoop and Elasticsearch. I've seen 2 ways of doing this currently: using Filebeat to consume from Kafka and send it to ES and using Kafka-Connect framework. There is a Kafka-Connect-HDFS and…
Adrian
  • 5,603
  • 8
  • 53
  • 85
9
votes
1 answer

Disable mirrormaker2 offset-sync topics on source kafka cluster

We're using MirrorMaker2 to replicate some topics from one kerberized kafka cluster to another kafka cluster (strictly unidirectional). We don't control the source kafka cluster and we're given only access to describe and read specific topics that…
Alexandre Juma
  • 3,128
  • 1
  • 20
  • 46
9
votes
3 answers

Kafka design questions - Kafka Connect vs. own consumer/producer

I need to understand when to use Kafka connect vs. own consumer/producer written by developer. We are getting Confluent Platform. Also to achieve fault tolerant design do we have to run the consumer/producer code ( jar file) from all the brokers ?
Anirban
  • 257
  • 4
  • 12
9
votes
2 answers

How to stream data from Kafka to MongoDB by Kafka Connector

I want to stream data from Kafka to MongoDB by using Kafka Connector. I found this one https://github.com/hpgrahsl/kafka-connect-mongodb. But there is no step to do. After googling, it seems to lead to Confluent Platform what I don't want to…
Vu Le Anh
  • 708
  • 2
  • 8
  • 21
9
votes
0 answers

How to use the Kafka Connect JDBC to source PostgreSQL with multiple schemas that contain tables with the same name?

I need to source data from a PostgreSQL database with ~2000 schemas. All schemas contain the same tables (it is a multi-tenant application). The connector is configured as following: { "name": "postgres-source", "connector.class":…
Daniel Ferreira Jorge
  • 377
  • 3
  • 13
9
votes
4 answers

Kafka Connect can't find connector

I'm trying to use the Kafka Connect Elasticsearch connector, and am unsuccessful. It is crashing with the following error: [2018-11-21 14:48:29,096] ERROR Stopping after connector error…
Boris K
  • 3,442
  • 9
  • 48
  • 87
9
votes
1 answer

Kafka Connect Out of Java heap space after enabling SSL

I have recently enabled SSL and tried to start Kafka connect in distributed mode. When running connect-distributed connect-distributed.properties I get the following errors: [2018-10-09 16:50:57,190] INFO Stopping task…
Giorgos Myrianthous
  • 36,235
  • 20
  • 134
  • 156
1 2
3
…
99 100