Stack Overflow
Stack Overflow

Questions tagged [apache-kafka-connect]

Apache Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other data systems.

Apache Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other data systems.

It was first released with Kafka 0.9. It allows to import data from external systems (e.g., databases) into Kafka and it allows to export data from Kafka into external system (e.g., Hadoop). Apache Kafka Connect is a framework that supports a plug-in mechanism allowing to provide custom connectors for your system of choice.

Documentation

3693 questions
16
votes
1 answer

Synchronizing data from MSSQL to Elasticsearch using Apache Kafka

I'm currently running a text search in SQL Server, which is becoming a bottleneck and I'd like to move things to Elasticsearch for obvious reasons, however I know that I have to denormalize data for best performance and scalability. Currently, my…
Evaldas Buinauskas
  • 13,739
  • 11
  • 55
  • 107
15
votes
2 answers

"The configuration foo.bar was supplied but isn't a known config"

When I'm starting a connector in distributed mode (connect-runtime v1.0.0), there are several configuration values that are mandatory. I'm speaking of values…
Mabi
  • 441
  • 2
  • 6
  • 17
14
votes
3 answers

What is a simple, effective way to debug custom Kafka connectors?

I'm working a couple of Kafka connectors and I don't see any errors in their creation/deployment in the console output, however I am not getting the result that I'm looking for (no results whatsoever for that matter, desired or otherwise). I made…
C. Ommen
  • 171
  • 1
  • 1
  • 10
14
votes
2 answers

Kafka-connect sink task ignores file offset storage property

I'm experiencing quite weird behavior working with Confluent JDBC connector. I'm pretty sure that it's not related to Confluent stack, but to Kafka-connect framework itself. So, I define offset.storage.file.filename property as default…
bsiamionau
  • 8,099
  • 4
  • 46
  • 73
13
votes
1 answer

Kafka Connect assigns same task to multiple workers

I'm using Kafka Connect in distributed mode. A strange behavior I observed multiple times now is that, after some time (can be hours, can be days), what appears to be a balancing error happens: same tasks get assigned to multiple workers. As a…
Tim
  • 12,318
  • 7
  • 50
  • 72
13
votes
2 answers

Use Confluent Hub without Confluent Platform installation

I am using these installation instructions for getting confluent hub client https://docs.confluent.io/current/connect/managing/confluent-hub/client.html But, when I get to the line to install the kafka-connect-elasticsearch connector confluent-hub…
dennismonsewicz
  • 25,132
  • 33
  • 116
  • 189
13
votes
2 answers

How to connect Kafka with Elasticsearch?

I am new in Kafka, I use kafka to collect netflow through logstash(it is ok), and I want to send the data to elasticsearch from kafka, but there are some problems. My question is how can I connect Kafka with Elasticsearch? netflow to kafka logstash…
張皓翔
  • 341
  • 1
  • 4
  • 16
13
votes
1 answer

How to use from_json with Kafka connect 0.10 and Spark Structured Streaming?

I was trying to reproduce the example from [Databricks][1] and apply it to the new connector to Kafka and spark structured streaming however I cannot parse the JSON correctly using the out-of-the-box methods in Spark... note: the topic is written…
carlos rodrigues
  • 317
  • 5
  • 12
12
votes
1 answer

Kafka Producer cannot validate record wihout PK and return InvalidRecordException

I have error with my kafka producer. I use Debezium Kafka connectors V1.1.0 Final and Kafka 2.4.1 . For tables with pk, all tables are flushed clearly, but unfortunately for tables with no pk on it, it give me this error: [2020-04-14 10:00:00,096]…
YVS1997
  • 682
  • 1
  • 7
  • 19
12
votes
3 answers

Kafka Connect with Amazon MSK

How do I use Kafka Connect adapters with Amazon MSK? As per the AWS documentation, it supports Kafka connect but not documented about how to setup adapters and use it.
Mrigank Kishore Varma
  • 376
  • 1
  • 3
  • 12
12
votes
1 answer

Kafka Connect vs Streams for Sinks

I am trying to understand what Connect buys you that Streams does not. We have a part of our application where we want to consume a topic and write to mariadb. I could accomplish this with a simple processor. Read the record, store in state store…
Chris
  • 1,299
  • 3
  • 18
  • 34
12
votes
4 answers

Securing access to REST API of Kafka Connect

The REST API for Kafka Connect is not secured and authenticated. Since its not authenticated, the configuration for a connector or Tasks are easily accessible by anyone. Since these configurations may contain about how to access the Source System…
Prabha
  • 121
  • 1
  • 3
12
votes
1 answer

Delete events from JDBC Kafka Connect Source

I am playing around with the Kafka Connect JDBC connector and specifically looking at what the actual format of the data that is put onto the topic is. I have been able to see new inserts and updates to the database, but I have not been able to…
ebensing
  • 6,409
  • 4
  • 18
  • 20
11
votes
1 answer

Kafka Dependencies - ccs vs ce

To develop my Kafka connector I need to add a connect-API dependency. Which one I should use? For example mongodb connector use connect-api from maven central But links from dev guide go to…
ortex
  • 719
  • 6
  • 15
11
votes
2 answers

Kafka connect cluster setup or launching connect workers

I am going through kafka connect, and i am trying to get the concepts. Let us say I have kafka cluster (nodes k1, k2 and k3) setup and it is running, now i want to run kafka connect workers in different nodes say c1 and c2 in distributed mode. Few…
Kalaiselvam M
  • 1,050
  • 1
  • 16
  • 25
1
2
3
…
99 100