Stack Overflow
Stack Overflow

Questions tagged [apache-kafka-connect]

Apache Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other data systems.

Apache Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other data systems.

It was first released with Kafka 0.9. It allows to import data from external systems (e.g., databases) into Kafka and it allows to export data from Kafka into external system (e.g., Hadoop). Apache Kafka Connect is a framework that supports a plug-in mechanism allowing to provide custom connectors for your system of choice.

Documentation

3693 questions
8
votes
2 answers

Get Kafka compressed message size

I would like to know the compressed size of a message in kafka. I use kafka 1.1.0 and java kafka-connect 1.1.0 to send messages from my producer to a topic. If the message is too large for my producer, I get a The message is xxx bytes when…
Marius
  • 365
  • 4
  • 18
8
votes
1 answer

Kafka SMT ValueToKey - How use multiple values as key?

I'm using the Confluent JDBCSourceConnector to read from an Oracle table. I am trying to use SMT to generate a key composed of 3 concatenated…
Alex
  • 45
  • 1
  • 7
8
votes
2 answers

Properly Configuring Kafka Connect S3 Sink TimeBasedPartitioner

I am trying to use the TimeBasedPartitioner of the Confluent S3 sink. Here is my config: { "name":"s3-sink", "config":{ "connector.class":"io.confluent.connect.s3.S3SinkConnector", "tasks.max":"1", "file":"test.sink.txt", …
Daniel
  • 1,522
  • 1
  • 12
  • 25
8
votes
1 answer

How to Process a kafka KStream and write to database directly instead of sending it another topic

I don't want to write processed KStream to another topic, I directly want to write enriched KStream to database. How should I proceed?
Megha
  • 188
  • 2
  • 10
8
votes
2 answers

Kafka Connect gets into a re balance loop

I've just deployed my Kafka Connect (I only use a connect source to MQTT) application on a cluster of two instances (2 containers on 2 machines) and now it seems to get into a sort of rebalancing loop,I've got a little bit of data at the…
user8363477
  • 655
  • 4
  • 14
  • 24
8
votes
1 answer

kafka connect hdfs sink connector is failing even when json data contains schema and payload field

I am trying kafka connect hdfs sink connector for moving json data from kafka to hdfs. Even when the json data in kafka has schema and payload kafka connect task is failing with error org.apache.kafka.connect.errors.DataException: JsonConverter with…
Deepak Kumar
  • 433
  • 4
  • 12
8
votes
4 answers

Kafka connect or Kafka Client

I need to fetch messages from Kafka topics and notify other systems via HTTP based APIs. That is, get message from topic, map to the 3rd party APIs and invoke them. I intend to write a Kafka Sink Connector for this. For this use case, is Kafka…
bhalochele
  • 227
  • 3
  • 11
7
votes
1 answer

Change-data-capture from Postgres SQL to kafka topics using standalone mode Kafka-connect

I have been trying to get data from postgres sql to kafka topics using the following command /bin connect-standalone.properties config/connect-standalone.properties postgres.sproperties, but am facing several issues with it here are the contents of…
Snehil Singh
  • 207
  • 3
  • 10
7
votes
0 answers

kafka avro console consumer does not deserialize DECIMAL correctly as decimal

I am using Confluent's kafka-connect-jdbc to read data from different RDBMS into kafka. Here is my test table: CREATE TABLE DFOCUSVW.T4( COL1 VARCHAR(100) NOT null, COL2 DECIMAL(6, 3) NOT null, COL3 NUMERIC(6, 3) NOT null, …
Behzad Pirvali
  • 764
  • 3
  • 10
  • 28
7
votes
2 answers

When is a Kafka connector preferred over a Spark streaming solution?

With Spark streaming, I can read Kafka messages and write data to different kind of tables, for example HBase, Hive and Kudu. But this can also be done by using Kafka connectors for these tables. My question is, in which situations I should prefer…
MetallicPriest
  • 29,191
  • 52
  • 200
  • 356
7
votes
2 answers

kafka s3 sink connector crashed when It gets NULL data

I had a working s3 sink connector until the source connector sent a NULL value; s3 connector crashed. The problem occured when I deleted a record from MS SQL db. The source connector shipped the deletion information to s3 connector and s3 connector…
Tireli Efe
  • 160
  • 2
  • 11
7
votes
2 answers

How to change the name of the topic generated by Kafka Connect Source Connector

I have an already running production deployed Kafka-Cluster and having Topic "existing-topic". I am using MongoDB-Source-Connector from Debezium. Here all what I want is to push the CDC events directly to the topic "existing-topic" so that my…
Abhinavece
  • 119
  • 1
  • 8
7
votes
2 answers

Kafka Connect date handling of debezium generated events

I'm using debezium SQL Server to track changes on a production base. The topic is created, CDC is working like a charm, but when trying to use jdbcSinkConnector to dump the data in another Sql Server DB, I'm encountering the following…
nicolasL
  • 136
  • 1
  • 7
7
votes
5 answers

Using a connector with Helm-installed Kafka/Confluent

I have installed Kafka on a local Minikube by using the Helm charts https://github.com/confluentinc/cp-helm-charts following these instructions https://docs.confluent.io/current/installation/installing_cp/cp-helm-charts/docs/index.html like so: helm…
Matthew Groves
  • 25,181
  • 9
  • 71
  • 121
7
votes
1 answer

Kafka Connect Transformation: Extract a Long value from json field and insert as key

I have the following json on a topic that the JDBC connector publishes to {"APP_SETTING_ID":9,"APP_SETTING_NAME":"my_name","SETTING_KEY":"my_setting_key"} Here's my connector…
Fizi
  • 1,749
  • 4
  • 29
  • 55
1 2 3
…
99 100