Stack Overflow
Stack Overflow

Questions tagged [apache-kafka-connect]

Apache Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other data systems.

Apache Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other data systems.

It was first released with Kafka 0.9. It allows to import data from external systems (e.g., databases) into Kafka and it allows to export data from Kafka into external system (e.g., Hadoop). Apache Kafka Connect is a framework that supports a plug-in mechanism allowing to provide custom connectors for your system of choice.

Documentation

3693 questions
1
vote
2 answers

Confluent Kafka-connect-JDBC connector showing hexa decimal data in the kafka topic

I'm trying to copy the data from a table in the oracle db and trying to put that data in a kafka topic. I've used the following JDBC source connector for that…
kuti
  • 161
  • 1
  • 3
  • 13
1
vote
1 answer

confluent - kafka-connect - JDBC source connector - ORA-00933: SQL command not properly ended

I've the following sql query in my kafka jdbc source connector properties file : query=SELECT * FROM JENNY.WORKFLOW where ID = '565231' If I run the same query in sql developer, it works fine and fetching the results. But if I use the same query in…
kuti
  • 161
  • 1
  • 3
  • 13
1
vote
1 answer

Kafka Connect Dynamic connectors in Java

This is what the developer guide has to say about Dynamic connectors https://docs.confluent.io/current/connect/devguide.html#dynamic-connectors Not all connectors have a static set of partitions, so Connector implementations are also responsible…
David Breton
  • 169
  • 1
  • 10
1
vote
0 answers

Kafka Connect - Failed to flush, timed out / Failed to commit offsets

I am getting the following error: "ERROR WorkerSourceTask(id=test-mysql-dbc-source-0) Failed to flush, timed out while waiting for producer to flush outstanding N messages. ERROR Failed to commit offsets.…
Kiryl Varanovich
  • 61
  • 1
  • 5
1
vote
1 answer

Confluent Control Center not intercepting stream

I'm using CCC with a Kafka stream, which is populated by the Postgres Connector from Debezium. I'm using the following docker-compose.yml: version: '2' services: zookeeper-1: image: confluentinc/cp-zookeeper:latest hostname: zookeeper-1 …
oooyaya
  • 1,773
  • 1
  • 15
  • 39
1
vote
1 answer

Kafka AVRO Consumer: MySQL Decimal to Java Decimal

I'm trying to consume records from a MySQL table which contains 3 columns (Axis, Price, lastname) with their datatypes (int, decimal(14,4), varchar(50)) respectively. I inserted one record which has the following data (1, 5.0000, John). The…
Mahmoud Elbably
  • 139
  • 2
  • 10
1
vote
1 answer

Configure debezium mysql connector with kafka connect getting failed

I am using kafka 2.0 with kafka connect running in distributed mode and tried configure debezium mysql connector but getting error HTTP/1.1 500 Internal Server Error Server: squid Mime-Version: 1.0 Date: Wed, 12 Sep 2018 06:14:40 GMT Content-Type:…
shobhna
  • 13
  • 6
1
vote
1 answer

Kafka MQTT connector with multiple topics

We are using Kafka-Mqtt connector from Confluent team. Is there any way to publish data from multiple Mqtt topics to multiple Kafka topics with same name as of Mqtt , i.e test1 in Mqtt -> test1 in kafka so on. ?
Kartik D
  • 11
  • 3
1
vote
1 answer

Kafka connect many to many tables in MSSQL

I'm currently looking into Kafka Connect to stream some of our databases to a data lake. To test out Kafka Connect I've setup a database with one of our project databases in. So far so good. Next step I configured Kafka Connect with mode following…
Vince V.
  • 3,115
  • 3
  • 30
  • 45
1
vote
2 answers

Kafka Connect separated logging

Currently we are using a couple of custom connetor plugins for our confluent kafka connect distributed worker cluster. One thing that bothers me for a long time is that kafka connect writes all logs from all deployed connectors to one File/Stream.…
Stephan
  • 323
  • 2
  • 11
1
vote
1 answer

How to access remote database in Apache Kafka?

I am a beginner to kafka. here i am trying to build real time data pipeline in kafka. I have database which is located remotely. I really don't understand how to access database and database continuously updating like real time. I read about…
P Singh
  • 51
  • 7
1
vote
1 answer

Debezium Kafka connect. Decimal schema error

I'm using PostgreSQL. In my table numeric column defined like this: "net_weight" numeric(19,2) But in Kafka, I'm receiving instead of decimal numbers (E.g. 14.0) I'm getting something like "E4g=" versions: debezium/connect:0.7 kafka…
Andrii Fedorenko
  • 153
  • 2
  • 11
1
vote
2 answers

How to pull the data from remote database in Apache Kafka?

I want to make real-time data pipeline in Apache Kafka. I have database which is located at remote location and that database continuously updating. Can anybody which Kafka connect API i should use to pull the data from database and ingest into…
DP0808
  • 33
  • 1
  • 6
1
vote
1 answer

How Kafka Connectors are reliable in case of failures?

I'm thinking of using a Kafka Connector vs creating my own Kafka Consumers/Producers to move some data from/to Kafka, and I see the value Kafka Connectors provide in terms of scalability and fault tolerance. However, I haven't been able to find how…
user3739116
  • 124
  • 1
  • 10
1
vote
1 answer

Kafka stream enrichment - Sourcing a lookup table

There is a Kafka stream component that fetches JSON data from a topic. Now I have to do the following: Parse that input JSON data and fetch the value of a certain ID (identifier) attribute Do a lookup against a particular table in Oracle…
Marco99
  • 1,639
  • 1
  • 19
  • 32
1 2 3
…
99
100