Stack Overflow
Stack Overflow

Questions tagged [apache-kafka-connect]

Apache Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other data systems.

Apache Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other data systems.

It was first released with Kafka 0.9. It allows to import data from external systems (e.g., databases) into Kafka and it allows to export data from Kafka into external system (e.g., Hadoop). Apache Kafka Connect is a framework that supports a plug-in mechanism allowing to provide custom connectors for your system of choice.

Documentation

3693 questions
1
vote
1 answer

Kafka Connect and Kafka Broker version compatibility

We have a "Kerberized Kafka cluster" running brokers version Apache Kafka 0.11.This cluster is managed by a different team and we dont have any control over this. We are now trying to install Kafka Connect cluster on our own K8S cluster. We were…
Ashika Umanga Umagiliya
  • 8,988
  • 28
  • 102
  • 185
1
vote
1 answer

How to create the JDBC sink connector with multiple topic using topic regex option

Created a JDBC source connector catalog.pattern = test_01 source connector configuration { "name": "jdbcsource", "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector", "connection.url":…
hepzi
  • 435
  • 4
  • 17
1
vote
1 answer

Is there a equivalent Debezium command to starting Kafka Connect without Docker container

The debezium kafka connect command is : docker run -it --rm --name connect -p 8083:8083 -e GROUP_ID=1 -e CONFIG_STORAGE_TOPIC=my_connect_configs -e OFFSET_STORAGE_TOPIC=my_connect_offsets -e STATUS_STORAGE_TOPIC=my_connect_statuses --link…
Anand Swaminathan
  • 83
  • 10
1
vote
2 answers

Building a new SourceRecord from an Object

I am writing a Kafka connector in order to download some data from several sources on Github (text and yaml files) and transform them into objects of a certain class, which is automatically generated from an avsc-file: { "type": "record", …
biomartin
  • 87
  • 11
1
vote
0 answers

When trying to connect to Salesforce Case Push Topic using Salesforce Kafka Connector getting Error

When object is set as 'Case' instead of the default 'Lead' in salesforce-example.properties the connect stand alone gives the following error. ERROR WorkerSourceTask{id=salesforce-0} Task threw an uncaught and unrecoverable exception…
srk
  • 11
  • 2
1
vote
1 answer

Is it possible to use a file-source as input AND jdbc-sink as output with kafka?

I am currently working on a Kafka project and my issue is that I am able to read a file with the file-source Connector and store the data into a topic. My configuration…
Camille SAURY
  • 63
  • 5
1
vote
0 answers

Kafka-Connect - Unable to start jdbc source connector with MySql using Confluent Platform

I am using Confluent platform 5.2.1 and jdbc source connector to read data from MySql table. I have done all the required config changes and also the JDBC source connector and MySql driver jar's are placed in share/java/kafka-connect-jdbc directory.…
Sachchidanand Singh
  • 1,456
  • 1
  • 15
  • 15
1
vote
2 answers

JDBC confluent connector mode

I am using a custom query in JDBC kafka source connector can any one told me what is the mode at the time of using custom query in JDBC kafka source connector if i am using bulk mode then it will reinsert all data in kafka topic. note:-i didn't have…
santoXme
  • 802
  • 5
  • 20
1
vote
1 answer

Confluent - Splitting Avro messages from one kafka topic into multiple kafka topics

We have an incoming kafka topic with multiple Avro schema based messages serialized into it. We need to split the messages in Avro format into multiple other kafka topics based on certain value of a common schema attribute. …
TechEnthusiast
  • 1,795
  • 2
  • 17
  • 32
1
vote
1 answer

Install Custom Connector To Kafka Connect on Kubernetes

I'm running the kafka kubenetes helm deployment, however I am unsure about how to install a custom plugin. When running custom plugin on my local version of kafka I mount the volume /myplugin to the Docker image, and then set the plugin path…
Sam Palmer
  • 1,675
  • 1
  • 25
  • 45
1
vote
0 answers

How to perform masking with custom values or random values in apache kafka source connector and then unmasking the message in kafka sink connector?

{ transforms=SSNMask,IPMask,PhoneMask transforms.SSNMask.type=org.apache.kafka.connect.transforms.MaskField$Value transforms.SSNMask.fields=ssn } transforms should have some additional property to using custom values or random values instead of…
Vignesh waran
  • 13
  • 2
1
vote
1 answer

Getting exception ,Failed to get current time from DB using query values(CURRENT_TIMESTAMP) on database DB2

I am trying to use kafka connect jdbc , to source data from DB2 to kafka topic , but am trying to run the application , am seeing below error , which i think to execute configured connector query and that is the reason kafka topic is empty , below…
Bravo
  • 8,589
  • 14
  • 48
  • 85
1
vote
1 answer

Kafka Connect JDBC sink - store aggregated data from KSQL

I'm trying to store a result of ksql query (windowed table) to database using kafka connect. What I want to do now is (using kafka jdbc connect) store them in database but the rows being updated (as kafka table currently updates the values) instead…
user3212350
  • 401
  • 1
  • 6
  • 18
1
vote
0 answers

HDFS Sink Connector troubleshooting distributed mode

Kerberized Hadoop Cluster. Connect-Instance started on the edge node.In standalone mode, everything works as expected. In distributed mode, the connector is added based on the logs, but once checked with the REST call, no connectors are returned at…
LHA
  • 37
  • 5
1
vote
1 answer

Is "register" a reserved keyword in Ksql and if so how can I select a field with that name

I am in the process of learning the Confluent platform (Kafka, Ksql etc). I am streaming data into a Kafka topic using Debezium with Kafka Connect. One of the fields in the my database table "log" is called "register" which is a timestamp of when…
pr.lwd
  • 140
  • 10
1 2 3
…
99 100