Stack Overflow
Stack Overflow

Questions tagged [apache-kafka-connect]

Apache Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other data systems.

Apache Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other data systems.

It was first released with Kafka 0.9. It allows to import data from external systems (e.g., databases) into Kafka and it allows to export data from Kafka into external system (e.g., Hadoop). Apache Kafka Connect is a framework that supports a plug-in mechanism allowing to provide custom connectors for your system of choice.

Documentation

3693 questions
1
vote
1 answer

Is it possible to specify a kafka topic in a Kafka Connect config?

Right now my Connect config writes its messages to a topic by the same name as the database table it's reading. Is it possible to specify the exact topic I want to write to instead?
user377628
1
vote
1 answer

How do I move avro files to kafka via a kafka connector?

I found the CSV Source Connector which can monitor a directory for files and reads them as CSVs. Is there a Avro Source Connector for avro files? if it is not, Any recommendation for dealing with reading AVRO files to Kafka via Kafka connect?
han
  • 103
  • 7
1
vote
1 answer

java.lang.IllegalArgumentException: A KCQL error occurred.FIELD_ID is not a valid field name

I have data stored into a kafka topic and i want to move to cassandra using the CassandraSinkConnector from landoop. I am having this error while i am trying to start the connector: Caused by: java.lang.IllegalArgumentException: A KCQL error…
jacktrade
  • 3,125
  • 2
  • 36
  • 50
1
vote
2 answers

kafka connect failed to start with 'Uncaught exception in herder work thread.... position could be determined''

I am using Kafka connect Version confluentinc/cp-kafka-connect:5.1.1-1. And kafka cluster kafka_2.11-0.11.0.3 (3 brokers) This kafka cluster is working fine with old producer / consumer - using spark-stream. Now I tried to add kafka connect and I…
Ehud Lev
  • 2,461
  • 26
  • 38
1
vote
1 answer

How to tag S3 bucket objects using Kafka connect s3 sink connector

Is there any way we can tag the objects written in S3 buckets through the Kafka Connect S3 sink connector. I am reading messages from Kafka and writing the avro files in S3 bucket using S3 sink connector. When the files are written in S3 bucket I…
user2531535
  • 11
  • 1
1
vote
1 answer

how to create multiple topics and multiple table in kafka in jdbc connector?

i tried to install kafka connect in command prompt . i have a connector look like this…
Jr Dones
  • 11
  • 4
1
vote
1 answer

Can't start Kafka Connect: Timeout expired while fetching topic metadata

Trying to run Kafka Connect for the first time, with an existing Kafka deployment. using SASL_PLAINTEXT and kerberos authentication. The first time I try and start connect-distributed, I see: ERROR Uncaught exception in herder work thread, exiting: …
UtterlyConfused
  • 983
  • 1
  • 10
  • 18
1
vote
2 answers

Kafka connect JDBC source connector not working

Hello Everyone, I am using Kafka JDBC Source connector using for postgres. Following is my connector configuration. Some how it is not bringing any data. What is wrong in this configuration? { "name": "test-connection", …
RajData
  • 117
  • 1
  • 11
1
vote
1 answer

Debezium Change Data Capture (CDC) not working on sql-server 2017

After following the instructions listed here debezium sqlserver connector and how to activate change data capture And also making sure that the SQL-Agent is running, debezium is still not working ( streaming data to Kafka).
theeGwandaru
  • 400
  • 2
  • 14
1
vote
0 answers

Couchbase kafka source connector order of messages

My question is regarding the ordering of messages from CouchBase to Kafka topic. This is not well documented in my opinion and hence this question. Use case The use case is to get the changes for every document and get the latest change for every…
Manikandan Kannan
  • 8,684
  • 15
  • 44
  • 65
1
vote
2 answers

Is there a way to configure kafka-connect jmx metrics to be captured using a jmx_exporter/prometheus?

I'm setting up monitoring for Kafka connect in our Kafka ecosystem. I have enabled JMX exporter for kafka brokers and is working fine. Now I am trying to enable JMX exporter for kafka connect. However, it is a bit unclear where to start. I can only…
floating_hammer
  • 409
  • 3
  • 10
1
vote
1 answer

Getting below exception while creating key out of one field coming in "value" part of payload

SMT is not working with below configuration, any idea "transforms": "createKey", "transforms.createKey.type":"org.apache.kafka.connect.transforms.ValueToKey", "transforms.createKey.fields":"column_id", getting below exception Exception: …
Girish
  • 1,717
  • 1
  • 18
  • 30
1
vote
1 answer

Kafka Connect FileStreamSink connector removes quotation marks and changes colon to equal sign for JSON message

Summary When I stream this with the console producer {"id":1337,"status":"example_topic_1 success"} I get this in from my filestream consumer /data/example_topic_1.txt {id=1337, status=example_topic_1 success} This is a major problem for me,…
Cameron Hudson
  • 3,190
  • 1
  • 26
  • 38
1
vote
1 answer

How to get data from Kafka into a store without Kafka Connect sink?

When reading about Kafka and how to get data from Kafka to a queryable database suited for some specific task, there is usually mention of Kafka Connect sinks. This sounds like the way to go if I needed Kafka to search indexing like ElasticSearch…
atkayla
  • 8,143
  • 17
  • 72
  • 132
1
vote
0 answers

How to solve "java.net.SocketException: Connection reset" when running confluent-hub install

when i running this command /root/heraldhealth/softwares/confluent-5.2.1/bin/confluent-hub install --no-prompt confluentinc/kafka-connect-datagen:latest on my centos7 machine , i am getting this error below : Running in a "--no-prompt" mode …
user2734915
  • 85
  • 1
  • 10
1 2 3
…
99 100