Questions tagged [apache-kafka-connect]

Apache Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other data systems.

Apache Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other data systems.

It was first released with Kafka 0.9. It allows to import data from external systems (e.g., databases) into Kafka and it allows to export data from Kafka into external system (e.g., Hadoop). Apache Kafka Connect is a framework that supports a plug-in mechanism allowing to provide custom connectors for your system of choice.

Documentation

3693 questions
1
vote
1 answer

Kafka topic seems to function first time only. Why?

I am working with Kafka Connect (using the Confluent implementation) and am seeing a strange behavior. I configure a source connection to pull data from a DB table, and populate a topic. This works. But, if I delete the topic, remove the Source…
M. Ferris
  • 341
  • 3
  • 13
1
vote
2 answers

Kafka connect - string cannot be casted to struct

I am doing poc of confluent kafka connect version 5.2.3. We are trying to copy message of topic a file as backup and from this file back to topic when we need it. Topic has Key =string Value=protbuf I am…
Abhishek
  • 519
  • 1
  • 6
  • 24
1
vote
2 answers

Build a combined docker image for snowflake-kafka-connector with cp-kafka-connect-base to deploy on kafka connect cluster

(Submitting on behalf of a client) ......................... We are using kafka connect cluster on kubernetes (helm charts). However to install plugin it is recommended to extend the image provided "cp-kafka-connect-base" with the local connector.…
1
vote
1 answer

Invalid snowflake.topic2table.map

Can someone tell me why my snowflake.topic2table.map value is being reported as invalid by my Snowflake Sink Connector? The KAFKA_TEST database has been created in Snowflake and I am ok to let the sink connector create the table for me. When I…
Joe P
  • 499
  • 1
  • 4
  • 11
1
vote
0 answers

Translate a Kafka ConnectRecord into an Avro GenericRecord

I have some code that is receiving fully formed Kafka ConnectRecords. I would like to take those records, and convert them to Avro's GenericRecord. I imagine a utility must exist to do this. I want to do something really simple: ConnectRecord…
Pablo
  • 10,425
  • 1
  • 44
  • 67
1
vote
1 answer

Kafka-connect topic.prefix without the table name

I'm using jdbc source connector, my table names have special chars (ie.$) that are acceptable to the DB engine but when I run kafka-connect with below configuration, it attempts to create the kafka topic with this prefix, plus the table name but…
Sahas
  • 3,046
  • 6
  • 32
  • 53
1
vote
2 answers

kafka connect and kafka host requirements

I am using Couchbase sink connector. CB and kafka are in 2 different EC2 instances, in different AWS regions. I am following these…
uzumas
  • 632
  • 1
  • 8
  • 23
1
vote
1 answer

Does confluent cloud allow custom connector deployment

Confluent cloud supports only following connector. GCS Sink Connector S3 Sink Connector Please validate If my understanding is correct or not. Can we deploy custom connector or cdc like debezium in confluent cloud.
1
vote
0 answers

Is it possible for a Kafka Connect connector to be configured to skip rows with specific values?

I have a sink connector that writes rows to a MySQL database. I'd like to skip rows which have a "source": "whatever" key: value pair. Is this possible?
user377628
1
vote
2 answers

Write kafka topic data to redis using docker

I am using this repository kafka connect to redis. Explain: What I want to do is to write kafka topics data into redis using docker. They have created a readme file to instruct how to set the configuration of the kafka: curl -s -X POST -H…
Mostafa Ghadimi
  • 5,883
  • 8
  • 64
  • 102
1
vote
0 answers

HDFS Kafka Connect - Hive integration create table exception

I am trying to sink data in to hdfs and that works fine but while configuring hive to see the data, I am getting error related to URI. I have already tried to use store.url in place of hdfs.url but failed with null pointer exception. My…
1
vote
1 answer

Kafka Connector - distributed - load balancing tasks

I am running development environment for Confluent Kafka, Community edition on Windows, version 3.0.1-2.11. I am trying to achieve load balancing of tasks between 2 instances of connector. I am running Kafka Zookepper, Server, REST services and 2…
Miki
  • 2,493
  • 2
  • 27
  • 39
1
vote
1 answer

userid is getting changed at the MQ side from kafka connect client

I'm using https://github.com/ibm-messaging/kafka-connect-mq-source project to build the connector to pull the data from IBM MQ to Kafka topic. I'm seeing an issue where the supplied used-id is getting modified to the system user and authentication…
kuti
  • 161
  • 1
  • 3
  • 13
1
vote
0 answers

Kafka-Connect JDBC Sink Connector and Avro Fails on Second insert

I'm trying to follow a basic JDBCSinkConnector example based on the tutorial from Confluent. the Sample uses MySql Database with an Avro Topic and a schema-registry. The sample works for the first message that arrives in the topic and inserts the…
1
vote
1 answer

How to create proper kafka-connect plugin without connector?

I try to create plugin with "transform" for my data to kafka-connect and use it with different sink connectors. When I install plugin, kafka-connect doesn't see my classes. I used kafka-connect maven plugin to create my bundle zip. Installation with…