Stack Overflow
Stack Overflow

Questions tagged [apache-kafka-connect]

Apache Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other data systems.

Apache Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other data systems.

It was first released with Kafka 0.9. It allows to import data from external systems (e.g., databases) into Kafka and it allows to export data from Kafka into external system (e.g., Hadoop). Apache Kafka Connect is a framework that supports a plug-in mechanism allowing to provide custom connectors for your system of choice.

Documentation

3693 questions
1
vote
0 answers

Failed to find the class io.confluent.connect.gcs.GcsSinkConnector while starting kafka-gcs confluent connector

I am using confluent connector for kafka and Google Cloud Storage(GCS). I installed confluent using: (I am doing this on Ubuntu 16.04 and all these commands were executed in the home directory.) curl…
python_noob
  • 266
  • 3
  • 14
1
vote
2 answers

Kafka sink Error "This connector requires that records from Kafka contain the keys for the Cassandra table"

I am tring to sync all tables read from Sap into cassandra using kafka here is my cassandra config { "name": "cassandra", "config": { "connector.class": "io.confluent.connect.cassandra.CassandraSinkConnector", "tasks.max":…
Sano
  • 469
  • 2
  • 6
  • 21
1
vote
0 answers

Kafka Connect MySQL to Redshift - Dates Cause Error

I am getting this error when attempting to stream an RDS MySQL table into Redshift: Error converting data, invalid type for parameter The problem field is a DATETIME in MySQL and timestamp without time zone in Redshift (same happens for timestamp…
tom d
  • 111
  • 1
  • 7
1
vote
1 answer

Kafka connect with HDFS Sink connector error

I'm using the 1.1 version of Kafka with Kafka connect and I'm facing an error that I don't understand. I'm not able to connect to HDFS if I set somethink like that: hdfs.url": "/user/myuser/mydirectory" or like that (note that there's no…
Nastasia
  • 557
  • 3
  • 22
1
vote
0 answers

Securing sensitive information over Kafka Connect's /connectors//config API

When using Kafka Connect Api to post connectors or check their statuses, the password are visible in plain text. To hide this sensitive information, we went with the approach of Externalising Secrets mentioned by in the documentation This helped us…
gunj_desai
  • 782
  • 6
  • 19
1
vote
1 answer

Kafka connect with mongo oplog

I want to tail mongo oplog and stream it through Kafka.But there are many databases and collections, and I just want to get the update data for one of them. If you want to filter out the desired operation records from all the operation records in…
DotWait
  • 13
  • 2
1
vote
1 answer

How can I manage Kafka connect schema errors?

I'm using kafka connect (confluent distribution) to connect an mqtt broker to a kafka topic (https://docs.lenses.io/connectors/source/mqtt.html), but when a message arrives and it isn't conform to the expected schema, the connector stops! How can I…
user1387219
  • 465
  • 4
  • 18
1
vote
0 answers

Nifi processors(publish and consume kafka) are not communicating with kafka on Ambari platform

I want to establish a communication channel between nifi and kafka. Four processors in nifi(GenerateFlowFile->publishKafka) and (ConsumeKafka->logAttribute). PublishKafka and ConsumeKafka are not communicating with Apache kafka(pictures are…
M.WAQAS JAMIL
  • 39
  • 3
1
vote
1 answer

How to import MS Sql Server tables to KSQL with Kafka connect

Hi I am trying to import all tables present on remote SQL Server to KSQL topics this is my file…
Sano
  • 469
  • 2
  • 6
  • 21
1
vote
1 answer

Is there a way I can define a function to decide S3 path based on topic message in kafka connect

This question is related to Kafka to S3. Requirement: One of the kafka topics we are interested has some particular information, i.e, timestamp, table and etc. We can use this data to decide which S3 path it will go to, i.e.…
Xiaohe Dong
  • 4,953
  • 6
  • 24
  • 53
1
vote
1 answer

Kafka Connect serialization error in Elastic Sink

I am using kafka elasticsearch sink connector to pass an incoming message to ES but I am running into the following issue [2018-10-05 13:01:21,388] ERROR WorkerSinkTask{id=elasticsearch.sink.direct- 10} Task threw an uncaught and unrecoverable…
Fizi
  • 1,749
  • 4
  • 29
  • 55
1
vote
3 answers

Kafka Connect and Streams

So i very recently started reading about Kafka and I am a little confused about the difference between Kafka Connect and Kafka Streams. As per the definition Kafka Streams can collect data from Kafka topic, process it and push the output to another…
Anuja Barve
  • 300
  • 1
  • 4
  • 23
1
vote
1 answer

Kafka connect ElasticSearch sink - using if-else blocks to extract and transform fields for different topics

I have a kafka es sink properties file like the…
Fizi
  • 1,749
  • 4
  • 29
  • 55
1
vote
0 answers

kafka connect transformations

I want to stream the data from postgreSQL to HDFS but all of my tables have multiple timestamp columns. I had tried simple jdbc connect and HDFS sink offered by confluent to read data from postgres and write it to HDFS but all of my date columns are…
jarry jafery
  • 1,018
  • 1
  • 14
  • 25
1
vote
1 answer

Kafka Connect (Confluent 5.0, 4.1.2 or 3.0) not starting

we have a Kafka cluster (as a 3rd party hosted service), which has SSL enabled. We are now trying to setup Kafka Connect (Confluent 5.0) with a 3rd party Sink (WePay BigQuery connector). When starting Kafka connect in standalone mode, everything…
Patrick Plaatje
  • 99
  • 1
  • 9
1 2 3
…
99
100