Stack Overflow
Stack Overflow

Questions tagged [apache-kafka-connect]

Apache Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other data systems.

Apache Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other data systems.

It was first released with Kafka 0.9. It allows to import data from external systems (e.g., databases) into Kafka and it allows to export data from Kafka into external system (e.g., Hadoop). Apache Kafka Connect is a framework that supports a plug-in mechanism allowing to provide custom connectors for your system of choice.

Documentation

3693 questions
5
votes
1 answer

Partition By Multiple Nested Fields in Kafka Connect HDFS Sink

We are running kafka hdfs sink connector(version 5.2.1) and needs HDFS data to be partitioned by multiple nested fields.The data in topics is stored as Avro and has nested elements.How ever connect cannot recognize the nested fields and throws an…
rookie
  • 386
  • 6
  • 19
5
votes
1 answer

Kafka Connect sink tasks ignore tolerance limits

I try to ignore bad messages in sink connector with errors.tolerance: all option. Full connector configuration: { "name": "crm_data-sink_pandora", "config": { "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector", …
Iskuskov Alexander
  • 4,077
  • 3
  • 23
  • 38
5
votes
1 answer

Kafka Connect, get Json Schema for JsonConverter

I'm trying to set up kafka-connector with a custom value converter. I'm using kafka to transfer serialised thrift objects. I want to set up a kafka-connector which is deserialising thrift-messages, converting them to json and sending to…
Viacheslav Shalamov
  • 4,149
  • 6
  • 44
  • 66
5
votes
2 answers

Authentication failed when connected to mongodb by debezium mongodb connector

Deployed the Strimzi Kafka, Strimzi Zookeeper and Debezium mongodb connector, and configured the Debezium mongodb, curl 'http://my-connect-cluster-connect-api:8083/connectors' -X POST -i -H "Content-Type:application/json" -d '{ "name":…
user3661933
  • 91
  • 3
  • 7
5
votes
1 answer

Apache kafka production cluster setup problems

We have been trying to set up a production level Kafka cluster in AWS Linux machines and till now we have been unsuccessful. Kafka version: 2.1.0 Machines: 5 r5.xlarge machines for 5 Kafka brokers. 3 t2.medium zookeeper nodes 1 t2.medium node for…
Ankur rana
  • 580
  • 10
  • 27
5
votes
4 answers

kafka-connect returning 409 in distributed mode

I'm running a kafka-connect distributed setup. I was testing with a single machine/process setup (still in distributed mode), which worked fine, now I'm working with 3 nodes (and 3 connect processes), logs do not contain errors, but when I submit an…
OmriManor
  • 255
  • 4
  • 14
5
votes
3 answers

Debezium-contains no connector type

I am trying to use Debezium to connect to a mysql database on my local machine. Trying with the following command to call kafka: sudo kafka/bin/connect-standalone.shsh kafka/config/connect-standalone.properties kafka/config/connector.properties Here…
Pravudutta Patra
  • 51
  • 3
5
votes
2 answers

Kafka Connect REST Interface "PUT /connectors/(string: name)/config" Return Error Code 500

I have a 3-nodes kafka-connect worker cluster in distributed mode, with a running s3 sink connector. To update the configuration of the connector at run-time, I run the command below: curl -X PUT -H "Content-Type: application/json" --data…
Double Infinity
  • 245
  • 2
  • 10
5
votes
1 answer

What's the key differences in existent approaches to mirror Kafka topics

Kafka MirrorMaker is a basic approach to mirror Kafka topics from source to target brokers. Unfortunately, it doesn't fit my requirements to be configurable enough. My requirements are very simple: the solution should be JVM application if…
yevtsy
  • 564
  • 1
  • 6
  • 18
5
votes
1 answer

How to configure Confluent Platform Kafka connect logs?

I am using confluent kafka connect service but it is not writing logs in /var/log/kafka . How to configure it so that it writes the logs in /var/log/kafka ? Currently /var/log/kafka only has following log files - -rw-r--r-- 1 cp-kafka confluent …
Tony
  • 671
  • 1
  • 9
  • 29
5
votes
2 answers

Kafka connect property relation between partition.duration.ms and flush size?

Can some one explain the significance of partition.duration.ms and flushsize in below configuration. What should be the thought behind setting up these properties? "connector.class": "io.confluent.connect.s3.S3SinkConnector", "s3.region":…
Raghav salotra
  • 820
  • 1
  • 11
  • 23
5
votes
1 answer

Kafka Connect : "Task already exists in this worker"

I see the status of my connector and i have this exception org.apache.kafka.connect.errors.ConnectException: Task already exists in this worker: *name of the connector* at org.apache.kafka.connect.runtime.Worker.startTask(Worker.java:365) …
Francesco
  • 129
  • 1
  • 10
5
votes
2 answers

Kafka connect HDFS sink ERROR failed creating a WAL

I'm using Kafka connect HDFS. When I'm trying to run my connector I'm got the following exception: ERROR Failed creating a WAL Writer: Failed to create file[/path/log] for [DFSClient_NONMAPREDUCE_208312334_41] for client [IP] because this file is…
Ya Ko
  • 509
  • 2
  • 4
  • 19
5
votes
2 answers

No suitable driver found for jdbc:mysql in Kafka Connect

connect-standalone.properties connector.class=io.confluent.connect.jdbc.JdbcSourceConnector bootstrap.servers=10.33.62.20:9092,10.33.62.110:9092,10.33.62.200:9092 key.converter=org.apache.kafka.connect.json.JsonConverter value.converter=org.apache.ka…
mukul
  • 433
  • 7
  • 18
5
votes
1 answer

Use message key in Kafka connect source connector

I'm using the Kafka connect JDBC source connector to read from a view in a database and post it on kafka, it is working fine. My use case is that a user can create multiple objects and the order of the objects is important in my application. I would…
Alejandro Agapito Bautista
  • 1,344
  • 1
  • 15
  • 35
1 2 3
…
99 100