Stack Overflow
Stack Overflow

Questions tagged [apache-kafka-connect]

Apache Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other data systems.

Apache Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other data systems.

It was first released with Kafka 0.9. It allows to import data from external systems (e.g., databases) into Kafka and it allows to export data from Kafka into external system (e.g., Hadoop). Apache Kafka Connect is a framework that supports a plug-in mechanism allowing to provide custom connectors for your system of choice.

Documentation

3693 questions
9
votes
2 answers

How to connect Apache Kafka with Amazon S3?

I want to store data from Kafka into a bucket s3 using Kafka Connect. I had already a Kafka's topic running and I had a bucket s3 created. My topic has data on Protobuffer, I tried with https://github.com/qubole/streamx and I obtained the next…
Eric Bellet
  • 1,732
  • 5
  • 22
  • 40
9
votes
2 answers

Add jar to general Kafka Connect classpath in the Confluent Docker

I am using the confluent docker for Kafka connect v4.1.1 and would like to add a jar with specific log4j appenders. With the classpath isolation by connector, I am not sure where to place that jar in the confluent docker, as it is used by the…
Cherif Naggar
  • 95
  • 1
  • 5
9
votes
2 answers

How to transform and extract fields in Kafka sink JDBC connector

I am using a 3rd party CDC tool that replicates data from a source database into Kafka topics. An example row is shown below: { "data":{ "USER_ID":{ "string":"1" }, "USER_CATEGORY":{ "string":"A" …
Giorgos Myrianthous
  • 36,235
  • 20
  • 134
  • 156
9
votes
1 answer

Kafka Connect with Spring Framework

Is anyone aware of any Spring Boot integration with Kafka Connect? There is a spring-kafka project that I think gives a nice integration with Kafka clients but not connect and streams APIs.
dmaity
  • 91
  • 1
  • 4
9
votes
1 answer

Put() vs Flush() in Kafka Connector Sink Task

I am trying to send the data in a batch to a NOSQL database using Kafka Sink Connector. I am following https://kafka.apache.org/documentation/#connect documentation and confused about where the logic of sending records has to be implemented. Please…
user2401547
  • 131
  • 1
  • 10
9
votes
1 answer

Setting Partition Strategy in a Kafka Connector

I am using a custom Kafka connector (written in Java, using Kafka Connect's Java API) to pull data from an outside source and store in a topic. I need to set a custom partitioning strategy. I understand that setting a custom partitioner is possible…
Raisin Bran Dan
  • 150
  • 1
  • 7
9
votes
2 answers

How to use Kafka Connect for Cassandra without Confluent

How we can use Kafka Connect with Cassandra without using the Confluent frameworks.
RAJ GUPTA
  • 428
  • 5
  • 7
8
votes
1 answer

What is the relationship between connectors and tasks in Kafka Connect?

We've been using Kafka Connect for a while on a project, currently entirely using only the Confluent Kafka Connect JDBC connector. I'm struggling to understand the role of 'tasks' in Kafka Connect, and specifically with this connector. I understand…
Andrew Ferrier
  • 16,664
  • 13
  • 47
  • 76
8
votes
1 answer

could kafka-connect distributed worker work with topic 'connect-offset' with configuration 'cleanup.policy=compact,delete'

I started a kafka-connect distributed worker cluster which uses the topic connect-offset for offset storage: offset.storage.topic=connect-offset Since the broker is provided with default policy 'cleanup.policy=delete', when creating a topic with…
shawnzhu
  • 7,233
  • 4
  • 35
  • 51
8
votes
0 answers

How to get rid of negative consumer lag in Kafka

Recently I monitor my cluster and I saw the negative consumer lag on it: This negative lag affects the sink connectors on the cluster and I got this error on the Kafka-Connect: Mar 25 15:56:30 Master1 connect-distributed[14530]: [2020-03-25…
Hossein Torabi
  • 694
  • 1
  • 7
  • 18
8
votes
4 answers

Kafka Connect: No tasks created for a connector

We are running Kafka Connect (Confluent Platform 5.4, ie. Kafka 2.4) in a distributed mode using Debezium (MongoDB) and Confluent S3 connectors. When adding a new connector via the REST API the connector is created in RUNNING state, but no tasks are…
user2732824
  • 141
  • 1
  • 5
8
votes
2 answers

Can we update/Upsert a record in mongodb? data source is kafka

We can update/upsert the record in mongodb BUT is there is any method or function from which we can update or upsert the document directly in mongodb and the source system is kafka and destination is mongodb.
nixxo_raa
  • 391
  • 8
  • 21
8
votes
1 answer

Kafka JDBC Sink Connector: no tasks assigned

I try to start JDBC sink connector with following configuration: { "name": "crm_data-sink_hh", "config": { "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector", "tasks.max": 6, "topics":…
Iskuskov Alexander
  • 4,077
  • 3
  • 23
  • 38
8
votes
1 answer

Kafka Connect | Cannot complete request because of a conflicting operation

1) We have 3 node kafka & kafka connect cluster 2) We are running kafka-connect on kafka nodes only in distributed mode 3) When i am trying to create a connector using below configuration : { "name": "connector-state-0", "config":…
Sahil Gupta
  • 121
  • 1
  • 7
8
votes
1 answer

Reading into SQL Server from Kafka feed

I need to read from a Kafka feed into a SQL Server database and i'm struggling to find any detail on how to do it. I realise it may be simple, but I've searched and found only stuff about feeding into Kafka from SQL
rosscv1
  • 89
  • 1
  • 2
1 2 3
…
99 100