Stack Overflow
Stack Overflow

Questions tagged [apache-kafka-connect]

Apache Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other data systems.

Apache Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other data systems.

It was first released with Kafka 0.9. It allows to import data from external systems (e.g., databases) into Kafka and it allows to export data from Kafka into external system (e.g., Hadoop). Apache Kafka Connect is a framework that supports a plug-in mechanism allowing to provide custom connectors for your system of choice.

Documentation

3693 questions
6
votes
4 answers

kafka connect exception, Replication factor: 3 larger than available brokers: 1

kafka connect 5.4 only one connector, one worker and use connect-distributed. Below is the error message: [2020-06-22 19:09:58,700] ERROR [Worker clientId=connect-1, groupId=test-cluster] Uncaught exception in herder work thread, exiting: …
GodBlessYou
  • 520
  • 1
  • 6
  • 17
6
votes
1 answer

Postgres Debezium does not publish the previous state of a record

I successfully installed Postgres Debezium CDC. Now, I'm able to catch all changes happening to the database. But the problem is "before" field always stays empty. So, if I insert a record (id = 1, name = Bill) I then get from Kafka this…
Jacobian
  • 10,122
  • 29
  • 128
  • 221
6
votes
1 answer

Automatically reconnect failed tasks in Kafka-Connect

I'm using a mongo-source plugin with Kafka-connect. I checked the source task state, and it was running and listening on a mongo collection. I manually stopped mongod service and waited about 1 minute, then I start it back again. I checked the…
toto
  • 1,197
  • 2
  • 15
  • 26
6
votes
0 answers

Kafka Connect Consumer Group Lag Metrics?

I have a Kafka Connect sink running. I'd like to monitor the lag for this. I can manually get the lag by shelling into a broker and using the kafka-consumer-groups tool like so: unset JMX_PORT; /usr/bin/kafka-consumer-groups --bootstrap-server…
clay
  • 18,138
  • 28
  • 107
  • 192
6
votes
3 answers

Is it possible to reset offsets to a topic for a kafka consumer group in a kafka connector?

My kafka sink connector reads from multiple topics (configured with 10 tasks) and processes upwards of 300 records from all topics. Based on the information held in each record, the connector may perform certain operations. Here is an example of the…
Jason Choi
  • 113
  • 1
  • 1
  • 7
6
votes
2 answers

How to transform all timestamp fields when using Kafka Connect?

I am trying to convert all timestamp fields to a string type with the format yyyy-MM-dd HH:mm:ss. To transform multiple fields, I have to create a transform for each one…
A. Saunders
  • 815
  • 1
  • 6
  • 19
6
votes
1 answer

Kafka connect with mysql custom query

I have done incremental data sync with help of kafka connect. Now i want to achieve same with custom query. But I am getting error. My config file is…
RKP
  • 750
  • 2
  • 12
  • 23
6
votes
2 answers

Kafka: make consumer group Inactive

I am running a data pipeline such that I read data from sql db into kafka topic through jdbc connect I sink this data in Elasticsearch using kafka sink connector for ES I have a need to reset this pipeline. To that end I want to reset the consumer…
Fizi
  • 1,749
  • 4
  • 29
  • 55
6
votes
2 answers

Unable to run a JDBC Source connector with Confluent REST API

I want to run a JDBC source connector using Kafka Connect REST API. Although stand-alone mode works perfect using the following properties file:…
Giorgos Myrianthous
  • 36,235
  • 20
  • 134
  • 156
6
votes
3 answers

Debezium flush timeout and OutOfMemoryError errors with MySQL

Using Debezium 0.7 to read from MySQL but getting flush timeout and OutOfMemoryError errors in the initial snapshot phase. Looking at the logs below it seems like the connector is trying to write too many messages in one…
Kamil Sindi
  • 21,782
  • 19
  • 96
  • 120
6
votes
2 answers

kafka connect multiple topics in sink connector properties

I am trying to read 2 kafka topics using Cassandra sink connector and insert into 2 Cassandra tables. How can I go about doing this? This is my connector.properties…
el323
  • 2,760
  • 10
  • 45
  • 80
6
votes
1 answer

How to acknowledge consume message in kafka using php-rdkafka?

I am using php-rdkafka as php kafka client. I successfully product my test message by using test group.and consume the message by using below code, $kafkaConsumer = new RdKafka\Consumer(); $kafkaConsumer->addBrokers("127.0.0.1:9292"); $topic =…
Keyur Shah
  • 11,043
  • 4
  • 29
  • 48
6
votes
6 answers

Kafka Connect Alerting Options?

Are there any alerting options for scenarios where a Kafka Connect Connector or a Connector task fails or experiences errors? We have Kafka Connect running, it runs well, but we've had errors that need to be manually traced and discovered. And…
clay
  • 18,138
  • 28
  • 107
  • 192
6
votes
2 answers

Restarting Kafka Connect S3 Sink Task Loses Position, Completely Rewrites everything

After restarting a Kafka Connect S3 sink task, it restarted writing all the way from the beginning of the topic and wrote duplicate copies of older records. In other words, Kafka Connect seemed to lose its place. So, I imagine that Kafka Connect…
clay
  • 18,138
  • 28
  • 107
  • 192
6
votes
3 answers

Parquet Output From Kafka Connect to S3

I see Kafka Connect can write to S3 in Avro or JSON formats. But there is no Parquet support. How hard would this be to add?
clay
  • 18,138
  • 28
  • 107
  • 192
1 2 3
…
99 100