Stack Overflow
Stack Overflow

Questions tagged [apache-kafka-connect]

Apache Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other data systems.

Apache Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other data systems.

It was first released with Kafka 0.9. It allows to import data from external systems (e.g., databases) into Kafka and it allows to export data from Kafka into external system (e.g., Hadoop). Apache Kafka Connect is a framework that supports a plug-in mechanism allowing to provide custom connectors for your system of choice.

Documentation

3693 questions
1
vote
1 answer

Kafka manual ackMode MANUAL_IMMEDIATE what if not acknowledge

I use Spring KafKa anf I set ackMode to MANUAL_IMMEDIATE props.setAckMode(AbstractMessageListenerContainer.AckMode.MANUAL_IMMEDIATE); the scenario is that for some reason my app could not acknowledge (acknowledgment.acknowledge()) and just miss it…
mohsenJsh
  • 2,048
  • 3
  • 25
  • 49
1
vote
2 answers

Kafka JDBC source connector time stamp mode failing for sqlite3

I tried to set up a database with two tables in sqlite. Once of my table is having a timestamp column . I am trying to implement timestamp mode to capture incremental changes in the DB. Kafka connect is failing with the below error: ERROR Failed to…
D Developer
  • 143
  • 1
  • 10
1
vote
1 answer

Cassandra WriteTimeOut Exception at ConsistencyLevel.LOCAL_QUORUM

I am facing WriteTimeOutException while writing to Cassandra using CassandraSinkConnector at ConsistencyLevel.LOCAL_QUORUM. I am using Cassandra 3.3.0 version So i need help how to handle and fix this exception? Stack-Trace:- WorkerSinkTask due to…
Naseem Ahmad
  • 9
  • 4
1
vote
1 answer

Purpose of +tmp in Kafka hdfs connect

I am planning to use Kafka hdfs connect for moving messages from Kafka to hdfs. While looking into it, I see there are parameters like flush size and rotate interval Ms with which you can batch messages in heap and write batch at once. Is the batch…
Athrey
  • 13
  • 5
1
vote
0 answers

cassandra source connector and kafka - what if u have just bigint epoch timestamp?

kafka-connect: kafka-connect-cassandra-1.2.1-2.1.0-all.jar My data inside cassandra are timestamped with epoch time saved as bigint. I want to retrieve those data into kafka topic using incremental mode inside kafka connect. According to spec - I…
Krzysztof Czarnomski
  • 11
  • 1
1
vote
0 answers

Sink connector not loading data into Postgres existing table?

I am trying to push messages from kafka topic (data is in avro format) to Postgres table. I have all privilege to create/insert/update/delete database/table. first time i ran the sink connector, it create a table automatically and loaded all data…
P Singh
  • 51
  • 7
1
vote
1 answer

Modify connector config in KafkaConnect before sending to task

I'm writing a SinkConnector in Kafka Connect and hitting an issue. This connector has a configuration as such : { "connector.class" : "a.b.ExampleFileSinkConnector", "tasks.max" : '1', "topics" : "mytopic", "maxFileSize" : "50" } I…
Marc Kossa
  • 21
  • 3
1
vote
3 answers

Configure debezium to put data on separate topics based on custom criteria

I am migrating to kafka as broker and debezium to get data (ETL data) from all of the micro-services to reporting and search databases. Is there any way to configure debezium so that it puts data on separate topics based on custom criteria (like…
Zeeshan Bilal
  • 1,147
  • 1
  • 8
  • 22
1
vote
1 answer

How to overcome the IllegalAccessError while start up of connector in Kafka

I am writing a connector for Kafka Connect. The error I see during the start up of connector is java.lang.IllegalAccessError: tried to access field org.apache.kafka.common.config.ConfigTransformer.DEFAULT_PATTERN from class…
Dilip Murupala
  • 61
  • 4
1
vote
1 answer

Message order with Kafka Connect Elasticsearch Connector

We are having problems enforcing the order in which messages from a Kafka topic are sent to Elasticsearch using the Kafka Connect Elasticsearch Connector. In the topic the messages are in the right order with the correct offsets, but if there are…
Yoni Gibbs
  • 6,518
  • 2
  • 24
  • 37
1
vote
1 answer

kafka-connect-jdbc does not fetch consecutive timestamp from source

I use kafka-connect-jdbc-4.0.0.jar and postgresql-9.4-1206-jdbc41.jar configuration of connector of kafka connect { "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector", "mode": "timestamp", "timestamp.column.name":…
GeoPamplona
  • 31
  • 7
1
vote
0 answers

Issue with Oracle JDBC Source Connector

We have Oracle Source from there need to get data, facing error in Avro and Json format. Connector File { "name": "LITERAL_VALUES", "config": { "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector", "key.serializer":…
Varun Bajaj
  • 1,033
  • 8
  • 16
1
vote
1 answer

Kafka streams vs Kafka connect for Kafka HBase ETL pipeline

I have straightforward scenario for the ETL job: take data from Kafka topic and put it to HBase table. In the future i'm going to add the support for some logic after reading data from a topic. I consider two scenario: use Kafka Streams for reading…
pacman
  • 797
  • 10
  • 28
1
vote
1 answer

Kafka Connect CDC to MSSQL "Table must have at least one primary key column"

I'm using Kafka Connect with MSSQL connector with a CDC enabled database and table. When creating the task with the REST API I receive the following error: ERROR Exception thrown while querying for ChangeKey{databaseName=WebAppDb, schemaName=dbo,…
Alessandro Dionisi
  • 2,494
  • 4
  • 33
  • 37
1
vote
1 answer

Kafka Connect - GET connector API exposes password in cleartext

I am using the latest confluent images (5.1.0) and externalized OracleDB passwords for Connect configurations in vault. I am able to successfully register the custom config provider for vault with following configuration. "connection.password":…
rasmuka
  • 31
  • 1
1 2 3
…
99 100