Stack Overflow
Stack Overflow

Questions tagged [apache-kafka-connect]

Apache Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other data systems.

Apache Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other data systems.

It was first released with Kafka 0.9. It allows to import data from external systems (e.g., databases) into Kafka and it allows to export data from Kafka into external system (e.g., Hadoop). Apache Kafka Connect is a framework that supports a plug-in mechanism allowing to provide custom connectors for your system of choice.

Documentation

3693 questions
1
vote
2 answers

Ensure exactly once delivery in Kafka Connect

do You know is it possible, and if yes what is the best way to ensure exactly one delivery to hdfs using kafka connect with kafka? I know that Kafka connect attempt to find offsets for its consumer group in the "'__consumer_offsets" but I need…
Rafał Wójcik
  • 61
  • 1
  • 7
1
vote
0 answers

Use ByteArrayFormat with TimeBasedPartitioner that extracts using RecordField

I'm trying to use TimeBasedPartitioner that extracts using RecordField with the following configuration: { "name": "s3-sink", "connector.class": "io.confluent.connect.s3.S3SinkConnector", "tasks.max": "10", "topics":…
moku
  • 4,099
  • 5
  • 30
  • 52
1
vote
1 answer

kafka-connect : Getting error in distributed configuration for connector sink cassandra

I get task error for a distributed configuration for a connector sink cassandra. I was running the command : curl -s localhost:8083/connectors/cassandraSinkConnector2/status | jq to get the status { "name": "cassandraSinkConnector2", …
BOUZIANE Mohamed Eldjilali
  • 33
  • 6
1
vote
0 answers

I am getting an error Kafka Connect HDFS connector and no data is written to HDFS

The Kafka connector is starting up OK and reads the data stream in the topic, however, it doesn't write any data and when I stop the connector i get an error I have ensured the topic and logs directory is created on the HDFS…
Richard Stagg
  • 63
  • 6
1
vote
1 answer

How to I parse tuple value as an Person Object?

I have implemented a Logger Bolt in storm, the input of the tuple is coming from Kafka Topic. I am using Kafka Connect to listen to changes to mySQL database. public class LoggerBolt extends BaseBasicBolt { private static final long…
Amar Kumar
  • 57
  • 8
1
vote
1 answer

Can a KafkaConsumer read data from topic written by SourceTask?

If we have written into a Kafka topic data using a SourceTask, in which we have defined a Schema for SourceRecords different than Schema.STRING_SCHEMA, that is a custom schema for example: private final Schema valueSchema =…
Novemberland
  • 530
  • 3
  • 8
  • 25
1
vote
2 answers

How to get the table-name and database-name in the CDC event received from debezium kafka connect

Setup: I have CDC enabled on MS SQL Server and the CDC events are fed to Kafka using debezium kafka connect(source). Also more than one table CDC events are routed to a single topic in Kafka. Question: Since I have more than one table data in the…
Philip
  • 13
  • 4
1
vote
1 answer

kafka connect for jdbc sink for microsoft sql server. it works for multiple keys for record_value and this error is poping up for record_key

I was using jdbc sink driver from kafka connect. it allows create table with one primary key when I try to add the 2 pk.key fields . it gives me error: java.lang.NullPointerException at…
Kumar Pinumalla
  • 51
  • 3
1
vote
1 answer

table.whitelist is working as case sensitive even after specifying quote.sql.identifiers=NEVER

I have used JDBC source connector to ingest data from oracle to kafka topics. I have kafka topics created in small letters so I have to specify table.whitelist=table_name (in small case). Since by default it takes everything in quotes so I have…
user147296
  • 51
  • 4
1
vote
1 answer

issue using incrementing ingest in jdbc connector

I'm trying to use an incrementing ingest to produce a message to a topic on update of a table in mysql. It works using timestamp but doesn't seem to be working using incrementing column mode. When I insert a new row into the table, I do not see any…
user3310115
  • 1,372
  • 2
  • 18
  • 48
1
vote
1 answer

Kafka Connect ignoring the Subject Strategies specified

I want to publish multiple table data on to same Kafka topic using the below connector config, but I am seeing below exception Exception Caused by: io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: Schema being registered…
Suraj
  • 97
  • 5
1
vote
1 answer

Kafka Connect S3 Dynamic S3 Folder Structure Creation?

I have manually installed Confluent Kafka Connect S3 using the standalone method and not through Confluent's process or as part of the whole platform. I can successfully launch the connector from the command line with the…
Liquidgenius
  • 639
  • 5
  • 17
  • 32
1
vote
1 answer

How get the stream from kafka topic to elasticsearch with confluent?

I'm read data from machine and stream it as JSON to a kafka topic. I would like to read this topic and store the streamdata into elasticsearch with confluent. My steps: 1. Create KSQL Streams to convert from JSON to AVRO json stream: CREATE STREAM…
ofitz
  • 152
  • 1
  • 2
  • 14
1
vote
1 answer

org.apache.kafka.connect.errors.DataException: Struct schemas do not match

Trying to create and populate an array schema of type string inside an array schema of particular object type using kafka.connect.data.Schema in java. But getting error org.apache.kafka.connect.errors.DataException: Struct schemas do not match. …
Aditya
  • 13
  • 4
1
vote
0 answers

Errors/Issues using different converters in kafka-connect to S3

I have been trying to implement the confluent kafka-connect image to connect or our on prem S3. We have successfully written to s3 from the box using Boto3. So we know it is not a connection issue. Depending on what converters I use..they produce…
RHammonds
  • 77
  • 1
  • 1
  • 8
1 2 3
…
99 100