Stack Overflow
Stack Overflow

Questions tagged [apache-kafka-connect]

Apache Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other data systems.

Apache Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other data systems.

It was first released with Kafka 0.9. It allows to import data from external systems (e.g., databases) into Kafka and it allows to export data from Kafka into external system (e.g., Hadoop). Apache Kafka Connect is a framework that supports a plug-in mechanism allowing to provide custom connectors for your system of choice.

Documentation

3693 questions
1
vote
1 answer

Error deserialising avro kafka message from JDBC Connector

I'm trying to listen to a topic to which I posted from using the kafka connect feature of confluent. However, I am not able to deserialize it. I believe that its avro serialisation but not able to find the right deserializer. the message is like…
user3310115
  • 1,372
  • 2
  • 18
  • 48
1
vote
0 answers

Kafka-connect elasticsearch how to define index mappings

I'm using kafka-connect-elasticsearch with a custom converter, which extends standard JsonConverter. I have 250+ topics with different event types, thus i'm happy that kafka-connect automatically creates indices for me in elasticsearch. However, I'd…
Viacheslav Shalamov
  • 4,149
  • 6
  • 44
  • 66
1
vote
1 answer

Does Kafka stand alone connector syncs automatically or it needs restart every time

I am new to Kafka. I have just configured one Kafka standalone connector by referring steps from confluent doc/guide. Job of Kafka connector is to sync data from file - test.txt. If I update anything in test.txt, it is not getting automatically…
RKP
  • 750
  • 2
  • 12
  • 23
1
vote
1 answer

What I have to do with SourceRecords in Kafka

After creating a Connector and a SourceTask objects in Kafka with Java I cannot figure out what can be done with SourceRecords returned form poll() methd of the SourceTask object. How can I push the record in the topic which is a parameter of the…
Novemberland
  • 530
  • 3
  • 8
  • 25
1
vote
0 answers

Kafka Connect Sink to S3: `AmazonS3Exception: We encountered an internal error`

I have a Kafka Connect S3 Sink writing records to Amazon S3. This particular sink is writing about 4k rec/sec. Every few days, one of the Kafka Connect worker tasks fails with the following error. A manual restart completely fixes the issue until it…
clay
  • 18,138
  • 28
  • 107
  • 192
1
vote
1 answer

Debezium Kafka connector mongodb

Is there any way to specify to kafka connector which actions on mongo should consider. Actually, I don't want that kafka does anything when a mongo document is deleted (just update and insert new doc should be considered). Thank you
Mahmoud EL Khattab
  • 11
  • 4
1
vote
0 answers

how to get partition info and offset for Kafka topic without knowing consumer group info

I am totally a squat in Kafka land If I run the command /cfintools/confluent-4.0.0/bin/kafka-avro-console-consumer --topic $t --bootstrap-server $bt --consumer.config /cfintools/avro_consumer_ssl_cfin_prod.properties --property…
user1874594
  • 2,277
  • 1
  • 25
  • 49
1
vote
1 answer

Is there a Kafka counterpart to AWS Kinesis Firehose?

Looks like AWS Kinesis Firehose is a way to ingest data (buffered together) to an object, is there a Kafka counterpart for this kind of functionality?
chen
  • 4,302
  • 6
  • 41
  • 70
1
vote
2 answers

Kafka-connect elasticsearch auto-lowercase topic name for for index

I'm using elasticsearch sink kafka-connector to index messages from multiple kafka topics to elasticsearch. I have my topics with camelCase naming, and I can't change it. So when starting up the ES sink connector, it does not index anything because…
Viacheslav Shalamov
  • 4,149
  • 6
  • 44
  • 66
1
vote
1 answer

Safely give secret/token to Kafka Connector?

We are using Kafka Connectors (JDBC and others), and configuring them using the REST API (using curl in shell scripts). Right now, when testing/developing, we are including secrets (for the JDBC connect - database user/pw) directly in the request.…
AnoE
  • 8,048
  • 1
  • 21
  • 36
1
vote
1 answer

Parsing dates in format dd.MM.yyyy in Kafka Connect using kafka-connect-spooldir connector

I am trying to use SpoolDirCsvSourceConnector from https://github.com/jcustenborder/kafka-connect-spooldir I have following configuration for connector in…
Regfor
  • 8,515
  • 1
  • 38
  • 51
1
vote
2 answers

Kafka-connect FileStreamSourceConnector doesn't produce to a topic

I'm trying to create a Kafka-connect connector to sink from an AVRO Topic to a file. And then restore this file to another topic using kafka-connect. The sink is working fine, I could see the sink file growing and read the data. But when I try to…
BigLeo
  • 6,906
  • 2
  • 13
  • 12
1
vote
0 answers

Timestamp field in Avro source connector without time zone

I have a SQL DATETIME field DT that I want to feed into Kafka using Avro. DT is in local time but has no time zone; that's stored in another column TZ as an offset from UTC in minutes. It seems that an Avro timestamp-millis would be appropriate for…
z0r
  • 8,185
  • 4
  • 64
  • 83
1
vote
2 answers

Kafka Connect to persist topic to Elasticsearch index using field from (json) message

I'm attempting to index messages in Elasticsearch using SMT's from Kafka's Connect API only. So far I had luck with simply using the topic and timestamp router functionality. However, now I'd like to create separate indices based on a certain field…
Julius
  • 11
  • 2
1
vote
2 answers

Getting info about Kafka connect cluster

I'm currently having Kafka connect cluster with two nodes, using the same group.id when using curl /connectors I can get list of my connectors created, but I can't see infos about alive nodes, Health check ...
Mohamed Ait Taleb
  • 65
  • 7
1 2 3
…
99 100