Stack Overflow
Stack Overflow

Questions tagged [apache-kafka-connect]

Apache Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other data systems.

Apache Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other data systems.

It was first released with Kafka 0.9. It allows to import data from external systems (e.g., databases) into Kafka and it allows to export data from Kafka into external system (e.g., Hadoop). Apache Kafka Connect is a framework that supports a plug-in mechanism allowing to provide custom connectors for your system of choice.

Documentation

3693 questions
6
votes
2 answers

When does a task in a connector move to unassigned state?

I am running CP3.2 in a distributed mode and some of the connector which are defined even with "tasks.max": "1" have task "UNASSIGNED" state. I have increased the memory allocated to worker and restart the worker has solved me the problem or…
Renukaradhya
  • 812
  • 2
  • 19
  • 31
6
votes
1 answer

Distributed Kafka Connect topic configuration

I had the problem that my Kafka Connect worker configuration was lost after a node restart. (http://broker:port/connectors/ -> empty array) Now I think it can have something to do with the "retention.ms" config. Because the connect config is also…
imehl
  • 789
  • 1
  • 9
  • 25
6
votes
1 answer

Kafka Streams with lookup data on HDFS

I'm writing an application with Kafka Streams (v0.10.0.1) and would like to enrich the records I'm processing with lookup data. This data (timestamped file) is written into a HDFS directory on daily basis (or 2-3 times a day). How can I load this in…
Bruckwald
  • 797
  • 8
  • 23
5
votes
1 answer

Connection timeout using local kafka-connect cluster to connect on a remote database

I'm trying to run a local kafka-connect cluster using docker-compose. I need to connect on a remote database and i'm also using a remote kafka and schema-registry. I have enabled access to these remotes resources from my machine. To start the…
Malkath
  • 77
  • 3
  • 13
5
votes
3 answers

Getting ConfluentHubClient exception while installing kafka connectors

I have set up Kafka cluster and installed confluent-hub on a EC2 instance. I have downloaded the confluent-hub tar file, extracted and placed the binary inside /usr/local/bin. When I am now trying to install any connectors, I am getting the…
zilcuanu
  • 3,451
  • 8
  • 52
  • 105
5
votes
1 answer

Postgres using debezium Creation of replication slot failed

I configured debezium with postgres as the below code, when i started my spring boot application a get an error Creation of replication slot failed @Bean public io.debezium.config.Configuration connector() { return…
Aymen Kanzari
  • 1,765
  • 7
  • 41
  • 73
5
votes
0 answers

Postgres replication slot showing inactive

I am using debezium postgres connector for change data capture. One of the connectors is streaming change events to Kafka but still the slot is showing as inactive in postgres table 'pg_replication_slots'. Anyone knows the reason for this ?
abhinav kumar
  • 153
  • 1
  • 9
5
votes
2 answers

How to use Kafka connect in Strimzi

I am using Kafka with strimzi operator, I created a Kafka cluster and and also deployed Kafka connect using yml file. But after this I am totally blank what to do next . I read that Kafka connect is used to copy data from a source to Kafka cluster…
rohit290554
  • 79
  • 3
  • 10
5
votes
1 answer

How to apply an SMT to a single topic in Kafka connect?

I would like to apply a set of SMT (Single Message Transforms) to one single topic. I have the following configuration.. connector.class: io.debezium.connector.postgresql.PostgresConnector name: test database.server.name:…
JavaTechnical
  • 8,846
  • 8
  • 61
  • 97
5
votes
1 answer

How to rename primary key when using Debezium and Kafka Connect JDBC sink connector to synchronize databases?

I am attempting to synchronize a table in an upstream database to a downstream database using Debezium, following the approach described on the Debezium blog here. In the downstream table, I only need certain columns from the upstream table. I also…
Zejji Zejji
  • 497
  • 3
  • 14
5
votes
0 answers

Kafka Connect - JDBC Source Connector - Setting Avro Schema

How can I make Kafka Connect JDBC connector to predefined Avro schema ? It creates a new version when the connecter is created. I am reading from DB2 and putting into Kafka topic. I am setting schema name and version during creation but it does not…
usernotexist
  • 51
  • 1
  • 5
5
votes
0 answers

Kafka Connection to 2 was disconnected before the response was read

We are getting the below exception in Kafka server.Please let me know why we are getting this issue? This issue keep on coming in the production environment. kafka version is 2.12-2.3.0 rEpoch=Optional[2708]), rds_exception-1=(fetchOffset=0,…
user3094331
  • 532
  • 6
  • 21
5
votes
2 answers

Field does not exist on transformations to extract key with Debezium

I am trying to create a Debezium MySQL connector with a transformation to extract the key. Before key transformations: create source connector mysql with( "connector.class" = 'io.debezium.connector.mysql.MySqlConnector', "database.hostname"…
4it med
  • 181
  • 1
  • 7
5
votes
1 answer

Debezium's ExtractNewRecordState transform cannot work

I am build an data synchronizer, which capture the data change from MySQL Source, and export the data to hive. I choose to use Kafka Connect to implement this. I use Debezium as source connector, and confluent hdfs as sink connector. Debezium…
user8510613
  • 1,242
  • 9
  • 27
5
votes
1 answer

Connect to a docker container service inside WSL from Windows

I have a Windows Subsystem Linux setup on my Windows 10 PC (Ubuntu 18.04). I installed Docker toolbox on Windows, and running through a VM I can run docker commands normally by setting: export DOCKER_HOST=tcp://192.168.99.101:2376 I can access it…
johan855
  • 1,578
  • 4
  • 26
  • 51
1 2 3
…
99 100