Stack Overflow
Stack Overflow

Questions tagged [debezium]

Debezium is an open source distributed platform to stream database changes. Use this tag for questions related to using, configuring or troubleshooting Debezium.

Overview

From [https://debezium.io/]:

Debezium is an open source distributed platform for change data capture.

Debezium uses Kafka to record all change events. It supports connectors for:

  • MySQL
  • Oracle
  • MongoDB
  • PostgreSQL
  • SQL Server (colloquially "MSSQL" or "MS SQL")

References

Related Tags

1029 questions
6
votes
2 answers

Best way to join two (or more) kafka topics in KSQL emiting changes from all topics?

We have a "microservices" platform and we are using debezium for change data capture from databases on these platforms which is working nicely. Now, we'd like to make it easy for us to join these topics and stream the results into a new topic which…
Phillip Fleischer
  • 1,023
  • 12
  • 14
6
votes
2 answers

Debezium: Unable to setup watcher on Kafka-Connect Docker container. Error while fetching metadata with correlation id 2 ... LEADER_NOT_AVAILABLE

Question: How do I setup watcher to watch a Kafka-Connect stream in real-time as changes are made to the database? I am getting Error while fetching metadata with correlation id 2 : {dbhistory.Test.Posts=LEADER_NOT_AVAILABLE}. What is causing the…
J Weezy
  • 3,507
  • 3
  • 32
  • 88
6
votes
1 answer

Postgres Debezium does not publish the previous state of a record

I successfully installed Postgres Debezium CDC. Now, I'm able to catch all changes happening to the database. But the problem is "before" field always stays empty. So, if I insert a record (id = 1, name = Bill) I then get from Kafka this…
Jacobian
  • 10,122
  • 29
  • 128
  • 221
6
votes
3 answers

Debezium flush timeout and OutOfMemoryError errors with MySQL

Using Debezium 0.7 to read from MySQL but getting flush timeout and OutOfMemoryError errors in the initial snapshot phase. Looking at the logs below it seems like the connector is trying to write too many messages in one…
Kamil Sindi
  • 21,782
  • 19
  • 96
  • 120
5
votes
1 answer

Connection timeout using local kafka-connect cluster to connect on a remote database

I'm trying to run a local kafka-connect cluster using docker-compose. I need to connect on a remote database and i'm also using a remote kafka and schema-registry. I have enabled access to these remotes resources from my machine. To start the…
Malkath
  • 77
  • 3
  • 13
5
votes
1 answer

Is `ALTER TABLE mytable REPLICA IDENTITY FULL;` a safe operation for a high volume postgres database?

I am working with debezium and a Postgres database. The database contains several tables with TOAST (The Oversized-Attribute Storage Technique) columns. To handle unchanged Postgres TOAST column values, one approach is to set the REPLICA IDENTITY of…
Ivan Feliciano Avelino
  • 51
  • 1
  • 1
  • 4
5
votes
1 answer

Postgres using debezium Creation of replication slot failed

I configured debezium with postgres as the below code, when i started my spring boot application a get an error Creation of replication slot failed @Bean public io.debezium.config.Configuration connector() { return…
Aymen Kanzari
  • 1,765
  • 7
  • 41
  • 73
5
votes
0 answers

Postgres replication slot showing inactive

I am using debezium postgres connector for change data capture. One of the connectors is streaming change events to Kafka but still the slot is showing as inactive in postgres table 'pg_replication_slots'. Anyone knows the reason for this ?
abhinav kumar
  • 153
  • 1
  • 9
5
votes
1 answer

How to rename primary key when using Debezium and Kafka Connect JDBC sink connector to synchronize databases?

I am attempting to synchronize a table in an upstream database to a downstream database using Debezium, following the approach described on the Debezium blog here. In the downstream table, I only need certain columns from the upstream table. I also…
Zejji Zejji
  • 497
  • 3
  • 14
5
votes
3 answers

External offset store with the debezium embedded connector

My team is building a CDC service with the Debezium embedded connector. For the offset storage we're thinking about using S3/DynamoDB. Just wondering if anyone here has written something similar to externalize the offset store and what they chose…
drexler
  • 51
  • 1
5
votes
2 answers

Field does not exist on transformations to extract key with Debezium

I am trying to create a Debezium MySQL connector with a transformation to extract the key. Before key transformations: create source connector mysql with( "connector.class" = 'io.debezium.connector.mysql.MySqlConnector', "database.hostname"…
4it med
  • 181
  • 1
  • 7
5
votes
1 answer

Debezium's ExtractNewRecordState transform cannot work

I am build an data synchronizer, which capture the data change from MySQL Source, and export the data to hive. I choose to use Kafka Connect to implement this. I use Debezium as source connector, and confluent hdfs as sink connector. Debezium…
user8510613
  • 1,242
  • 9
  • 27
5
votes
4 answers

Debezium: No maximum LSN recorded in the database; please ensure that the SQL Server Agent is running

This question is related to: Debezium How do I correctly register the SqlServer connector with Kafka Connect - connection refused In Windows 10, I have Debezium running on an instance of Microsoft SQL Server that is outside of a Docker container. I…
J Weezy
  • 3,507
  • 3
  • 32
  • 88
5
votes
1 answer

org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped

Using postgres source connector in kafka. It works properly for some time and suddently stops with above error. Please assist if someone knows this issue.
Joseph N
  • 540
  • 8
  • 28
5
votes
1 answer

Kafka Connect - Failed to commit offsets and flush

I had my Kafka Connectors paused and upon restarting them got these errors in my logs [2020-02-19 19:36:00,219] ERROR WorkerSourceTask{id=wem-postgres-source-0} Failed to commit offsets…
AnonymousAlias
  • 1,149
  • 2
  • 27
  • 68
1
2
3
…
68 69