Stack Overflow
Stack Overflow

Questions tagged [debezium-engine]

30 questions
2
votes
0 answers

Debezium misssing some events

During the process of data discrepancy, I realised some events are missed and not written to kafka during CDC. These are the steps I did: 1- Setup two different debezium connector each fetch data from similar mysql table and write to two different…
ArefehTam
  • 367
  • 1
  • 6
  • 20
2
votes
0 answers

Debezium Ebedded cannot instatiate transformation 'unwrap'

I managed to stream CDC events from Postgres using Debezium Embedded version 1.6.2.Final. Now I'm interested in flattening the events so only the after data is sent. As per the following article:…
franDayz
  • 883
  • 10
  • 22
1
vote
0 answers

Embedded Debezium configurations for dynamic table and filters addition without connector restart

I'm using an embedded debezium engine (v1.8.1) within my spring boot application. Our application allows the user to create workflows for their businesses and they can create user-defined triggers to start the workflow. The user can define triggers…
Shank
  • 11
  • 1
1
vote
0 answers

Debezium: Stream messages in one topic

I'm fairly new to Debezium and I understand that Debezium generates a topic for a specific table and that if there is an update/ delete on that table, it will send a message to Kafka via that topic. I was wondering if you could have 1 topic that…
rm12345
  • 1,089
  • 3
  • 18
  • 32
1
vote
0 answers

Debezium: Detect changes from a related table in one topic

I am using Debezium with Postgres connector. So far, things are working fine where I can make an update on a table (e.g. List table) and I can see the message via the relevant Kafka topic. Here is an example of the Debezium…
rm12345
  • 1,089
  • 3
  • 18
  • 32
1
vote
0 answers

Debezium: Replicating data from Oracle read-only database

My use case is to capture CDC data from Oracle read-only PDB database using Debezium. When I tried installing and run Debezium, it's throwing below error message Can someone please help with the correct Debezium config? ORA-16000: database or…
Raj
  • 53
  • 1
  • 6
0
votes
0 answers

Embedded Debezium WAL piling up restart_lsn not moving

So we are using debezium in our spring boot application. And while the messages are getting consumed, the restart_lsn is not moving and the replication slot lag is increasing. postgres=# SELECT slot_name, …
Ankitha N
  • 39
  • 4
0
votes
0 answers

Apache Flink not continuing from last place read when using Debezium IO connector

I am experimenting with a Flink pipeline which I've configured to read change data events from PostgreSQL database on AWS RDS. The configuration works as expected for the most part, I am able to see change data with before and after properties set…
Gayal Rupasinghe
  • 114
  • 8
0
votes
0 answers

How do I configure File schema history storage for Debezium MySql connector?

I'm writing a Debezium (2.3.2) Engine program. After running successfully with Postgres, I switched to MySQL but hit the following problem: Exception in thread "pool-1-thread-1" org.apache.kafka.common.config.ConfigException: Invalid value…
andreasmartens
  • 96
  • 5
0
votes
1 answer

Where to begin with fixing an offsets issue with Debezium Engine

I'm using Debezium engine to sync data from a MySQL database. Since I'm using Debezium Engine I'm using the org.apache.kafka.connect.storage.FileOffsetBackingStore to record my current changes offset. I think my computer had a power outage recently…
omatase
  • 1,551
  • 1
  • 18
  • 42
0
votes
1 answer

Debezium MySql | 2 connectors sharing the Processing the Load & High Availability

Wondering if this is possible? I'd like to have 2 debezium connector processes (with different server ids) pointing to the same MySQL database and receiving the data as if they are workers sharing the load. Currently I observe the behaviour of…
user1189332
  • 1,773
  • 4
  • 26
  • 46
0
votes
0 answers

On Debezium startup, initial snapshot excludes existing triggers, functions and other object types

I created a database called dictionary as my master DB which has 2 tables(state and employees). In addition to the tables, I also added few functions, triggers, stored procedures, indexes. I am hoping to replicate everything to a slave. On debezium…
arsenics
  • 13
  • 1
  • 6
0
votes
0 answers

Clearing messages in heartbeat Kafka topics

I'm current setting up Debezium connected to Amazon RDS for Postgres. I did have some issue with WAL consuming huge amount of disk space. After some research, I've set up additional heartbeat config for debezium source connector, here my…
FuLygon
  • 1
  • 1
0
votes
0 answers

Error "No implementation of Debezium engine builder was found" while connecting Debezium to the Java code

I'm trying to connect Debezium to my Java code and expose the event messages in json format. I'm delivering this a a jar file which will be embedded into another application. Here when I setup params and trying to initialize the Debezium with the…
RusJaI
  • 666
  • 1
  • 7
  • 28
0
votes
1 answer

Track a particular table using Debezium

I am using Debezium embedded oracle connector for spring to track changes in my tables. Is there a property/way to track a particular table from a schema. My schema has a lot of tables and I do not want to track all of them for changes. I have added…
Pranav Gaba
  • 1
  • 1
1
2