Stack Overflow
Stack Overflow

Questions tagged [change-data-capture]

Change data capture (CDC) encompasses database design patterns to keep track of changed data and perform actions with it.

In databases, change data capture (CDC) is a set of software design patterns used to determine (and track) the data that has changed so that action can be taken using the changed data. Also, Change data capture (CDC) is an approach to data integration that is based on the identification, capture and delivery of the changes made to enterprise data sources.

CDC solutions occur most often in data-warehouse environments since capturing and preserving the state of data across time is one of the core functions of a data warehouse, but CDC can be utilized in any database or data repository system.

271 questions
3
votes
2 answers

SQL Server Change Data Capture - Operation Type 3 shows NULLS in fields that aren't NULL

We've just started using Change Data Capture on SQL Server 2017. I'm running some tests by executing a few simple UPDATE statements then checking the CDC table. To view the table changes, we run the CDC function cdc.fn_cdc_get_all_changes passing…
Stpete111
  • 3,109
  • 4
  • 34
  • 74
3
votes
2 answers

Implementing cdc but getting value error in Python Pandas

I am trying to perform CDC operation via Python. I am trying to perform union of the unchanged data (master file / base table) with the new file (delta file). Below is the function I have written: def processInputdata(): df1 =…
akash sharma
  • 411
  • 2
  • 24
3
votes
5 answers

How to compare SQL Server CDC LSN values in C#?

In SQL its easy as it supports the binary(10) LSN values for comparison: SELECT *, __$start_lsn, __$seqval FROM cdc.fn_cdc_get_all_changes_dbo_sometable(@startLsn, @endLsn, 'all update old') WHERE __$seqval > @seqval ORDER BY __$start_lsn,…
Simon Hughes
  • 3,534
  • 3
  • 24
  • 45
3
votes
1 answer

Postgres/JDBC/Logical replication - out of memory issues

I am developing an application which connects to a logical replication slot, to consume the WAL events. These WAL events are then forwarded to a MQ broker. This works great, but I noticed that I am running out of memory after some time. I managed to…
YasonTR
  • 41
  • 1
  • 6
3
votes
0 answers

Embed SymmetricDs for CDC (Change Data Capture )

Is it possible to embed Symmetric Ds for CDC operations ? If yes how can I do that ? If no is there other open source tools that can be embedded for CDC ? Other than kafka and Debezium ?
elpazio
  • 697
  • 2
  • 9
  • 25
3
votes
1 answer

Take output from C++ exe and work on it in Python in real time

I have a C++ executable, for an eye tracker, that printf's gaze coordinates to the terminal. I need to be able to get these into Python for data analysis etc. It would be awesome if there was a way that I could do this in real time, so I could…
skailasa
  • 121
  • 8
3
votes
2 answers

SQL Server Change Data Capture - Capture user who made the change

Concerning SQL Server Change Data Capture, can you track the User who has made the change to the row/column data or is there anyway to extend CDC to allow this? I couldn't see anything in the documentation.
garfbradaz
  • 3,424
  • 7
  • 43
  • 70
3
votes
1 answer

Function sys.fn_cdc_get_max_lsn() returns null

I'm try use the cdc, and i was following these steps. EXEC sys.sp_cdc_enable_db -- Enable DB EXEC sys.sp_cdc_enable_table @source_schema = N'DW', @source_name = N'APIprocess', @role_name = NULL, …
Lean Bonaventura
  • 411
  • 5
  • 13
3
votes
2 answers

How to integrate Oracle and Kafka

I've been trying to find the most efficient/effective way capture change notifications in a single Oracle 11g R2 instance and deliver those events to an Apache Kafka queue, but I haven't been able to find any simple examples or tutorials along these…
rcurrie
  • 329
  • 1
  • 3
  • 17
3
votes
0 answers

Trouble with Change Data Capture in SQL Server 2008 R2

After activating CDC for a database I'm working on, the capture job fails to actually capture anything, I'm getting these kind of errors: The SQL Server is SQL Server 2008 R2 SP3 on a VMWare 10.x VM, with 8GB RAM assigned. The steps I did where…
Adrian E. Labastida Cañizares
  • 479
  • 4
  • 11
3
votes
1 answer

How do I process an initial load of a CDC table?

I'm having a little trouble following the article on how to use the CDC Control Task. Specifically, I seem to be unable to process the initial load in such a way that the subsequent incremental load is seamless (that is, no gap and no overlap) with…
Ben Thul
  • 31,080
  • 4
  • 45
  • 68
3
votes
2 answers

Change Data Capture For Updates and Deletes Only

Our database is insert inensive (200-500k per night) but update light (maybe a few hundred per day). I need to preserve a history of all changes to the inserted rows themselves for an indefinite period of time (but not the actual insert). I would…
PseudoToad
  • 1,504
  • 1
  • 16
  • 34
2
votes
1 answer

Change Data Capture - initial load of historical data

I'm upgrading my SQL Server 2000 database to SQL Server 2008 R2. I want to make use of Change Data Capture feature. Im my existing application I have the similar functionality, but I'm using triggers and historical table with Hst_ prefix with almost…
Lukasz Lysik
  • 10,462
  • 3
  • 51
  • 72
2
votes
1 answer

Change Data Capture for several tables

SQL Server 2008. Is it possible to create Change Data Capture (or Change Tracking) for several related tabled in a db? For example many-to-many relation. How it will look like?
Anton Lyhin
  • 1,925
  • 2
  • 28
  • 34
2
votes
2 answers

Can i watch only one field in Couchbase with kafka-connect (CDC)?

We are trying to move our database from mysql to couchbase and implement some CDC (change data capture) logic for copying data to our new db. All enviroments set up and running. mysql, debezium, kafka, couchbase, kubernetes, pipeline etc. And also…
MC_delta_T
  • 596
  • 1
  • 9
  • 22
1 2
3
…
18 19