Questions tagged [change-data-capture]

Change data capture (CDC) encompasses database design patterns to keep track of changed data and perform actions with it.

In databases, change data capture (CDC) is a set of software design patterns used to determine (and track) the data that has changed so that action can be taken using the changed data. Also, Change data capture (CDC) is an approach to data integration that is based on the identification, capture and delivery of the changes made to enterprise data sources.

CDC solutions occur most often in data-warehouse environments since capturing and preserving the state of data across time is one of the core functions of a data warehouse, but CDC can be utilized in any database or data repository system.

271 questions
2
votes
1 answer

How to use Change Data Capture with Entity Faramework CodeFirst

I want create report from Change Data Capture on some web app builded on ASP.NET MVC and EF. EF using Code First and the database is created in dbo schem. CDC generate tables under cdc scheme in system tables of database that using EF. Maybe…
2
votes
1 answer

How does Hibernate Envers work?

I am using Hibernate envers in one of my projects. It works really good and does that job. I am trying to understand the internal working mechanism of envers. How does envers work internally? What happens when the actual transaction is succeeded…
zeagord
  • 2,257
  • 3
  • 17
  • 24
2
votes
1 answer

CDC multiple insert/delete of the same identity value

I have a table T that contains an ID set as identity and primary key. I have enabled CDC on the table and then later added an XML field that I didn't care capturing so I did not do anything further (to recreate the capture table and/or migrate old…
P. Kouvarakis
  • 1,893
  • 12
  • 21
2
votes
0 answers

Does SQL Server support a feature to track audit, BOTH row update and schema changes in the same place

I currently capture audit history of billing changes tables data in a SQL Server database, on the row updates/inserts. This question/answer has been beaten to death (but also evolved both from Microsoft SQL Server features, and features for user…
aggie
  • 798
  • 2
  • 8
  • 23
2
votes
4 answers

Is Log Sequence Number (LSN) unique for database or table in SQL Server?

I am using SQL CDC to track changes for multiple tables in SQL Server. I would want to report out these changes in right sequence for each I have a program which collects the data from each CDC table. But I want to make sure that all the changes…
rauts
  • 1,018
  • 9
  • 21
2
votes
1 answer

SSIS and CDC - Incorrect state at end of "Mark Processed Range"

The Problem I currently have CDC running on a table named subscription_events. The corresponding CT table is being populated with new inserts, updates, and deletes. I have two SSIS flows that move data from subscription_events into another table in…
FooBar
  • 384
  • 2
  • 4
  • 14
2
votes
0 answers

Change Data Capture on Hadoop

I need to migrate an existing CDC system based on Oracle to Hadoop. Anybody implemented CDC on Hadoop? Please share the approaches of CDC on Hadoop. Thanks!
Venkat Ankam
  • 916
  • 11
  • 17
2
votes
1 answer

Cannot rename a column that is "replicated", probably due to CDC

I want to run the following rename EXECUTE sp_rename N'dbo.Semesters.IsPublic', N'Tmp_ShowNCs', 'COLUMN' I get the error Msg 4928, Level 16, State 1, Procedure sp_rename, Line 547 Cannot alter column 'IsPublic' because it is 'REPLICATED'. Msg 0,…
usr
  • 168,620
  • 35
  • 240
  • 369
2
votes
1 answer

Transfer an image from CDC to CBitmap

How can I transfer an image from CDC to CBitmap? The problem in whole: I have a big image into CBitmap A. I need to transfer parts of this image to a number of CBitmap for the storage into a vector, because I can't use a number of CDC for this :) I…
Seer
  • 45
  • 2
  • 7
2
votes
2 answers

Get cdc tables in visual studio 2012/2010 database project

I am trying to create a database project in Visual Studio 2012/2010 where I need the CDC (Change Data Capture) tables , because lot of my views are dependent on the CDC tables. I couldn't find any way to import the cdc schema/tables :(. Read in many…
2
votes
1 answer

Remove Change data capture manually

I'm having a serious problem with one of my production databases. The database was enabled for change data capture which was performing quite well. Last month we had a server meltdown where several disks were affected and data was lost. We then…
1
vote
1 answer

How to know number of rows affected by CDF merge in pyspark?

I have a CDF logic where I need to know of row impacted due to merge i.e I need to know number of inserted, Updated and deleted row in order to make some decision. i am able to get to the required information in SQL, but did not get in…
1
vote
0 answers

Kafka Connect's Debezium Connector for DB2 not capturing column with XML data type

I have a local instance of a kafka connect cluster. I am trying to replicate data from a DB2(v11.5.8.0) database. I am using debezium connector(v2.2.1) to capture a table in DB2 that contains a column with an XML data type. Based on the…
1
vote
0 answers

CDC change data capture in GCP using Datastream, Postgres and Cloud storage

I want to setup CDC from postgres to cloud storage using Datastream through Terraform. I am referring to Terraform Docs. But the example given in this doc doesnt work out of box. Following is what I have built based on docs: provider "google" { …
1
vote
0 answers

Debezium resulting in high transaction log disk usage

I am trying to setup CDC based replication for different databases (Amazon Postgres RDS) using multiple Debezium Kafka Connectors (using MSK Connector). With addition of more connectors, the replication is creating issues. The Oldest Replica Slot…