1

I was wondering if anyone knows if snowflake can do change data capture, and how it would be done.

I can't seem to find any documentation on it.

Thanks

2tone_tony
  • 47
  • 1
  • 10
  • Try Update command: https://docs.snowflake.net/manuals/sql-reference/sql/update.html – Ambrish Dec 19 '18 at 00:40
  • Are you looking to capture changes occurring in Snowflake, or capture changes from some other source, and write those changes into Snowflake? – metadaddy Feb 04 '19 at 21:12
  • I'm trying to capture changes from data from an outside source that is writing into a snowflake table – 2tone_tony Feb 12 '19 at 01:33
  • As of now, Snowflake doesn't support triggers or let you monitor data changes in a table so that you can act on them. I'd suggest writing your own python code simulating trigger based on the change detected in a table. – mevdiven Mar 21 '19 at 14:23

2 Answers2

3

Change Data Capture (CDC) in Snowflake can be done with streams: https://docs.snowflake.net/manuals/user-guide/streams.html

For CDC with an external source you rely on data integration providers like Alooma, Matillion, Golden Gate or implement it on your own.

Plus, Snowflake also integrates with Kafka. So that you "only" need to push changes to Kafka for integration with Snowflake: https://docs.snowflake.net/manuals/user-guide/kafka-connector.html

Rick
  • 2,080
  • 14
  • 27
0

If you want to use a cost effective system, go with existing cdc tools like hvr-software which is providing its best as per my experience with them.

Best thing in hvr is that it has many connectors like s3, snowflake and inbuilt hive support to create tables on top of the data which we load.

Note: ofcourse snowflake supports the stream with kafka or any other source but why do we need to make our cluster up and running all the time, instead cdc tool runs on a single ec2 instance.

MARTHA VIJAY KUMAR
  • 31
  • 1