0

My Scenario is I have data in AWS S3 flat files. I am using SNS to trigger the Snow-pipe when new file arrives in S3. To load the data from flat files in S3 to Snowflake table I am using Snow-pipe. So While loading data from flat files to snowflake table by Snow-pipe, Can I handle data-validation and couple of calculations on source data? Please help me if we have any way to do this... Thanks in Advance.

Rajesh Atikela
  • 11
  • 3

1 Answers1

0

Validation_mode copy option is not yet supported by snowpipe. However, snowpipe does support simple transformations like column reordering, cast etc are supported. The best way to perform calculations and transform your data would be to load the data into a staging table and process downstream into target tables.

Reference: https://docs.snowflake.net/manuals/sql-reference/sql/create-pipe.html#usage-notes https://docs.snowflake.net/manuals/user-guide/data-load-transform.html

Abhi Reddy
  • 467
  • 2
  • 6