Stack Overflow
Stack Overflow

Questions tagged [snowpipe]

31 questions
6
votes
0 answers

Setting SQS Destination for S3 Event Notification - Invalid S3 Resource Format

I was attempting to setup an event notification in S3 that would send a message to an SQS generated by Snowflake on any object creation event. This is fairly straightforward, but when I attempted to test the configuration, I noticed that no event…
Woody1193
  • 7,252
  • 5
  • 40
  • 90
2
votes
1 answer

How to load 10 days of data from amazon S3 to snowflake using snowsql

I want to load last 10 days data from amazon S3 into snowflake tables using snowsql. Say today is 10th may, then I want all the data that has been changed in the past 10 days using snow sql. I want to do it at once instead of querying for each day…
phoenix..
  • 21
  • 2
2
votes
1 answer

How to trigger Snowflake task after snowpipe completes its run

I am new to snowflake and I am trying to load data from S3 into Table A and trigger a Task to load data into table B from Table A. I have successfully implemented an automated snow pipe that loads data into Table A from an S3 immediately after data…
Abiodun
  • 959
  • 6
  • 17
  • 38
2
votes
1 answer

Snowflake ingestion: Snowpipe/Stream/Tasks or External Tables/Stream/Tasks

For ingesting data from an external storage location into Snowflake when de-duping is necessary, I came across two ways: Option 1: Create a Snowpipe for the storage location (Azure container or S3 bucket) which is automatically triggered by event…
Yasaman
  • 53
  • 4
2
votes
1 answer

Terraform Snowpipe on Azure error on integration

I'm trying to set up snowpipe with terraform on azure using https://registry.terraform.io/providers/chanzuckerberg/snowflake/latest/docs/resources/pipe#optional my resource looks like this: resource "snowflake_pipe" "pipe" { provider =…
Blue Moon
  • 4,421
  • 20
  • 52
  • 91
2
votes
4 answers

Snowflake Snowpipe - Email Alert Mechanism

I am planning to use Snowpipe to load data from Kafka, but the support team monitoring the pipe jobs needs an alert mechanism. How can I implement an alert mechanism for Snowpipe via email/slack/etc?
Minesh Mohan
  • 31
  • 4
1
vote
1 answer

i am facing an error while loading snowpipe into snowflake

Pipe Notifications bind failure "Cross cloud integration is not supported for pipe creation in AZURE using a stage in AWS." auto_ingest=true; //couldn't compile, facing the above error. i couldn't find any possible solutions to try
Kevin nash
  • 11
  • 1
1
vote
0 answers

Execute stored procedure at the end of Snowpipe job

Kind of self-explaining title ^_^ I have a Snowpipe job that ingests datafiles into a Snowflake staging table. Is it possible to trigger a stored procedure call at the end of Snowpipe job, instead of having Snowflake task running it, based on…
tivivi
  • 77
  • 7
0
votes
0 answers

trigger dbt after Snowpipe load

I want to check the possibility of triggering dbt models after snowpipe load data into Snowflake database. I have a requirement where realtime xml data comes everytime and I'll use snowpipe to ingest that data into snowflake. I want to know if there…
Ven Raj
  • 47
  • 6
0
votes
1 answer

Snowflake - Error in validation function on a pipe load

I have a pipe defined like this: CREATE OR REPLACE PIPE PIPE_NAME AUTO_INGEST=TRUE AWS_SNS_TOPIC= 'arn:aws:sns:...' AS COPY INTO TABLE1 FROM '@STAGE/TABLE1' FILE_FORMAT = (TYPE = 'csv' FIELD_DELIMITER = ';' SKIP_HEADER = 1 EMPTY_FIELD_AS_NULL =…
Robertino Bonora
  • 20
  • 5
0
votes
0 answers

Snowflake - Load error rows in a copy into script

I have a pipe that copies into a table with transformations, such us COPY INTO t1 ( col1, col2) FROM ( SELECT t.$1 AS col1, t.$2 AS col2 FROM '@STAGE/file1/' t ) file_format = (format_name =…
Robertino Bonora
  • 20
  • 5
0
votes
2 answers

Failed to access remote file: access denied. Please check your credentials in loading data into snowflake from aws s3 bucket file - SSE S3 encrypted

I am trying to load a simple CSV file from s3 to snowflake using aws external stage. I am able to list the files using list @stage_name command but get the access denied error while loading the file via copy into sql. I checked the encryption which…
Pawan Rawat
  • 495
  • 1
  • 7
  • 25
0
votes
0 answers

snowpipe to load internal stage to permanent tables

suggest me the notification channel for snow pipe method to load internal stage (input files land in internal stage) to permanent tables, --whenever files land in internal stage the snowpipe should start and load to internal table -- no external…
Rana
  • 9
  • 1
0
votes
1 answer

Where to find the Snowpipe errror log

I have created the below snow pipe. when I check the status of the pipe it says running. For some reason it is not copying the data, could someone please let me know how to check the error logs of Snowpipe? The copy into…
mahesh
  • 3,067
  • 16
  • 69
  • 127
0
votes
0 answers

Load only new data from stage into table with Snowpipe

I have a table in snowflake GAME.ALL_GAME_SAVES_RAW with allot of data already init. And inserted daily via snowpipe (upon new files landing in s3). -- Stage CREATE STAGE IF NOT EXISTS GAME.GAME_SAVES_STAGE URL = 's3://my-bucket/game-saves' …
Anum Sheraz
  • 2,383
  • 1
  • 29
  • 54
1
2 3