Stack Overflow
Stack Overflow

Questions tagged [snowpipe]

31 questions
0
votes
1 answer

How can i implement new feature snowflake email notification on snowpipe and Task

Can anyone provide use-case on new notification integration of type email I wanted to implement this feature in snowpipe and task but i got error "MY_EMAIL_INT" is a notification integration of type email which i have created 1 but when i used this…
Lucifer Morning star
  • 3
  • 2
0
votes
1 answer

Snowflake table

how can we create table in snowflake without running create table query, directly our snowpipe will detect data format and will create a table for us. Answer for my query related to snowflake
Nishant Sharma
  • 1
0
votes
1 answer

Can I perform transformations using Snowflake Streams?

Currently I have a snowflake table being updated from a kafka connector in near-realtime, I want to be able to then in near-real time take these new data entries through something such as snowflake cdc / snowflake streams and append some additional…
Sam Richardson
  • 15
  • 3
0
votes
0 answers

How does snowflake route notifications to pipes knowing they share the same SQS queue?

When I run SHOW PIPES I can see that all the snowpipes have the same notification_channel value. I'm using SQS to send event notifications to snowpipes, with filters on the file path. But, even if all the snowpipes have the same…
Mehdi Sidi Boumedine
  • 57
  • 6
0
votes
2 answers

Can snowpipe pick up files from within sub-folders?

I'm looking to try to pick up all parquet files from an s3 bucket that have been placed into partitioned sub-folders by date. In the past I've used snowpipe with a sort of 1-1 relationship, one sub-folder to one table; but I would be interested to…
Sam Richardson
  • 15
  • 3
0
votes
0 answers

Using snowpipe to read parquet file's timestamp

I am using snowpipe to ingest data from azure blob to snowflake. Issue I am facing is that there are multiple file (rows) of data with the same primary key. Is there a way that snowpipe can get this files time stamp? Using timestamp will allow me…
user18355480
  • 1
0
votes
0 answers

Snowpipe error : Failed to access remote file: access denied. Please check your credentials

We are using snowpipe to upload files from AWS S3 to snowflake via storage integration. The auto ingestion is set True for this. Files are placed/ uploaded into S3 bucket by MWAA script. The problem which is occurring here, that sometimes data is…
user2388773
  • 155
  • 8
0
votes
1 answer

Amazon SNS (Terraform) Invalid parameter: Policy statement action out of service scope

I want to create an IAM policy document and attached two values taken from snowflake error integration as Trusted Relationship in the policy. Following this tutorial Step 5. The idea is that to add SF_AWS_IAM_USER_ARN and SF_AWS_EXTERNAL_ID created…
user6308605
  • 693
  • 8
  • 26
0
votes
2 answers

What privileges are required to create a dataframe using Snowpark?

I am using Snowpark with Python 3.8. I have successfully executed a CREATE OR REPLACE TABLE statement from my code. However when I add this next line of code df1 = session.create_dataframe(data) I receive a privileges error: <<
Nick Hall
  • 49
  • 1
  • 11
0
votes
1 answer

Need help finding correct pattern to be used in snowpipe copy into command

Need help on copy into pattern in snowpipe. facing challenge while using pattern for below file present in s3 bucket folder S3bucket/Folder/files 1)changes_tags.csv.12345 2)changes_tags.csv.987987 3)tags.csv.67587 4)tags.csv.9987 to load tags.csv…
phenix
  • 1
  • 1
0
votes
3 answers

Snowpipe Auto Ingest Commit Interval

I have a file which has 3 millions record and I am using snowpipe auto ingest feature to load it automatically. Wanted to know the behavior of snowpipe will it load 3 million records in stage in one shot then commit or it will be some sort…
rjain550
  • 105
  • 2
  • 8
0
votes
1 answer

Snowflake in One Region, AWS bucket in another. Costs?

Any Snowflake experts? We have Snowflake Database in eu-west-2 and we have to read data from s3 buckets in us-west. Created snowpipes and SQS notification in s3 buckets. Will there be additional costs for reading from us-west-2 notification channel.…
Leo
  • 868
  • 1
  • 13
  • 32
0
votes
0 answers

facing issue in triggering SQS notification from aws S3 to Snow PIPE

I am new to Snowflake, Trying to notify the snowpipe when a file is placed in a s3 bucket and thru SQL event Notification the snowpipe needs to load the data into the respective table. Below are the steps I have followed. Step -1: created a table in…
Gayathri
  • 1
0
votes
0 answers

Snowpipe recreation and possibility of sqs changing

When recreating a snowpipe, how likely is it that the pipe's sqs arn changes? The documentation recommends double checking everything after recreation. It implies that the sqs might change. But I'm interested to hear has anyone noticed such…
antruo
  • 1
  • 2
0
votes
0 answers

Snowpipe loading

I am loading files to snowflake tables using Snowpipe. I have already loaded a file named employees1 to snowflake table named Employee_table. I truncated the table and wanted to load that file again. As snowpipe will store load history in metadata…
Alka
  • 1
1
2
3