Stack Overflow
Stack Overflow

Questions tagged [snowflake-pipe]

41 questions
1
vote
0 answers

Snowpipe skips files

We have used Snowpipe for ~10 months now and we recently ran into a case where part of the files in a stage got uploaded to the corresponding snowflake table and any future files were not detected. Verified that the underlying stage and pipe were in…
Karthik
  • 11
  • 1
1
vote
0 answers

In Snowflake task do we have something like child task will wait until dependency are met other then parent task condition

I have 4 stream. A_STREAM, B_STREAM, C_STREAM, D_ STREAM I have chain of task where A_TASK is parent and it has 3 child task (B_TASK, C_TASK, D_TASK). CREATE TASK A_TASK WAREHOUSE = XYZ SECHDULE = '15 MINUTE' WHEN…
rjain550
  • 105
  • 2
  • 8
1
vote
2 answers

Is it necessary to have md5-content property for a file to be auto-ingested by snowpipe from azure adls gen2?

I am trying to load data through snowpipes and somehow all the files are not getting loaded, these files are from different sources including automated jobs, manual uploads, data extracts. The one thing I am seeing in common is - the files that do…
Ashish Singh
  • 25
  • 4
1
vote
1 answer

Snowflake Data Pipeline problems - in particular stream issue

Background I have implemented a snowflake data pipeline (s3 log file > SNS > pipe > stage table > stream > task > stored proc/UDF > final table) in our production snowflake database. While things were working on a smaller scale in our dev database,…
rakcs
  • 13
  • 3
0
votes
0 answers

increase snowpipe speed to process 1 record per file

I have a snowpipe. I do not have control over the incoming files. The files each contain a single, relatively small json record. The stage had about 38000 files when I restarted the pipe due to an error. I am currently only processing 6-8 files…
Stephen Lloyd
  • 733
  • 8
  • 21
0
votes
3 answers

Snowpipe vs Airflow for Continues data loading into Snowflake

I had a question related to Snowflake. Actually in my current role, I am planning to migrate data from ADLS (Azure data lake) to Snowflake. I am right now looking for 2 options Creating Snowpipe to load updated data Create Airflow job for same. I…
Devyani Bothra
  • 1
  • 1
0
votes
1 answer

how to extract snowflake tables schema and store-procedures using python script?

I'm intermediate with python and beginner with snowflakes. Here i'm able to connect snowflakes and fetch table data. But main problem is to extract tables schema and store-procedures from snowflakes using python script. thanks in advance.
sairamdgr8
  • 47
  • 1
  • 10
0
votes
1 answer

Snowflake- After MERGE t1 into t2, how do I ensure that next time I run MERGE, old records from t1 don't get merged again?

I'm trying to merge data in t1 that I Snowpiped from my S3 into t2. What is best practice to do with the data in t1 after they have been merged with t2? Do I delete the data in t1 or there is some metadata job tracker to ensure that the same data…
RoniFinTech
  • 25
  • 1
  • 8
0
votes
1 answer

Snowflake Validate Option does not return Failed records When using TO_DATE function in Copy Command

I have used Copy Command as below: copy into test2 from @%test2 file_format = (format_name = 'CSV') on_error = 'CONTINUE'; My file contains some Character data in Number field ( for all records) so Copy result is LOAD_FAILED and I can get fail…
PythonDeveloper
  • 289
  • 1
  • 4
  • 24
0
votes
1 answer

How to call SQL scripts from snowpipe

I have created a file with multiple DMLs. And then I call that file from SNOWSQL client. Is is possible to do that same via snowpipe so that I don't need to call the file manually every time? Below is the script that I am using in snowsql…
vasundhara guglani
  • 1
0
votes
1 answer

snowflake drop all pipes like pattern

I would like to drop all pipes in a snowflake schema that match a pattern. You can show pipes that match a pattern as shown here. Example: show pipes like '%NAME_LIKE_THIS%' in MY_DB.MY_SCHEMA However, it doesn't appear that a similar functionality…
vbp13
  • 1,040
  • 1
  • 10
  • 20
0
votes
1 answer

Maximum number of notification integrations in cloud AZURE exceeded

it turns out that after we've created 10 notification integrations, we are unable to create more and getting following error. Is it a hard limit? Is it on snowflake's or is it on azure's side? Can it be changed? Maximum number of notification…
Damian Graczyk
  • 3
  • 1
0
votes
1 answer

Snowflake object_construct unloading Key Order should be preserved as same. how to maintain the order not by alphabetical?

I use copy command of snowflake which is below returns a file with content json copy into @elasticsearch/product/sf_index from (select object_construct('id',id, alpha,'alpha')from table limit 1) file_format = (type = json, COMPRESSION=NONE),…
Sundar
  • 95
  • 1
  • 13
0
votes
1 answer

Can we have task start condition dependent on Sucess condition of PIPE in SNOWFLAKE

I have a requirement where 3 different file will be loaded to a single table with 3 different PIPE. I want target my target process to be triggered only once all 3 file has been loaded to my stage. I don't want to run my target process multiple…
rjain550
  • 105
  • 2
  • 8
0
votes
2 answers

Snowpipe auto ingest configuration looks like it triggers all pipes

I am new to Snowflake data warehouse and want to load data using snowpipe and auto ingest from an external stage (S3 bucket) which contains multiple folders (one for each database) and inside those folders there are multiple files with 1 for each…
user3165854
  • 1,505
  • 8
  • 48
  • 100
1
2
3