1

I am trying to load data through snowpipes and somehow all the files are not getting loaded, these files are from different sources including automated jobs, manual uploads, data extracts. The one thing I am seeing in common is - the files that do not have md5 are not ingested automatically, while the files that do are. So wanted to confirm if anybody knows that md5 is necessary for a file to be auto-ingested?

Thanks

Ashish Singh
  • 25
  • 4

2 Answers2

0

No, it is not necessary to have a MD5 (file) for auto-ingest. It relies on notification services (SNS, SQS, Event Grid).

I suggest you to query SYSTEM$PIPE_STATUS to understand how many files are pending, timestamp of the oldest file in the queue, timestamp of last forwarded notification message:

https://docs.snowflake.com/en/user-guide/data-load-snowpipe-auto-s3.html#system-pipe-status-output

Gokhan Atil
  • 9,278
  • 2
  • 11
  • 24
  • Thanks for replying. The last forwarded time stamp is the one of the file which had md5, after that the messages are not being forwarded. Just to check I added a file manually(which gets md5 auto-generated) and that was forwarded too.Is there any option for this check that I can enable or disable or does it not matter at all ? I mean the md5 content for the files – Ashish Singh Jun 05 '20 at 12:53
0

Turns out ADLS gen2 API needs explicit flush close in order to create a notification for snowflake which leads to auto ingest. Added this to the azure function and it seems to be working fine now

Ashish Singh
  • 25
  • 4