0

I'm looking to try to pick up all parquet files from an s3 bucket that have been placed into partitioned sub-folders by date.

In the past I've used snowpipe with a sort of 1-1 relationship, one sub-folder to one table; but I would be interested to know if it is possible to crawl over partitioned data to a single table.

Many thanks!

Sam Richardson
  • 15
  • 3

2 Answers2

0

Short answer: Yes!

With COPY INTO you can load a particular file, a whole folder or all sub-folders within a certain folder. All you need to do is to adjust your path accordingly. Just mention the path in the FROM clause and all subfolders will be copied.

copy into mytable
from @my_stage/your_main_folder/;

Docs: https://docs.snowflake.com/en/sql-reference/sql/copy-into-table.html

Edit: There are variations possible. Also the stage itself can point to a certain main folder already and you do not need to extend the COPY INTO path.

Marcel
  • 2,454
  • 1
  • 5
  • 13
0

Yes you can use sub folders as part of Snowflake Stage which you want to crawl in the Pipe Definition.

https://docs.snowflake.com/en/user-guide/data-load-snowpipe-auto-s3.html#step-3-create-a-pipe-with-auto-ingest-enabled

Make sure that the S3 Stage has the Sub folder path which you want to crawl.

Anshul Thakur
  • 484
  • 2
  • 5