0

I'm currently using Azure Data Factory to load flat file data from our Gen 2 data lake into Synapse database tables. Unfortunately, we receive (many) thousands of files into timestamped folders for each feed. I'm currently using Synapse external tables to copy this data into standard heap tables.

Since each folder contains so many files, I'd like to move (or Copy/Delete) the entire folder (after processing) somewhere else in the lake. Is there some practical way to do that with Azure Data Factory?

DavidStein
  • 3,149
  • 18
  • 41
  • 62

1 Answers1

1

Yes, you can use copy activity with a wild card. I tried to reproduce the same in my environment and I got the below results:

First, add source dataset and select wildcard with folder name. In my scenario, I have a folder name pool.

enter image description here

Then select sink dataset with file path

enter image description here

The pipeline run is successful. It transferred the file from one location to another location with the required name. Look at the following image for reference.

enter image description here

enter image description here

B. B. Naga Sai Vamsi
  • 2,386
  • 2
  • 3
  • 11