0

I have to read 10 files from a folder in blob container with different schema(most of the schema among the table macthes) and merge them into a single SQL table file 1: lets say there are 25 such columns

enter image description here

file 2: Some of the column in file2 matches with columns in file1

enter image description here

file 3:

enter image description here

output: a sql table

enter image description here

How to setup a pipeline in azure data factory to merge these columns into a single SQL table.

my approach: get Metadata Activity---> for each childitems--- copy activity

for the mapping--- i constructed a json that containes all the source/sink columns from these files

enter image description here

enter image description here

enter image description here

enter image description here

bin
  • 11
  • 3

1 Answers1

0

You can create a JSON file which contains your each source file name and Tabular Translator. Then use Lookup activity to get this file's content(Don't check first row only). Loop this array in For Each activity and pass source file name in your dataset. Finally, create a copy data activity and use Tabular Translator as your mapping.

Steve Johnson
  • 8,057
  • 1
  • 6
  • 17