0

the reqirement is simple , i have a folder having 4 txt files(1.txt,2.txt,3.txt,4.txt) . the Flow is controlled by a parameter called all or some which is of string type. This is the adf pipeline, with the activities

If i select all in the parameter, all 4 file should be processed. the requirement starts here >>

  1. IF any file is missing from the folder(for ex 2.txt and 3.txt is not present and i selected ALL in the parameter) , i need a mail saying file is 2.txt and 3.txt is missing.

  2. If i select some in the parameter, for ex 1.txt and 4.txt and if any of the file is missing 1.txt and 4.txt is missing(for example 1.txt is missing) , i need a mail with the missing file name(i.e 1.txt in our case).

Daniel Mann
  • 57,011
  • 13
  • 100
  • 120
Avenue Guy
  • 27
  • 5
  • What is the requirement? Is this to find the missing files? or to send the mail from ADF pipeline? – Aswin Jan 02 '23 at 03:54
  • Hi Aswin,both , but sending mail is not of much preference , as if i can capture missing file details in one variable or something , i can call that variable in the email Script. – Avenue Guy Jan 02 '23 at 04:10

1 Answers1

0

capture missing file details in one variable

I tried to repro this capturing missing files using azure data factory. Below is the approach.

  • Take a parameter of array type in the pipeline. During runtime, you can give the list of file names in a folder to be processed in this array parameter. enter image description here

  • Take a get metadata activity and add a dataset in the activity. Click +New in the field list and select child items as an argument.

enter image description here

  • Take a filter activity and give the array parameter value in items and write the condition to filter the missing files in condition box

item: @pipeline().parameters.AllorSome

condition: @not(contains(string(activity('Get Metadata1').output.childItems),item()))

  • I tried to run this pipeline. During run time, four file names are given to the array parameter. enter image description here

  • Get Metadata activity output has three file names.

enter image description here

  • Parameter has 4 filenames and Get meta data activity has 3 filenames. Missing file names are to be filtered out.
  • Output of filter activity:

enter image description here

Aswin
  • 4,090
  • 2
  • 4
  • 16
  • hey aswin , thank you, just for simplicity i used , 4 files , that requirement is for 12-16 files , generally we dont keep all the file names , we Give "ALL" as parameter value , by default ,the parameter value goes to the switch statement , where we have diferent Dataflow running for all the different types of files( 1.txt,2.txt,3.txt,4.txt ) – Avenue Guy Jan 02 '23 at 06:10
  • 2 ) there is already a Filter Activity i am using to filter the Text file from the folder and then to use the filter activity result to Another Filter Activity as u show. the Files are present in a Folder ,,which is in SFTP folder – Avenue Guy Jan 02 '23 at 06:42
  • hey aswin , thank you, just for simplicity i used , 4 files , that requirement is for 12-16 files , generally we dont keep all the file names , we Give "ALL" as parameter value , by default ,the parameter value goes to the switch statement , where we have diferent Dataflow running for all the different types of files( 1.txt,2.txt,3.txt,4.txt ) i am trying to use the below in Filter items :@array(if(equals(pipeline().parametrs.AllorSome,string(["1.txt","2.txt",........,"5.txt"]),pipeline().parametrs.AllorSome) error because i have the file name as 1*.txt,2*.txt,3*.txt * is Date concat. – Avenue Guy Jan 02 '23 at 11:39
  • that doesnt give any error , thankyou, but that doest solve , because as i am trying to compare it with a parameter and the parameter value passed is All so the output i am getting is "All" only – Avenue Guy Jan 02 '23 at 12:22
  • when Values is passed as all , right , the value is passed to a Switch statement , where Seperate Dataflow Activities are available for the 4 files as per the business need , this get s processed – Avenue Guy Jan 02 '23 at 12:26
  • Let us continue our discussion in this [chat](https://chat.stackoverflow.com/rooms/250789/74967032) – Aswin Jan 02 '23 at 12:35
  • yeah ashwin , so i am storing it in a Variable as default values – Avenue Guy Jan 02 '23 at 15:55
  • Try using If activity and check if parameter is "All"; in true section of if activity, Add the similar filter activity Enter the item value as variable output and for false section, Give the above (answer) approach. – Aswin Jan 02 '23 at 16:27
  • Thank you for you help, aswin , is there any way , to show non common values between 2 variables of Array type? for example Var1["1.txt","2.txt","3.txt","4.txt"] and var2["4.txt","3.txt"] answer :: ["1.tst","2.txt"] which i will store in Another Variable var3 – Avenue Guy Jan 02 '23 at 16:44
  • Yes, It is possible. Similar [SO thread](https://stackoverflow.com/questions/74486349/how-to-check-and-compare-the-file-names-that-are-inside-a-folder-datalake-usin). If you find any difficulties in this approach, create another new Question in Stack overflow on that. So, Community members will help there :) – Aswin Jan 03 '23 at 02:08