0

We are invoking ADF pipeline based on event based trigger. Is there a way to trigger this pipeline only when a file arrives in both of these child folders

e.g ParentFolder -- ChildFolder1 -- ChildFolder2

Now we would like to trigger our pipelines only if a new file arrives in both of these folders.i.e ChildFolder1 and ChildFolder2

skoundin
  • 192
  • 1
  • 6
  • 19
  • 1
    My gut instinct it that this is not possible, because each Event is independent, and the Trigger is tied to a single event. Are the files related/consistently named? Can you use GetMetadata to look up whether the 2nd file exists? If not, then I believe you'll need an external mechanism for tracking these events. A couple logic apps and a SQL database would be a simple way to approach the problem. – Joel Cochran Nov 19 '21 at 15:10

1 Answers1

0

There is no out of box approach this. I can think of the below alternatives.

First Approach

You can set a trigger at the ChildFolder2

You can use a 'lookup activity' or 'Get Metadata Activity' which fetches the file with the name in the ChildFolder1 - See whether the file is created at the ChildFolder1.

If you would like to check after some time - delay it say for 10- 15 minutes. You could make use of the Wait activity

Now, if the file is existent - then you could continue with the rest of the execution of the pipeline. If the file is not created in Childfolder1 - then you could end the pipeline with no activity carried out.

The pipeline will eventually be triggered when the file is created in childfolder2. The Execution flow changes based on an If activity and existence of the file in the childfolder1.

Second Approach

If you don't have filename - and would like to get a file dynamically time created.

In the same way as above - you could set a Event trigger at the childfolder2.

In the pipeline execution you filter the files based out of the timestamp of the file pipeline start. This is slightly tricky.

You do a GetMetada for the childfolder1 and filter it using the foreach and if condition. (get the latest added file in a folder [Azure Data Factory])

If there is any file then execute the pipeline with rest of the activities else you could end the pipeline execution.

Satya V
  • 3,811
  • 1
  • 6
  • 9