1

I have an ADF pipeline that I need to run based on a csv file containing sporadic dates.

Is there anyway to implement this? my only thought is to trigger a pipeline daily that has a databricks script that checks if the current date matches a date in the file?

Thanks

Hubert Dudek
  • 1,666
  • 1
  • 13
  • 21
Alex Grimshaw
  • 11
  • 1

2 Answers2

0
  • In ADF data flow there is option list of files to specify data to be processed:

enter image description here

  • data factory is lazy evaluated so you can just add another step and filter that for example you can add "exist" (or not exists) rule and have second dataset (one when you specify dates) to check is that have to be processed:

enter image description here

  • partitioning per date also can be helpful
Hubert Dudek
  • 1,666
  • 1
  • 13
  • 21
0

Create a new pipeline in ADF. This pipeline will have lookup activity to read data(dates) from your file. After that use Foreach activity to iterate over that dates and then If activity to make a decision to actually run your original pipeline or not.

When your condition matches, inside IF activity, use Execute pipeline activity to actually run your original pipeline.

ShaikMaheer
  • 137
  • 4