1

I am having a runtime argument set at namespace which is business_date: ${logicalStartTime(yyyy-MM-dd)} . I am using this argument in my pipeline and want to use the same in other pipeline. There are many pipelines back to back and I want to the value to be same throughout the pipelines once calculated in the first pipeline. suppose the value is calculates as '2020-08-20 20:14:11' and once the pipeline one succeeded i am passing this argument to pipeline 2, but as this arguments are defined at namespace level it is getting overrided when pipeline2 starts.

How can I have prevent this value to be calculated again ?

SUDHIR GARG
  • 75
  • 9
  • 2
    you can setup one pipeline to trigger another pipeline. You can do this from the pipeline detail page, there's a button on the side of the screen. As part of the trigger, you should be able to configure arguments that should be passed from pipeline to pipeline2. – Albert Shau Aug 20 '20 at 18:28
  • 1
    Thanks for the answer Albert . I am also doing the same but the bug in above approach is that suppose if your PIpeline 1 which is "DataFusionQuickstar" triggers at 23:55 on 2020-05-22 then normally the second pipeline which is "DataFusionQuickstart2" pipeline will get trigger on next day which would be say 00:02 on 2020-05-23. So in these kind of scenarios the business_date used by the pipeline1 would be 2020-05-22 but for Pipeline2 it would be 2020-05-23 because these triggers pass the expression as argument value not the actual value which is calculated in Pipeline1. – SUDHIR GARG Aug 27 '20 at 09:17

3 Answers3

1

As it was commented before, you could set up one pipeline to trigger another pipeline; you can set a runtime variable in the first pipeline and this variable will be set in the triggered pipelines. You can create inbound trigger by following the next steps:

  1. Once you have created your pipelines, select the last pipeline you want to be run. In my case I have DataFusionQuickstart2 pipeline.

enter image description here

  1. Into the pipeline application, on the left side, click on "Inbound triggers" -> "Set pipeline triggers" and you will see the pipes which you can be triggered. Check the event that will trigger the DataFusionQuickstart2 pipeline from DataFusionQuickstart and enable it.

enter image description here

  1. If you take a look to the previous pipeline DataFusionQuickstar you will see, into the outbound trigger option (right side), the pipelines that will be triggered by DataFusionQuickstar.

enter image description here

  1. Finally set your runtime argument.

enter image description here

Additional information

In this post, it was mentioned that there are three ways you can set the runtime argument of a pipeline:

  1. Argument Setter plugin (You can write down that value in a file into the first pipeline. In all subsequent pipelines, create a parameter to read that file.)
  2. Passing runtime argument when starting a pipeline (The one described above)
  3. Setting Preferences (It provides the ability to save configuration information at various levels of the system, including the CDAP instance, namespace, application, and program levels.)
July
  • 442
  • 4
  • 11
  • Thanks for the answer julio . I am also doing the same but the bug in above approach is that suppose if your PIpeline 1 which is "DataFusionQuickstar" triggers at 23:55 on 2020-05-22 then normally the second pipeline which is "DataFusionQuickstart2" pipeline will get trigger on next day which would be say 00:02 on 2020-05-23. So in these kind of scenarios the business_date used by the pipeline1 would be 2020-05-22 but for Pipeline2 it would be 2020-05-23 because these triggers pass the expression as argument value not the actual value which is calculated in Pipeline1. – SUDHIR GARG Aug 23 '20 at 14:07
  • As it was mentioned in the @adiideas answer, you can write down that value in a file into the first pipeline. In all subsequent pipelines, create a parameter to read that file. https://cloud.google.com/data-fusion/docs/tutorials/reusable-pipeline#read_from – July Sep 03 '20 at 17:46
0

You can write down that value in a file in first pipeline. In all subsequent pipelines, create a parameter to read that file. That way, objective should be achieved.

adiideas
  • 25
  • 4
0

@Sudhir, you may explore PREFERENCES. https://cdap.atlassian.net/wiki/spaces/DOCS/pages/477561058/Preferences+HTTP+RESTful+API

You have set the variable at namespace level and as per your finding it is getting evaluated each time its being used.

Can you try setting it at Application level? And pass it to next pipeline. I believe in that case, it should be evaluated only once in that specific application (pipeline) and thereafter value would be passed.

Preference is also available at program level.

adiideas
  • 25
  • 4