1

I'm experimenting with the loop 'hack' to iterate the same workflow over a list of of values and I am trying to load that list at run-time. The list contains a number of files that I would like to process and so this list will change every time.

In the example the value is passed via the job.properties file and I can also pass it in the workflow itself as a key/value pair but than the values are hard-coded in my workflow and I would like to prevent that (as the list will change every time).

I found a similar question: OOZIE: properties defined in file referenced in global job-xml not visible in workflow.xml and the answer there explains also my initial mistakes (parameters vs properties) where I tried to pass it via a global job.xml file (which is only for properties and not for parameters) but a good solution is not provided.

One solution would be to read the parameter from a file via a shell action and capturing the output but I think there could be a more elegant way of reading this parameter in my workflow?

Community
  • 1
  • 1
R. Sluiter
  • 162
  • 1
  • 1
  • 13
  • At the end of the day, it might be simpler to develop a shell script (or Python script, or whatever) that iterates on the values, dynamically building the XML for the Oozie workflow (based on pre-defined fragment templates), then submits the WF. – Samson Scharfrichter Apr 10 '17 at 20:40
  • I appreciate the thought but then it would be better to pass the variables with -D when starting the workflow. Generating hardcoded xmls is not really a good solution in my opinion. – R. Sluiter Apr 12 '17 at 14:26
  • _"Hard-coded"_?? We clearly don't have the same definition of hard-coding. What I was suggesting is closer to "code generation", just like a BI tool generating SQL queries dynamically, depending on user choices in the UI. Anyway, it's your life, have it your way... – Samson Scharfrichter Apr 12 '17 at 19:24
  • Sorry, didn't mean to offend you, I'm just trying to understand your idea. What would be the benefit of generating several xmls over having just one 'variablized master xml' and passing the variables at launch (by using a bash/python script to iterate over the list)? – R. Sluiter Apr 13 '17 at 07:50
  • No offence there, it was simply a jaw-dropping moment :-0 The only benefit I see in creating dynamically an Oozie script, is that you have full control on the process; wheras I don't fully understand that "loop hack" and its limitations (for starters, how do you debug it??) – Samson Scharfrichter Apr 13 '17 at 12:06

0 Answers0