2

I have encountered a use case where i am fetching rows from tMyslInput and then iterating it one by one to complete a subjob for each of the rows retrieved.

tMysqlInput-----> iterate -------> (job having multiple components such as writing a file, logging it, entry into the database and different processes like this, which is a complete process in itself).

Problem is that since the subjob after iterate link itself takes care of everything, i just want to fork as many subjobs as number of rows fetched from tMysqlInput with different context parameters.

So i tried to do following

tMysqlInput ------>iterate(*n, where n is number of rows fetched)----->(job)

But here what is happening , threads are reading each other context variables hence ending up writing similar context in similar files, same db entry etc..

I want to parrallelize the child job depending on number of rows fetched with threads being in synchronize.

tMysqlInput query lets say, select file_id, input_path , output_path from some table where status='copied';

lets say 4 tuples i got then i want to iterate 4 tuples at the same time. Just execute the child job and let the child job execute on its own.

thanks

Nitish Sharma
  • 331
  • 4
  • 7
  • possible duplicate of [Running parrallel jobs in talend](http://stackoverflow.com/questions/21309946/running-parrallel-jobs-in-talend) – ydaetskcoR Feb 10 '14 at 09:45
  • Is this not just the same question as http://stackoverflow.com/questions/21309946/running-parrallel-jobs-in-talend ? Did it not solve your issue? – ydaetskcoR Feb 10 '14 at 09:45

1 Answers1

3

try this -

1) Click on Iterate Link - and in component properties Tab you can see Basic Settings - Enable Parallel Execution checkbox (once you check this checkbox) you can enter values of number of iterations you want to run in parallel. This could be number of rows returned by tMysqlInput component (however total number of rows variable will have value AFTER execution of tMysqlInput - globalMap.get("tMysqlInput_X_NB_LINE"))

2) you can pass context variables values in sub jobs - for this first you have to define context variables in your sub job, and then once you have it after iterate link tSubJob click on component properties tab and you will see context Param (table/grid) where you click on + symbol to select context variable and assign its value.

garpitmzn
  • 1,001
  • 6
  • 9