0

I need guidence on implementing WHILE loop with Kettle/PDI. The scenario is

(1) I have some (may be thousand or thousands of thousand) data in a table, to be validated with a remote server.

(2) Read them and loopup to the remote server; I use Modified Java Script for this as remote server lookup validation is defined in external Java JAR file (I can use "Change number of copies to start... option on Modified java script and set to 5 or 10)

(3) Update the result on database table. There will be 50 to 60% connection failure cases each session.

(4) Repeat Step 1 to step 3 till all gets updated to success

(5) Stop looping on Nth cycle; this is to avoid very long or infinite looping, N value may be 5 or 10.

How to design such a WhILE loop in Pentaho Kettle?

RVandakar
  • 81
  • 1
  • 5
  • 16
  • Try making the Database transactional. when a transformation ends without error or you want to perform a roll back when an error occurs somewhere. This is possible simply by enabling the **"Make the transformation database transactional"**. Check [this link](http://wiki.pentaho.com/display/EAI/Database+transactions+in+jobs+and+transformations). Check if this solve your query... – Rishu Shrivastava Apr 22 '15 at 06:39
  • No Rishu, its not about making database transactional. We will encounter network related errors while doing lookup with remote server while validating each record from our database table. e.g. if source is 1000 records, hardly 450+ records will get succeed in 1st cycle and rest will fail with various reasons, including network connectivity. So we need to implement WHILE loop kind of structure, so that 4 to 5 looping cycles can be executed till all records gets validated. – RVandakar Apr 23 '15 at 06:02

1 Answers1

2

Have you seen this link? It gives a pretty well detailed explanation of how to implement a while loop.

You need a parent job with a sub-transformation for doing a check on the condition which will return a variable to the job on whether to abort or to continue.

Dezzie
  • 934
  • 3
  • 18
  • 35