#PDI# Would transformation executor with reference to other transformations slow down the whole process?

Question

Here's a simple illustration of my transformation(click the link)

Step1:

Data flow would join with the meta data which contain the name of transformation file that each data would eventually be transferred to.

Here's an example of the data after the join

And then the transformation executor set the name of transformation file as variables which refers to a middle transformation

Heres's an example of the transfomation executor

Step2:

The middle transformation sends the data to certain transformation based on the variables of transformation file.

Here's an example of how middle transformation use the variables of ktr file

Step3:

Data is eventually executed in certain transformation file.

I got data containing about 30000 rows and it tooks me nearly 2.5 hours to complete the entire process.

After changing the number of copies to start of transformation executor step in Step 1 and transformation executor step in Step 2, it got better that nearly 1.5 hour was taken.

In order to further speed up the process,I have to know that if transformation executor with reference to other transformations slow down the whole process ?

Looking forward to your reply,thanks in advance!

Answer 1

It seems to me that instead of doing bulk loads of the data (inserts, updates or deletes) you are processing the rows one by one, so it takes a lot of time to process the 30.000 rows of data.

If you can think of a way to separate your rows in three groups and in the final transformations 1 to 3 you just read all the rows you need to process in that transformation and do a bulk DML operation you'll process would be much faster.