0

I have a get file names step with a Regular expression that gets 4 csv files. After that I have a text file input step which sets the fields of the csv, and read these files.

Once this step is completed a Table output step is executed.

The problem is that the text file input seems to read all 4 files in a single statement, so the table output statement inserts the rows of the 4 files. So my output table has 20 rows (5 per each file)

The expected beahivour is read one file, insert the 5 rows of the file in the output table and execute sql script which moves this table to a final table and truncate temp table. Now repeat the process for the second, third and last file.

The temporary table is deleted in every step of load a file, but final table not, it is incremental.

How can I do that in pentaho?

Kara
  • 6,115
  • 16
  • 50
  • 57
mrc
  • 2,845
  • 8
  • 39
  • 73

1 Answers1

1

Change your current job to a subjob that executes once for each incoming record.

In the new main job you need:

  • a transformation that runs Get Filenames linking to Copy Rows to Result
  • a Job entry with your current job. Configure it to execute for each row.

In the subjob you have to replace Get Filenames with Get Rows from Result and reconfigure the field that contains the filename.

Cyrus
  • 2,135
  • 2
  • 11
  • 14