-1

I need to built generic workflow solution catering to data loads of different types. Like a)from a flat file to flat file-(can be fixed width file or delimited or excel file) b) from flat file to database-(can be any database or any file type type like before c)from database to database

The metadata for above can be configured via setting up web based GUIs.

Metadata: a)File type b)Columns c)Filter logic d)Lookup if yes condition e)Load type:Insert,update or UPSERT based on primary keys f)Database type g)JNDI for DB h)Business Rules-at the record level or aggregate level. ....

Recently,I have watched the video "https://skillsmatter.com/skillscasts/5115-how-amia-used-pentaho" and very much believe that the above workflow can be created using PDI with minimal coding.

Would anyone be able to help to provide sample examples to achieve the same?

Regards, Dikesh Shah.

dikesm
  • 1
  • 2
  • I can confirm that can be done. We built a metadata driven system which reads from ANY pdi data source and writes the data into hadoop. Adding new feeds is purely metadata in the database, no changes to the transforms. – Codek Jan 14 '15 at 14:19
  • Thanks Codek.Would appreciate if you can provide a working example if possible? – dikesm Jan 26 '15 at 09:30
  • I'm sorry, but no; This was a massive project. I didn't say it was easy! But it can definitely be done. – Codek Jan 27 '15 at 10:13
  • Thanks codek for the reply.Any hinters or clue would be highly appreciated. – dikesm Feb 17 '15 at 06:54

1 Answers1

0

Pentaho is a metadata driven tool. I am sure you can build a solution like you are looking for. PDI or kettle provides you with various steps like "Set Variables" and "Get Variables". Check out Dynamic naming of excel sheets using pentaho kettle . Also check Pentaho wiki.

Also in the Tranformation settings, you will find an option to add Parameters. Check the images below for Adding Parameter:

enter image description here

What you need to do from the UI is to dynamically pass the values to the Kettle by adding Parameters and use the get and set variables to do the rest of your business logic.

Hope this helps :). If required i can provide you a sample solution.

Community
  • 1
  • 1
Rishu Shrivastava
  • 3,745
  • 1
  • 20
  • 41