Talend internal variables

Question

I am doing a data migration project in Talend and for one of the tasks i need to process a big table with many columns and map the (old) data to a different value for the new model.
I have a unique mapping table with three columns.

Example:

 Column name | Value old | Value new
      "col 1"         1           3
      "col 1"         3           2
      "col 2"         10          7
        etc

That way I can refer using "column name" to the values related to the column I need to map.

Using old value as a Lookup in mapping table using "column name": when old value matches, then it return the new value.

I could do this manually for each column, but there are hundreds so that would mean at least one hash repeated for every column.

I am stumped as to how to do this more simply and only once for every possible column in the original table.

Right now I am doing a HUGE tMap and using lots of lookups: one for each column.

Any ideas are appreciated.

-

Some extra ideas I had:
1) Is there any way to know the NAME OF THE LINK joining two components? I could then just reuse the same connection and filter on the tMap more easily

done, took me long enough :) – Juan Jose Mar 13 '14 at 17:02 — Juan Jose, Mar 13 '14 at 17:02

score 2 · Accepted Answer · answered Mar 13 '14 at 17:02

I could solve it using tMemorizeRows, saving the whole mapping table to memory using a tJavaFlex and then reading them using a routine.

What i did was, memorize the whole table, i only had one table, with three columns, the first indicating the type of mapping i had to do, the second being the original code, and the third one being the replacement code.

Once memorized the data, i used a tJavaFlex to create three lists, one for each column and then move those lists to the global variables map.

Then i created a routine that receives all three lists plus two codes, the first one is the mapping name (first column filter) and the second one is the original code (second column filter). Using both i could pinpoint the position of the replacement code, and return it.

Finally, when reading the main flow of data, using a tMap, i created a variable for each distinct value of the first column (that is, one for each type of mapping) and then called the routine i developed using the oriinal code + the mapping name.

The code i used in the tMemorizeRows was:

START CODE

java.util.Set<String> iLista;
java.util.List<String> lLOV=new java.util.ArrayList<String>();
java.util.List<String> lS6=new java.util.ArrayList<String>();
java.util.List<String> lS8=new java.util.ArrayList<String>();

MAIN CODE

lLOV.add(LOV_tMemorizeRows_1[icount]);
lS6.add(S6_NAME_tMemorizeRows_1[icount]);
lS8.add(S8_NAME_tMemorizeRows_1[icount]);

END CODE

globalMap.put("lLOV",lLOV);
globalMap.put("lS6",lS6);
globalMap.put("lS8",lS8);

ROUTINE CODE

/*
 * Toma los valores generados de la LOV y matchea con S8, requiere la ejecucion del job
 * Genericos\GeneradorLOV
 * 
 * {talendTypes} String | String
 * 
 * {Category} MigracionDatos
 * 
 * {param} string(entrada.LOV) Identifica el tipo de LOV
 * 
 * {param} string(entrada.S6_NAME) Indica el valor del campo en Siebel 6 
 * 
 * {param} ((java.util.List<String>) globalMap.get("lLOV"))
 * 
 * {param} ((java.util.List<String>) globalMap.get("lS6"))
 * 
 * {param} ((java.util.List<String>) globalMap.get("lS8"))


 * 
 */
   public static String calculaLOV(String CampoLOV, String CampoS6, java.util.List<String> listaLOV, java.util.List<String> listaS6,java.util.List<String> listaS8   ) {
       /*
        * java.util.List<String> listaLOV = ( java.util.List<String>) globalMap.get("lLOV");
        * java.util.List<String> listaS6 = ( java.util.List<String>) globalMap.get("lS6");
        * java.util.List<String> listaS8 = ( java.util.List<String>) globalMap.get("lS8");
        */

       String C1 = CampoLOV;
       String C2 = CampoS6;

       int posicionC1 = listaLOV.indexOf(C1);

       // encontró el LOV
       if(posicionC1 >= 0) {

        java.util.List<String> listaS6_Iterada = new java.util.ArrayList<String>();

        // Genera la lista intermedia con los valores
        for (int contador = posicionC1; contador < listaLOV.size() ; contador++) {
            listaS6_Iterada.add(listaS6.get(contador));
            if(!listaLOV.get(contador).toString().equals(C1)) {break;}
        }

        int posicionC2 = listaS6_Iterada.indexOf(C2);
        if(posicionC2 >= 0) {

            int posicionFinal = posicionC1 + posicionC2;

            return listaS8.get(posicionFinal);
        } else {
            return "";
        }
       } else {
        return "";
       }
   }

Talend internal variables

1 Answers1