0

In DL4J, is there a way to access the columns of preprocessed data after at the scoring step? I have a case where I have a csv of data that includes fields that are not used by the neural net for prediction, but they are important to include in my output after the predictions are made.

Before I train my model, I have been using this step for preprocessing to remove the columns prior to training:

TransformProcess transformProcess = 
    new transformProcess.Builder(schema).removeColumns(columnsToOmit).build());
RecordReader transformProcessRecordReader = 
    new TransformProcessRecordReader(recordReader, transformProcess);

The problem I am running into is that after this transformation, I can of course train or make predictions, but I can no longer access those columns that are removed.

Is there a way to "ignore" the columns rather than removing them so that I can access them after the model makes a prediction?

In my debugger I can see some protected fields that show the data is still there but i'm really trying to avoid a custom implementation of iterator if there is a simpler way to do this.

Isaiah Gillenwater
  • 1
  • Have you tried looking at the TransformProcess filtering steps? – Adam Gibson Jul 24 '21 at 06:36
  • I have. From the site: "The Add a filter operation, based on the specified condition. If condition is satisfied (returns true): remove the example or sequence If condition is not satisfied (returns false): keep the example" That will remove columns. The transformation to remove columns is what I have included in the question. https://deeplearning4j.konduit.ai/datavec/transforms – Isaiah Gillenwater Jul 26 '21 at 13:13
  • @AdamGibson If I am misunderstanding how I should be using that step I am of course open to further clarification – Isaiah Gillenwater Jul 27 '21 at 20:26

0 Answers0