TransformProcess transform data while using DataSetIterator

Question

I have a CSV dataset with both numerical and nominal attributes. I defined Schema for dataset that list all possible values for nominal attributes. After that I created TransformProcess to convert nominal values to numeric values using CategoricalToOneHotTransform. How can I use this TransformProcess on RecordReaderDataSetIterator to prepare for my neural network ?

        Schema schema = new Schema.Builder()
        .addColumnInteger("age")
        .addColumnCategorical("workclass", "Private", "Self-emp-not-inc", "Self-emp-inc", "Federal-gov", "Local-gov", "State-gov", "Without-pay", "Never-worked")
        .addColumnInteger("fnlwgt")
        .addColumnCategorical("education", "Bachelors", "Some-college", "11th", "HS-grad", "Prof-school", "Assoc-acdm", "Assoc-voc", "9th", "7th-8th", "12th", "Masters", "1st-4th", "10th", "Doctorate", "5th-6th", "Preschool")
        .addColumnInteger("education-num")
        .addColumnCategorical("marital-status", "Married-civ-spouse", "Divorced", "Never-married", "Separated", "Widowed", "Married-spouse-absent", "Married-AF-spouse")
        .addColumnCategorical("occupation", "Tech-support", "Craft-repair", "Other-service", "Sales", "Exec-managerial", "Prof-specialty", "Handlers-cleaners", "Machine-op-inspct", "Adm-clerical", "Farming-fishing", "Transport-moving", "Priv-house-serv", "Protective-serv", "Armed-Forces")
        .addColumnCategorical("relationship", "Wife", "Own-child", "Husband", "Not-in-family", "Other-relative", "Unmarried")
        .addColumnCategorical("race", "White", "Asian-Pac-Islander", "Amer-Indian-Eskimo", "Other", "Black")
        .addColumnCategorical("sex", "Female", "Male")
        .addColumnInteger("capital-gain")
        .addColumnInteger("capital-loss")
        .addColumnInteger("hours-per-week")
        .addColumnCategorical("native-country", "United-States", "Cambodia", "England", "Puerto-Rico", "Canada", "Germany", "Outlying-US(Guam-USVI-etc)", "India", "Japan", "Greece", "South", "China", "Cuba", "Iran", "Honduras", "Philippines", "Italy", "Poland", "Jamaica", "Vietnam", "Mexico", "Portugal", "Ireland", "France", "Dominican-Republic", "Laos", "Ecuador", "Taiwan", "Haiti", "Columbia", "Hungary", "Guatemala", "Nicaragua", "Scotland", "Thailand", "Yugoslavia", "El-Salvador", "Trinadad&Tobago", "Peru", "Hong", "Holand-Netherlands")
        .addColumnCategorical("class", ">50K", "<=50K")
        .build();

    TransformProcess tp = new TransformProcess.Builder(schema)
        .transform(new CategoricalToOneHotTransform("workclass"))
        .transform(new CategoricalToOneHotTransform("education"))
        .transform(new CategoricalToOneHotTransform("marital-status"))
        .transform(new CategoricalToOneHotTransform("occupation"))
        .transform(new CategoricalToOneHotTransform("relationship"))
        .transform(new CategoricalToOneHotTransform("race"))
        .transform(new CategoricalToOneHotTransform("sex"))
        .transform(new CategoricalToOneHotTransform("native-country"))
        .transform(new CategoricalToIntegerTransform("class"))
        .build();

    Schema outputSchema = tp.getFinalSchema();

    int numLinesToSkip = 0;
    String delimiter = ",";
    CSVRecordReader recordReader = new CSVRecordReader(numLinesToSkip, delimiter);
    recordReader.initialize(new FileSplit(Paths.get("..\\adult.data").toFile()));


    int labelIndex = outputSchema.getColumnNames().size() - 1;
    int numClasses = 2;
    int batchSize = 2000;

    RecordReaderDataSetIterator iterator = new RecordReaderDataSetIterator(recordReader, batchSize, labelIndex, numClasses);

    DataSet allData = iterator.next();
    allData.shuffle();
    SplitTestAndTrain testAndTrain = allData.splitTestAndTrain(0.65);

score 3 · Accepted Answer · answered Jun 04 '17 at 02:24

3

Take a look at the: https://github.com/deeplearning4j/DataVec/blob/master/datavec-api/src/main/java/org/datavec/api/records/reader/impl/transform/TransformProcessRecordReader.java

The RecordReaderDataSetItertor takes in a record reader and handles the vectorization process. This wraps a record reader and outputs a transformed record which will then get fed to the recordreaderdatasetiterator.

answered Jun 04 '17 at 02:24

Adam Gibson

3,055
1
10
12

Thanks. But i saw that this is not included at 0.8 release. – serdar Jun 04 '17 at 13:54
Use snapshots for now. Just add this declaration: https://github.com/deeplearning4j/dl4j-examples/blob/master/dl4j-examples/pom.xml#L16 You can also copy and paste it. It's a standalone class. – Adam Gibson Jun 04 '17 at 23:21
@AdamGibson Here's the code I tried exactly what you mentioned: `RecordReader transformProcessRecordReader = new` `TransformProcessRecordReader(reader,transformProcess);` `DataSetIterator iterator = new RecordReaderDataSetIterator(transformProcessRecordReader,labelIndex,numClasses,batchSize);` But it throws error `next() is unsupported` – Rahul Raj May 16 '18 at 17:15
I need to get the entire data set so that I can split for training. – Rahul Raj May 16 '18 at 17:29
We would need a lot more context, please post a fresh new question here or come in to our gitter channel. – Adam Gibson May 16 '18 at 23:34
I explained in detail here: https://github.com/deeplearning4j/deeplearning4j/issues/5220 – Rahul Raj May 17 '18 at 04:40

TransformProcess transform data while using DataSetIterator

1 Answers1