Zero Columns In DataVec Matrix When Running Deeplearning4J Classification

Question

I'm attempting to run the deeplearning4j quickstart classification problem.

I'm using Windows 8 64 bit, Oracle JDK 8, IntelliJ 2017.2.5, and Maven 3.3.9.

I got past the Canova issue that plagued me earlier. Now I have a new problem at runtime.

Here's the pom.xml that I constructed for myself. I did not take @newOne advice to use the example pom.xml, because I want to make sure I can create a stripped down one from scratch without depending on their modules.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>deeplearning4j.example</groupId>
    <artifactId>deeplearning4j</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <maven.test.skip>false</maven.test.skip>
        <timestamp>${maven.build.timestamp}</timestamp>
        <maven.build.timestamp.format>MM-dd-yyyy HH:mm</maven.build.timestamp.format>
        <timestamp>${maven.build.timestamp}</timestamp>
        <maven.build.timestamp.format>MM-dd-yyyy HH:mm</maven.build.timestamp.format>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <java.version>1.8</java.version>
        <deeplearning4j.version>0.9.1</deeplearning4j.version>
        <nd4j.version>0.9.1</nd4j.version>
    </properties>

    <dependencies>
        <!-- https://mvnrepository.com/artifact/org.deeplearning4j/deeplearning4j-core -->
        <dependency>
            <groupId>org.deeplearning4j</groupId>
            <artifactId>deeplearning4j-core</artifactId>
            <version>${deeplearning4j.version}</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.deeplearning4j/deeplearning4j-nlp -->
        <dependency>
            <groupId>org.deeplearning4j</groupId>
            <artifactId>deeplearning4j-nlp</artifactId>
            <version>${deeplearning4j.version}</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.deeplearning4j/deeplearning4j-ui-components -->
        <dependency>
            <groupId>org.deeplearning4j</groupId>
            <artifactId>deeplearning4j-ui-components</artifactId>
            <version>${deeplearning4j.version}</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.deeplearning4j/deeplearning4j-ui-model -->
        <dependency>
            <groupId>org.deeplearning4j</groupId>
            <artifactId>deeplearning4j-ui-model</artifactId>
            <version>${deeplearning4j.version}</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.deeplearning4j/deeplearning4j-nearestneighbors-model -->
        <dependency>
            <groupId>org.deeplearning4j</groupId>
            <artifactId>deeplearning4j-nearestneighbors-model</artifactId>
            <version>${deeplearning4j.version}</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.nd4j/nd4j-api -->
        <dependency>
            <groupId>org.nd4j</groupId>
            <artifactId>nd4j-native-platform</artifactId>
            <version>${nd4j.version}</version>
        </dependency>
        <dependency>
            <groupId>log4j</groupId>
            <artifactId>log4j</artifactId>
            <version>1.2.17</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.deeplearning4j/deeplearning4j-nn -->
        <dependency>
            <groupId>org.deeplearning4j</groupId>
            <artifactId>deeplearning4j-nn</artifactId>
            <version>${deeplearning4j.version}</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/jfree/jfreechart -->
        <dependency>
            <groupId>jfree</groupId>
            <artifactId>jfreechart</artifactId>
            <version>1.0.13</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.datavec/datavec-api -->
        <dependency>
            <groupId>org.datavec</groupId>
            <artifactId>datavec-api</artifactId>
            <version>0.9.1</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.deeplearning4j/dl4j-test-resources -->
        <dependency>
            <groupId>org.deeplearning4j</groupId>
            <artifactId>dl4j-test-resources</artifactId>
            <version>0.4.0</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.12</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.mockito</groupId>
            <artifactId>mockito-all</artifactId>
            <version>1.10.19</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-source-plugin</artifactId>
                <version>3.0.1</version>
                <executions>
                    <execution>
                        <id>attach-sources</id>
                        <goals>
                            <goal>jar</goal>
                        </goals>
                    </execution>
                    <execution>
                        <id>attach-javadocs</id>
                        <goals>
                            <goal>jar</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.5.1</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-site-plugin</artifactId>
                <version>3.4</version>
                <dependencies>
                    <dependency>
                        <groupId>org.apache.maven.wagon</groupId>
                        <artifactId>wagon-http-lightweight</artifactId>
                        <version>1.0</version>
                    </dependency>
                </dependencies>
            </plugin>
            <plugin>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>buildnumber-maven-plugin</artifactId>
                <version>1.4</version>
                <executions>
                    <execution>
                        <phase>generate-resources</phase>
                        <goals>
                            <goal>create-metadata</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <attach>true</attach>
                    <!--make it available for jar/war classpath resource -->
                    <addOutputDirectoryToResources>true</addOutputDirectoryToResources>
                    <revisionOnScmFailure>na</revisionOnScmFailure>
                    <doCheck>false</doCheck>
                    <doUpdate>false</doUpdate>
                    <revisionOnScmFailure>true</revisionOnScmFailure>
                    <format>{0,date,yyyy-MM-dd_HH-mm}_{1}</format>
                    <items>
                        <item>timestamp</item>
                        <item>${user.name}</item>
                    </items>
                </configuration>
            </plugin>
        </plugins>
    </build>

</project>

IntelliJ has no red in my Maven window. I can build and package the code successfully without error messages.

When I run the classification problem I get this message at runtime:

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Exception in thread "ADSI prefetch thread" java.lang.RuntimeException: org.nd4j.linalg.exception.ND4JIllegalStateException: Invalid shape: Requested INDArray shape [50, 0] contains dimension size values < 1 (all dimensions must be 1 or more)
    at org.deeplearning4j.datasets.iterator.AsyncDataSetIterator$AsyncPrefetchThread.run(AsyncDataSetIterator.java:442)
Caused by: org.nd4j.linalg.exception.ND4JIllegalStateException: Invalid shape: Requested INDArray shape [50, 0] contains dimension size values < 1 (all dimensions must be 1 or more)
    at org.nd4j.linalg.factory.Nd4j.checkShapeValues(Nd4j.java:5022)
    at org.nd4j.linalg.factory.Nd4j.create(Nd4j.java:5012)
    at org.nd4j.linalg.factory.Nd4j.create(Nd4j.java:4965)
    at org.nd4j.linalg.factory.Nd4j.create(Nd4j.java:4093)
    at org.deeplearning4j.datasets.datavec.RecordReaderMultiDataSetIterator.convertWritables(RecordReaderMultiDataSetIterator.java:377)
    at org.deeplearning4j.datasets.datavec.RecordReaderMultiDataSetIterator.convertFeaturesOrLabels(RecordReaderMultiDataSetIterator.java:271)
    at org.deeplearning4j.datasets.datavec.RecordReaderMultiDataSetIterator.nextMultiDataSet(RecordReaderMultiDataSetIterator.java:234)
    at org.deeplearning4j.datasets.datavec.RecordReaderMultiDataSetIterator.next(RecordReaderMultiDataSetIterator.java:177)
    at org.deeplearning4j.datasets.datavec.RecordReaderDataSetIterator.next(RecordReaderDataSetIterator.java:306)
    at org.deeplearning4j.datasets.datavec.RecordReaderDataSetIterator.next(RecordReaderDataSetIterator.java:393)
    at org.deeplearning4j.datasets.datavec.RecordReaderDataSetIterator.next(RecordReaderDataSetIterator.java:51)
    at org.deeplearning4j.datasets.iterator.AsyncDataSetIterator$AsyncPrefetchThread.run(AsyncDataSetIterator.java:423)

What should I adjust in the input to get past this matrix with zero columns?

How look your pom.xml? Its must look like this: https://github.com/deeplearning4j/dl4j-examples/blob/master/pom.xml without dependecy to canova directly. — newOne, Nov 20 '17 at 12:38
I wanted to set up a Maven pom.xml by hand from scratch. I wasn't referring to the example pom.xml, because it looks like a lot of machinery. Surely it isn't necessary to use all your example modules and Maven pom to do a simple problem. I'll admit that it could be a hole in my Maven skills. — duffymo, Nov 20 '17 at 12:50

score 0 · Answer 1 · answered Nov 20 '17 at 16:31

The pom I posted above did the job.

Turns out my next issue was with data. The training and eval files I had were not correct.

Once I had the right files I got this result in my console window:

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Evaluate model....

Examples labeled as 0 classified by model as 0: 100 times
Examples labeled as 1 classified by model as 1: 100 times


==========================Scores========================================
 # of classes:    2
 Accuracy:        1.0000
 Precision:       1.0000
 Recall:          1.0000
 F1 Score:        1.0000
========================================================================
****************Example finished********************

As an added bonus, I got two lovely popup windows:

Training data

Eval data

I'm glad to have gotten off the dime with deeplearning4j. I hope my simplified pom.xml can help someone else out.

Well done, deeplearning4j team! I look forward to diving deeper.

Zero Columns In DataVec Matrix When Running Deeplearning4J Classification

1 Answers1