0

I have hit a brick wall, in using DL4J. When I try to train my model inside a memory map, it only results in a JVM error. I am using DL4J 1.0.0 beta-6 (beta 7 was fussing over some source code issues that I couldn't resolve)

I will post the rest of my source code if I need. I know that the way the memory map works with the JVM is a "conceptual rabbit hole". I also know that a JVM error is not resolvable.

WorkspaceConfiguration mmap = WorkspaceConfiguration.builder()
    .initialSize(1000000000)
    .policyLocation(LocationPolicy.MMAP)
    .build();

//.............................................................................

try ( MemoryWorkspace ws = Nd4j.getWorkspaceManager().getAndActivateWorkspace(mmap, "M2")) {
    model.fit(dsi);
}

What can I do about this? I have absolutely no idea how to resolve this.

Full error:

    #
    # A fatal error has been detected by the Java Runtime Environment:
    #
    #  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x000000006e753800, pid=27336, tid=25676
    #
    # JRE version: Java(TM) SE Runtime Environment (16.0.1+9) (build 16.0.1+9-24)
    # Java VM: Java HotSpot(TM) 64-Bit Server VM (16.0.1+9-24, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, windows-amd64)
    # Problematic frame:
    # [thread 12844 also had an error]
    [thread 18452 also had an error]
    [thread 2036 also had an error]
    [thread 9712 also had an error]
    [thread 21996 also had an error]
    [thread 14232 also had an error]
    [thread 23364 also had an error]
    C  [libopenblas_nolapack.dll+0xf93800]
    #
    # No core dump will be written. Minidumps are not enabled by default on client versions of Windows
    #
    # An error report file with more information is saved as:
    # C:\Local Files\Code\Java\Projects\AI\DeepLearning4J\projects\Dl4JTest\hs_err_pid27336.log
    #
    # If you would like to submit a bug report, please visit:
    #   https://bugreport.java.com/bugreport/crash.jsp
    # The crash happened outside the Java Virtual Machine in native code.
    # See problematic frame for where to report the bug.
    #
Developer
  • 98
  • 1
  • 9
  • Could you post more of the error please? If the error is an actual native JVM crash, then more on that would be what I would need to help you. – Adam Gibson Jun 25 '21 at 02:52
  • This works fine in the latest release, note that when you call net.fit(..) it actually creates its own managed workspaces internally. For performance, you would mainly want the dataset mem mapped, but then the necessary mini batches in memory. If that's what you're going for, then just map the dataset and see if createing a ViewIterator works for you. As with anything, I can't really tell you much with your upgrade issue. I would advise staying on the next release if possible. If you have issues, migrating, please open a separate topic. – Adam Gibson Jun 25 '21 at 08:26

0 Answers0