3

we're facing a memory leak using a simple, simple, simple code as follows. The code is intended to get files from a source, use each file to do something and go on. This simple code uses always the same file but the behaviour is unchanged.

package it.datapump.main;

import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;


public class TifReader {

public static void main (final String[] a){

    for (int i = 0; i < 100000; i++) {
        try {
            getBytesFromFile(new File("test.tif"));
            Thread.sleep(1000);
            System.gc() ;
        } catch (Exception ex) {
        }
    }
}

public static byte[] getBytesFromFile(File file) throws IOException {
    InputStream is = new FileInputStream(file);
    long length = file.length();

    byte[] bytes = new byte[(int)length];
    int offset = 0;
    int numRead = 0;
    while (offset < bytes.length
           && (numRead=is.read(bytes, offset, bytes.length-offset)) >= 0) {
        offset += numRead;
    }
    is.close();

    // Do something with the read bytes
    //

    if (offset < bytes.length) {
        throw new IOException("Could not completely read file "+file.getName());
    }
    return bytes;
}
}

Now... we just can't see a valid reason for this code to consume memory up to the top and in the end throws an OutOfMemoryError exception.
Any idea?

Something More
The problem arises using Java Development Kit Version 6 Update 23 but it does not on JRE 1.7

Mr.Gate
  • 419
  • 4
  • 12
  • Do I get it right - exactly *this* snippet creates an OOM error, the OOM is not related to the commented "Do something with read bytes" part? And no exceptions being thrown? – Andreas Dolk Feb 02 '11 at 09:24
  • @Andreas_D, no interest in analyzing the snippet. This is totally wrong way of reading the file. – Vladimir Ivanov Feb 02 '11 at 09:25
  • @Vladimir: IMHO, you need to respect the question posted by the OP rather than forcing your way through. And BTW, it's not a *wrong* way, it's just a *round-about* way. – Sanjay T. Sharma Feb 02 '11 at 09:30
  • How many iterations (if any) does it get through before running OOM? – dogbane Feb 02 '11 at 09:33
  • No, the OOM isn't concerning the "Do something" and so on, just because we realized this simple code to verify the OOM happens all the same even if you do nothing and just read the file. – Mr.Gate Feb 02 '11 at 09:34
  • I just can't say how many iterations but it obviusly depends on how many memory you give to the JVM. But a few thousands of iterations goes over 500MB of used RAM and the process just can't reach the end. – Mr.Gate Feb 02 '11 at 09:37

4 Answers4

2

This code should run fine (except for the fact that creating new File so many times makes no sense).

I ran your code in a loop 10 million times and it did not produce OOME. In fact it's memory use was constant at about 50Mb.

So, I think the problem should be something else.

Peter Knego
  • 79,991
  • 11
  • 123
  • 154
  • Very fine... could you explain what IDE are you using? What scenario? I mean, JRE/JDK version and so on? We faced something very similar since the problem arises only on some machines and not on others. – Mr.Gate Feb 02 '11 at 09:41
  • Latest Java SDK 1.6 on OS X. Ide is Intellij IDEA 10. – Peter Knego Feb 02 '11 at 11:22
1

I would run your code against a profiler like Visual VM and look to see where the memory is getting used up.

I am guessing that the do "something with bytes code" could be causing a problem. Also calling System.gc() is generally not advised as the garabge collector should know itself when it should be invoked.

Greg McGowan
  • 1,360
  • 1
  • 10
  • 20
  • As I told, in the sample cose nothing is done in the "do something" point but the problem arises all the same. – Mr.Gate Feb 02 '11 at 09:40
  • okay. I missed those comments while I was writing my answer. If you are using java 1.6 then you can use visual vm. It has plugins for netbeans and eclipse. – Greg McGowan Feb 02 '11 at 09:52
0

Since you are effectively throwing away the bytes read from the file on each iteration, I see no reason for getting an OOME unless the size of the file is more than the default Xmx for your JVM process (just a thought process since I know TIF files are notorious for their massive size when compared to other image formats).

Also, print out the iteration number. Do you get an OOME always at the same iteration count? Or is it that you are "using" the bytes read from the file stream?

Sanjay T. Sharma
  • 22,857
  • 4
  • 59
  • 71
  • The iterations number is proportional to the amount of memory you give to JVM, so, it doesn't seem that a particular file causes the problem. – Mr.Gate Feb 02 '11 at 09:49
  • I just forgot to say that those TIF files are pretty small. – Mr.Gate Feb 02 '11 at 10:10
0


After loosing time and healt on this, we found that the problem is due to how the package has been installed on target machine.
Our programmer built a runnable jar, containing every needed jar, instead of using the ones altready installed on the target machine. So... "something" was different, even if we don't actually know what, and the procedure, now, doesn't loose memory anymore.

Does anybody can explain this to me? (I have to say I'm a C/C++ developer, not a Java one, and I'm writing about the work of a collegue of mine).

Mr.Gate
  • 419
  • 4
  • 12