ByteArrayOutputStream : java.lang.OutOfMemoryError for files greater than 2gb

Question

java.io.ByteArrayOutputStream is giving OutOfMemoryError when i am trying to upload a file greater than 2gb.

I checked the class and found the following code is causing the issue.

 private static int hugeCapacity(int minCapacity) {
    if (minCapacity < 0) // overflow
        throw new OutOfMemoryError();
    return (minCapacity > MAX_ARRAY_SIZE) ?
        Integer.MAX_VALUE :
        MAX_ARRAY_SIZE;
}

MAX_ARRAY_SIZE has been defined as: private static final int MAX_ARRAY_SIZE = Integer.MAX_VALUE - 8;

Is there a way to overcome this issue and upload the files of any size using this class? Or is there any other way to upload the file.

Answer 1

If this is coming from an HttpURLConnection upload as suggested by your tag httpurlconnection, i.e. you are writing to its output stream, you should used chunked or fixed-length transfer mode. Otherwise HttpURLConnection has to buffer the entire upload in memory so as to know the Content-length before it sends any of it. If you let it chunk, or tell it the content-length by using fixed-length, it writes direct to the wire.

Answer 2

Is there a way to overcome this issue and upload the files of any size using this class?

No. There isn't.

You are uploading files into memory, and there is no way that you can load an indefinitely large file into memory. Memory is finite.

(Yea you could get a bigger server with more memory, but that just makes the upload limit larger ... and introduces secondary problems.)

Or is there any other way to upload the file.

Stream it to a local file; i.e. use a FileOutputStream instead of a ByteArrayOutputStream. Now this leaves you with another couple of problems:

1. How to avoid filling the file system with a large upload.
2. How to clean up the uploaded files once they have been processed ... to avoid filling the file system with lots of small uploads.

There are no simple answers. But these are the kind of problems you get when you allow people to upload files to your server.