Best way to load a large file into arraylist in java

Question

I have a file whose size is about 300mb. I want to read the contents line by line and then add it into ArrayList. So I have made an object of array list a1 , then reading the file using BufferedReader , after that when I add the lines from file into ArrayList it gives an error Exception in thread "main" java.lang.OutOfMemoryError: Java heap space.

Please tell me what should be the solution for this.

  public static void main(String[] args) {
    // TODO Auto-generated method stub
    try {
      FileReader file = new FileReader(
          "/home/dmdd/Desktop/AsiaData/RawData/AllupperairVcomponent.txt");
      ArrayList a1 = new ArrayList();
      BufferedReader br = new BufferedReader(file);
      String line = "";
      while ((line = br.readLine()) != null) {
        a1.add(line);
      }
    } catch (Exception e) {
      // TODO: handle exception
      e.printStackTrace();
    }
  }

Not trying to put 300MB of data in your memory? Or if you really must, increase the maximum allowed `java` (heap) memory by using the `-Xmx` flag. — Veger, Dec 17 '12 at 11:01
If you don't have enough memory to load this, you will need to process the data progressively. You would expect a 300 MB to use about 800 MB of memory as your ArrayList and Strings have an overhead and each character uses two bytes. — Peter Lawrey, Dec 17 '12 at 11:01

score 4 · Answer 1 · edited May 23 '17 at 12:27

4

Naively, increase the size of the heap via the Xmx command line argument (see this excellent answer for some guidance)

This'll only work up to a point though, instead consider structuring your data so that the memory requirements are minimized. Do you need the whole thing in memory at once? Perhaps you only need to test whether an item is in that set, consider using a hash or a bloom filter (etc).

edited May 23 '17 at 12:27

Community

1
1

answered Dec 17 '12 at 11:00

Jeff Foster

43,770
11
86
103

when I tries to print the line it prints eaisly , but when i try to add into array list it throws an exception. – Himanshu Jain Dec 17 '12 at 11:04
What are you going to do with these strings? What problem are you trying to solve? – Jeff Foster Dec 17 '12 at 11:05
1

@user1730833 Your code in a messy comment like that is not very good, please edit your question and add it there... – hyde Dec 17 '12 at 11:07

Murali · Answer 2 · 2012-12-24T07:02:11.947

4

Just increase the heap size of Java

java -Xmx250m

If you running your project from IDE set -Xmx250m in arguments.

250m is 250mb

edited Dec 24 '12 at 07:02

answered Dec 17 '12 at 11:02

Murali

774
6
12

score 2 · Answer 3 · answered Dec 17 '12 at 10:59

2

If you have to have it in memory, you could try increasing the heap size by passing the -mx option to the java executable.

It may also be worth considering the question if you really need all that data in memory at the same time. It could be that you can either process it sequentially, or keep most or all of it on disk.

answered Dec 17 '12 at 10:59

NPE

486,780
108
951
1,012

Is a 300MB file to big to be read all at once in memory nowadays? – Cratylus Dec 17 '12 at 11:00
@Cratylus: I don't think your question has an obvious answer other than "it depends". Besides, it's not at all clear whether the solution would need to scale *beyond* 300MB and, if so, how far beyond. – NPE Dec 17 '12 at 11:04
1

@Cratylus For 64 bit JVM, it's not much, but for 32 bit JVM it may be. If it is 300 MB of 8-bit text on disk, it immediately expands to 600 MB of UTF-16 characters when you read it in as Strings in Java. And then on top of this raw data, you will have the data structure overhead, which is probably minimum 10-20 bytes per line, depending on data structure. So it starts to add up... – hyde Dec 17 '12 at 11:05
tell me the optimized way of reading a file and then adding the contents into arraylist. – Himanshu Jain Dec 17 '12 at 11:05

Subhrajyoti Majumder · Answer 4 · 2012-12-17T11:47:51.637

1

Pass -Xmx1024m to increase your heap sapce to 1024 mb.

java -Xms1024m -Xmx512m HelloWorld

You can increase up-to 4GB on a 32 bit system and on a 64 bit system you can go much higher.

edited Dec 17 '12 at 11:47

answered Dec 17 '12 at 11:00

Subhrajyoti Majumder

40,646
13
77
103

score 0 · Answer 5 · answered Dec 17 '12 at 11:03

0

Use java.nio.file.Files.readAllLines, it returns List<String>. And if you're getting OOME increase heap size as java -Xmx1024m

answered Dec 17 '12 at 11:03

Evgeniy Dorofeev

133,369
30
199
275

score 0 · Answer 6 · answered Dec 17 '12 at 11:05

I agree with @Murali partly this will fix the problem you are facing. But it is advisable to use Caching when handling large files. What if the file size becomes 500Mb in a rare case. Make use of a Caching API like Memcached this will eliminate Memory Outages in JVM.

score 0 · Answer 7 · answered Dec 17 '12 at 11:15

0

If you can: process the file in batches of 10000 lines or so.

read 10k lines process repeat until done

answered Dec 17 '12 at 11:15

opi

244
2
9

Best way to load a large file into arraylist in java

7 Answers7

Linked