3

I want to efficiently read specific lines of a file in Java.

As of now, I have been using the method public int read(byte[] b) from the class Random access file. Its fast, but is there any other technique which is much faster.

The file has around 200,000+ lines. I have collected offsets and length of each line. I use this in read(byte) method. I usually have to read around 1-75000 specific lines. It takes more than 50s for this. Is it possible to optimise this?

            RandomAccessFile dicti = new RandomAccessFile(file,"r");
            for(int i:totalLines){
            Long start = getOffset(ID);
            dicti.seek(start);  
            byte[] bytes = new byte[lengthinBytesfortheLine(i)];
            dicti.read(bytes);
            line =new String(bytes);}
faiz fz
  • 41
  • 6
  • Given that you want to read something like 1/3 of the lines, I would just use a BufferedReader and use readLine() in a loop, skipping the lines I don't care about. I expect this to be much faster: your file is probably only 20MB-large. – JB Nizet Jul 15 '19 at 16:50
  • We will probably be better equipped to help you if you can show us the code instead of describing it. – mypetlion Jul 15 '19 at 16:51
  • You may want to use a database; It sounds like you've already created the index for it. What if, instead of reading the file and collecting offsets and lengths of each line, you just dumped it into a database table with the data you cared about extracted as an index column? – Gus Jul 15 '19 at 16:54
  • @JBNizet, Thanks for the quick response. i mentioned one thing wrong in my question. I have to read anywhere between 1-75000 lines. – faiz fz Jul 15 '19 at 17:00
  • 1
    @Gus, I considered that but also wanted to know if there was a solution in Java itself. Thanks for the response. If incase i dont find any, i'll do it using a database. – faiz fz Jul 15 '19 at 17:02
  • @mypetlion, I've updated the question with my current code. – faiz fz Jul 15 '19 at 17:17

1 Answers1

1

You can read line by line and perform some manipulation. I provide below the example.

try (Stream<String> stream = Files.lines(Paths.get(yourFileName))) {

            stream.forEach( line -> performSomeOperation(line));

        } catch (Exception e) {
            e.printStackTrace();
        }

performSomeOperation(String line) is a method where you can do some operation based upon the need.

Sambit
  • 7,625
  • 7
  • 34
  • 65
  • Thanks for the quick answer. In this we load the entire file in to Ram right? This could overload the system. Also, Is this method faster than reading specific lines from a file? I mentioned one thing wrong in the question. i have to read anywhere between 1-75000 lines. – faiz fz Jul 15 '19 at 16:57
  • 1
    No, that won't load the entire file to RAM. https://docs.oracle.com/javase/8/docs/api/java/nio/file/Files.html#lines-java.nio.file.Path-java.nio.charset.Charset- – JB Nizet Jul 15 '19 at 17:00
  • Thanks JB Nizet Sir for answering it.The link is really helpful. – Sambit Jul 15 '19 at 17:02
  • @JBNizet , this sounds interesting. I'll try it out. Thanks a lot. – faiz fz Jul 15 '19 at 17:05