Best way to search a content ( specific ID ) from a huge file ( contains 10 millions of lines ) by using java

Question

I have a huge file containing at least 10 millions of lines and I need to search a particular unique word (ID) from that file using java. Please suggest me the best and fastest way which will consume very less processing time to achieve this.

Is java.util.concurrent package will be useful. If yes please suggest me how?

are you looking for Data Structure option or native JAVA method? — NoobEditor, Oct 15 '14 at 10:45
Is that 1,010,000,000 lines, or 10 million lines? Also, how often will the file change? Is it worth creating an index to the words in the file? — Jaydee, Oct 15 '14 at 10:46
If you need to read the file from a hard drive, the bottleneck is going to be I/O and parallelising won't help... — assylias, Oct 15 '14 at 10:48
That file will be updated very frequently (in every minute). I need the solution to achieve this by using java 7. — user3868764, Oct 15 '14 at 10:50
how will the file be updates? can you index it before/while its gettin updated? — nafas, Oct 15 '14 at 10:52
No Some other application is updating the file, so I cannot add index. My task is just to read and search that ID and do some operation for that ID. — user3868764, Oct 15 '14 at 10:57

score 0 · Answer 1 · answered Oct 15 '14 at 10:51

0

You could split the reading and searching operation into separate threads. That way, you searching will not interrupt your reading. However, as IO will be the major bottleneck, I doubt you will see much of an performance increase.

answered Oct 15 '14 at 10:51

PeterK

1,697
10
20

score 0 · Answer 2 · answered Oct 15 '14 at 11:04

Try the following code: hopefully it'll finish within your time limit of 1min

String search="searchMe";     
String  thisLine = null;
      try{
         // open input stream test.txt for reading purpose.
         BufferedReader br = new BufferedReader("MYFILE");
         boolean found=true;
         while ((thisLine = br.readLine()) != null) {
            if(thisLine.equals(search)){
              System.out.println(search + " was found.");
              found=true
              break;
            }
         }
         if(!found){
             System.out.println("not found");
         }
         br.close();       
      }catch(Exception e){
         e.printStackTrace();
      }

It might be faster if you use some other program to split the file into smaller "temporary" files. and use threads to work on each file.(Not 100% sure if that would work).

Obviously the best solution would be to be able to edit the program that generates this file and make it to create few different files (possibly in different hard drives). then u can easily use threading to increase the speed.

Best way to search a content ( specific ID ) from a huge file ( contains 10 millions of lines ) by using java

2 Answers2