0

I have very less experience of working on large files and constraints about memory. Please note that I am using Java as programming language.

I have to find records in file size of 15GB and then write result to another file. Basically it is a search range function implementation. My goal is to write the records that lie in my range to result file separated by new line. That is each record should come on separate line. And the number of such queries is around 400-1000. So each query will result in writing many records that lie in its range.

So which of the following is faster approach?

1- As soon as I start getting my query range records, I start writing them in File serially?

2- I store my records in some string, and append in string for each record found, and then in the end write it?

3- Use string builder to concatenate result of each query and then write to file at the end? but string builder doesnot support new line.

4- Make my own datastrucuture ?

5- Make an array list of records and then iterate over it at the end to write to file?

Please let me know which approach is fastest both in terms of IO and MM.

Lightness Races in Orbit
  • 378,754
  • 76
  • 643
  • 1,055
Basmah
  • 829
  • 3
  • 13
  • 25
  • Well, we can't use Database as it is a project to build database in Java, so we cant use any libraries that provide any indexing or Database related functionality. Sorry, what do you mean by not marked answered questions? are you talking about the questions other than this that I asked on Stack overflow? Thank you ! – Basmah Nov 30 '11 at 07:56
  • Not saying StringBuilder is the best solution, but I'm almost positive it supports newline – Esben Skov Pedersen Feb 10 '12 at 10:39

3 Answers3

4

OS level write buffering is quite effective. Go with #1 unless you have profiler data suggesting there's a problem.

Ben Voigt
  • 277,958
  • 43
  • 419
  • 720
0

2 and 3 both sound like home-grown implementations of buffering. A simpler and more effective approach would be to do #1 but to wrap your FileOutputStream with a BufferedOutputStream.

Number 5 is probably a bad idea if you're talking about a very large number of records, as you may actually cause swapping (which will only hurt performance).

Laurence Gonsalves
  • 137,896
  • 35
  • 246
  • 299
  • The approach I am using till now is that I was writing the records to my buffer of size equal to PageSize (4096K) and then using Java Random Access File. write chars to write one page size string buffer. But seems the approach told by you is much faster. So, now I will try using this. Thank you ! – Basmah Nov 30 '11 at 07:41
0

I think Map and Reduce is best suited for this. I havent worked on this but attended a session on the same and the guy used the same example for this. Split your file up and let multiple programs run the same search query - Map and then once the search is done all the answers can be combined into one using Reduce

sethu
  • 8,181
  • 7
  • 39
  • 65
  • Actually my key is not unique. My data has only two columns , one is key and the other is value. The key is variable size ranging from 7-15 digits and keys are repeated in data as well. So i doubt if I can use hashing for key as MapReduce uses hashing for keys. Thank you very much for the suggestion. – Basmah Nov 30 '11 at 07:58