I have a buffered reader that reads a large file line-by-line to remove duplicate lines.
Instead of loading the whole file in the memory I'd like to do this by using two buffered readers: The first iterates over fixed portions of the file, loading each portion one by one into memory.
In each iteration, the second buffered reader would from where the first one stops to the end of the file to check that the loaded portion doesn't exist anymore in the file.
The problem is that I can't make new independent buffered reader object (not reference) to start in the position the first one stopped.
I need a way to find out the first buffered reader's file position so I can tell the second buffered reader where to begin.
What I've tried so far:
Sending the first object to the second's constructor.
This actually worked, but both had the same iterator, so the first one moved with the second one to the end of the file
BufferedReader cleanfilereader2 = new BufferedReader(cleanfilereader);
bufferedReader.mark()
sets the position of the buffered reader but I still need to know the position of the first one.
Notes:
- The number of lines is not constant
- Can't load the whole file in the memory
- Both time and memory are issues