Reading records from buffers

Question

I have a huge file to be read. I have a IO thread which reads data ( 4mb ) from the disk and stores in a circular array of 6 elements ( 4mb each ). I have another thread which reads from the circular buffer to convert the data into a some records.

The problem is I can have records which spans across 2 different buffers ( ie. say a record can start from the end of 1st buffer and extend upto start of next buffer )

How do I handle such cases ?

Could you point to some sample implementation ?

I take it your problem is knowing which buffer is "the next one"? — Mats Petersson, Mar 05 '13 at 00:10
Yes, I meant that you don't know which one of the 6 elements in your circular array is the one you want? If that's NOT what you are asking for, perhaps you'd like to explain what you are struggling with in a bit more detail. — Mats Petersson, Mar 05 '13 at 00:31
Try writing or drawing your algorithm on paper. Use smaller sized containers (such as only 3 records). What can you learn by using your example as a test case? — Thomas Matthews, Mar 05 '13 at 00:45
Are your records fixed length? Different algorithms for fixed length versus variable length records. — Thomas Matthews, Mar 05 '13 at 00:46

score 0 · Answer 1 · answered Mar 05 '13 at 00:51

Your function for reading from a buffer should read from the next buffer when a record spans two buffers.

More precisely, create a function that assembles a record from data in the buffer. If the data pointer hits the end of a buffer before the record is finished, set the data pointer to the beginning of the next buffer.

Hmmm, looks like this can be applied more generically. Build items by reading from a data pointer. Before the data pointer is accessed, check for end of buffer. If the pointer is past the end of the buffer, set it to the beginning of the next buffer. This concept is very similar to buffered I/O. Hmmm, perhaps you can modify the iostreams or create your own, that will fetch data from your buffers instead of cin or a file. Look at std::istringstream.

Laurent · Answer 2 · 2013-03-05T10:28:56.067

You should split your record reading process in to steps :

convert your buffers chain into an input stream
parse the input stream to produce record

You can use standard classes to achieve the first step as Thomas said or implement your own solution. A trivial solution may look like this (assuming fixed size for records)

class BufferReader{
...
public :
   // this function will read data from buffers. 
   // size of readed data is arbitrary and does not depend on buffer size
   // it will return -1 when eof reached, readed size in other case 
   int readData(char *data, int length);
...
}

Then you can parse your records :

int size = /* size of the record */;
BufferReader br(/* some construction parameters here */)
char data[size];
while(br.readData(data, size) == size){
   // parse your data to fill your record
...

Reading records from buffers

2 Answers2