Read with BinaryReader line by line

Question

I am using the following code to serialize data, from dataTable.

var rows = new List<Dictionary<string, object[]>>();

I am filling the rows from DataTable and placing them in Dictionary. Don't ask why :)

using(var fileStream = new FileStream(@"D:\temp.bin", FileMode.Create, FileAccess.Write, FileShare.None))
using(var bw = new BinaryWriter(fileStream))
{
    foreach(Dictionary<string, object[]> row in rows)
    {
        byte[] bytes = ObjectToByteArray(row);
        bw.Write(bytes);
    }
}

With following method:

private static byte[] ObjectToByteArray(Dictionary<string, object[]> rows)
{
    var bf = new BinaryFormatter();
    using(var ms = new MemoryStream())
    {
        bf.Serialize(ms, rows);
        return ms.ToArray();
    }
}

What I am trying to do is to deserialize line by line, if that is possible with BinaryReader. The problem is I am stuck with reading only first row.

What I would like to achieve is:

using(BinaryReader reader = new BinaryReader(File.Open(@"D:\temp.bin", FileMode.Open)))
{
    int pos = 0;
    int length = (int)reader.BaseStream.Length;
    while(pos < length)
    {
        byte[] v = reader.ReadBytes(pos);
        Dictionary<string, object[]> row = FromByteArray(v);
        // Advance our position variable.
        pos += row.Count;
    }
}

The biggest problem is reader.ReadBytes(XXX) -> what should be the value to read? I don't know in advance that. I need to read whole line and convert to Dictionary. The method I am using for conversion back is:

public static Dictionary<string, object[]> FromByteArray(byte[] data)
{
    BinaryFormatter bf = new BinaryFormatter();
    using(MemoryStream ms = new MemoryStream(data))
    {
        object obj = bf.Deserialize(ms);
        return (Dictionary<string, object[]>)obj;
    }
}

As I said FromByteArray works fine for first line, I am not finding any way to read next line.

When I use BinarryFormatter to serialize complete file, it passes if the file is not that large. If it is OOM occurs. Same stands for deserialization. That is why I want it to serialize/deserialize partially.

Tried everything and searched everywhere. Thanks on helping with this one.

What's your definition of a "line" in a binary file? In a text file, lines are delimited by a carriage return and linefeed. Perhaps when you write the data to the file, you should write out the length of each "line" followed by its data. Then when you read in the file, you would read an integer, then read the number of bytes specified by that integer for each line. — Chris Dunaway, Mar 07 '17 at 20:41

Ofir Winegarten · Answer 1 · 2017-03-07T20:59:28.597

For each iteration save in the file also the length of the following serialized objects.

When reading, every iteration first read 4-bytes (reader.ReadInt32) to get this value and read this much bytes to de-serialize.

I think it should look like this:

using(var fileStream = new FileStream(@"D:\temp.bin", FileMode.Create, FileAccess.Write, FileShare.None))
{
    using(var bw = new BinaryWriter(fileStream))
    {
        foreach(Dictionary<string, object[]> row in rows)
        {
            byte[] bytes = ObjectToByteArray(row);
            bw.Write(bytes.Length);
            bw.Write(bytes);
        }
    }
}


using(BinaryReader reader = new BinaryReader(File.Open(@"D:\temp.bin", FileMode.Open)))
{
    int length = (int)reader.BaseStream.Length;
    while(reader.BaseStream.Position != length)
    {
        int bytesToRead = reader.ReadInt32();
        byte[] v = reader.ReadBytes(bytesToRead);
        Dictionary<string, object[]> row = FromByteArray(v);
    }
}

Read with BinaryReader line by line

1 Answers1