How to determine length of the string while deserializing it from byte array?

Question

I use MessagePackSerializer and try to deserialize byte array. But array can be very big (10-20 MBs).

I am reading data into an intermediate buffer of 1000 bytes. And I read the data from them.

But there is a problem: when I try to read a line too long, an error may occur

System.ArgumentOutOfRangeException: Index and count must refer 
to a location within the buffer.

I somehow need to find out that the line I'm going to read goes beyond the bounds of the array and I need to expand the buffer to the size of the line.

How can I do it?

I use this code:

var stringValue = MessagePackBinary.ReadString(bytes, off, out readSize);

Simple example:

public class Example
{


    public void Serialize(Stream inputStream,string value)
    {
        MessagePackBinary.WriteString(inputStream, value);
    }

    public string Deserealize(Stream stream)
    {
        var off = 0;
        byte[] bytes = new byte[1000];
        int readSize = 0;

        stream.Read(bytes, off, bytes.Length);
        var stringValue = MessagePackBinary.ReadString(bytes, off, out readSize); //string can be very long
        return stringValue;
    }
}

You need to use a loop if you want to do it with a smaller buffer. Take a look at [this answer](https://stackoverflow.com/a/19388259/10883465). — Joelius, Aug 14 '19 at 07:11
@Joelies, no. i found solution: i need to write lenght of string in serializer — Admiral Land, Aug 14 '19 at 08:21

score 2 · Answer 1 · answered Aug 14 '19 at 06:31

2

20MB is really not all that much nowadays, not even on phones. Just read the entire array into memory and decode it.

answered Aug 14 '19 at 06:31

zmbq

38,013
14
101
171

it can be 20MB or 20 GB, i do not know size exactly. It depends on data size at user machine – Admiral Land Aug 14 '19 at 06:34
If it can be really large, can you use a memory mapped file, and let Windows worry about allocating physical memory? – zmbq Aug 14 '19 at 07:14

score 1 · Answer 2 · answered Aug 14 '19 at 08:28

1

Solution is simple:

I need to write lenght of string first (at serializer).

off += MessagePackBinary.WriteInt32(inputStream, str.Length);

And when i deserialize read lenght from stream and then make desigion about buffer lenght and string lenght.

answered Aug 14 '19 at 08:28

Admiral Land

2,304
7
43
81

How to determine length of the string while deserializing it from byte array?

2 Answers2