ToArray() function limitation

Question

I am using the .ToArray() method to convert my string to char array whose size i have kept char[] buffer = new char[1000000]; but when I am using the following code:

using (StreamReader streamReader = new StreamReader(path1))
{
    buffer = streamReader.ReadToEnd().ToCharArray();
}
// buffer = result.ToArray();
threadfunc(data_path1);

The size of the buffer getting fixed up to 8190, even it is not reading the whole file after using .ToCharArray() or .ToArray(). What is the reason for this does .ToCharArray() or .ToArray() have size limitations? As if I do not use this function I'm able to read whole file in string format, but when trying to convert it into char array by using this function I am getting size limitations.

Will you try separating the call of `ReadToEnd()` and `ToCharArray()`. (i.e. Put the result of first one in the a string and then call the second on the string.) — Emad, Feb 27 '17 at 07:34
The way you are getting the characters with `ToCharacterArray`... you can declare the array `buffer` without a size and it will be sized to the correct size when read. In other words `char[] buffer;` will work. — JohnG, Feb 27 '17 at 07:37
[ToCharArray()](https://referencesource.microsoft.com/#mscorlib/system/string.cs,81c2d980f5d0ee35,references) converts the whole string to a byte array. It does not have a limit on the size of the generated array. — Serge, Feb 27 '17 at 07:39
First of all, whatever you assign to `buffer` before that code is overwritten so the 1MB buffer you allocated is just left for the garbage collector. The question is, how big is the file? — Lasse V. Karlsen, Feb 27 '17 at 08:08

score 2 · Answer 1 · answered Feb 27 '17 at 07:45

2

My guess is the problem is that read to end should finish before you call the ToCharArray(). This might help you. You don't need to define buffer since ToCharArray() creates a new instance of char[] itself.

string content;
using (StreamReader streamReader = new StreamReader(path1))
{
    content = streamReader.ReadToEnd();
}
var buffer = content.ToCharArray();

answered Feb 27 '17 at 07:45

Emad

3,809
3
32
44

_should finish before you call the ToCharArray_ - Do you mean that `streamReader.ReadToEnd()` return "not finished" string until reader is disposed? – Fabio Feb 27 '17 at 08:58
Dispose is a super set. you may call `streamReader.Flush()` streams have this bad habit of not finishing the job when their functions return. Just an experience of mine – Emad Feb 27 '17 at 09:40

Fabio · Accepted Answer · 2017-02-27T09:45:49.507

ToCharArray() returns new instance of of array. So your buffer will refer to the new instance which is the size of data returned by ReadToEnd.

If you want keep buffer same size just add new array to the existed one

char[] buffer = new char[1000000];
using (StreamReader streamReader = new StreamReader(path1))
{
    var tempArray = streamReader.ReadToEnd().ToCharArray();
    tempArray.CopyTo(buffer, 0);
}

If you want just use the result array - you don't need to "predict" the size of array - just use returned one

public char[] GetArrayFromFile(string pathToFile)
{
    using (StreamReader streamReader = new StreamReader(path1))
    {
        var data = streamReader.ReadToEnd();
    }
    return data.ToCharArray();
}    

var arrayFromFile = GetArrayFromFile(@"..\path.file");

score 0 · Answer 3 · answered Feb 27 '17 at 07:44

You are probably using incorrect encoding. By default StreamReader(String) uses UTF8 encoding:

The complete file path is specified by the path parameter. This constructor initializes the encoding to UTF8Encoding and the buffer size to 1024 bytes.

Don't pre-allocate the buffer size, unless you have a specific need.

If your file is in ASCII format, you need to update your StreamReader constructor:

char[] buffer = null;

using (StreamReader streamReader = new StreamReader(path1, Encoding.ASCII))
{
    buffer = streamReader.ReadToEnd().ToCharArray();
}
// buffer = result.ToArray();
threadfunc(data_path1);

score 0 · Answer 4 · answered Feb 27 '17 at 07:56

Does your file contain binary data? If it contains EOF character and the stream is opened in text mode (which StreamReader does), that character will signal end of file, even if it is not actually the end of the file.

I can reproduce this by reading random .exe files in text mode.

ToArray() function limitation

4 Answers4