StreamReader is not reading my entire file?

Question

Resolved:

The file on the SD card has a bunch of NULLs at that point in the file.

I am using the Netduino Plus to read text from a file on the SD card. I'm using C# .NET Micro Framework 4.2 and FileReader / StreamReader to get it done. I've read the StreamReader buffer is 512 bytes long. Can you only read 512 bytes of data with the StreamReader and that's it? Here's why I'm wondering and a description of my problem...

Here's an example of the file that I'm reading...

40
3,241,17,17,1000,2000
3,14,92,223,1000,2000
3,101,229,12,1000,2000
3,16,215,228,1000,2000
3,50,240,11,1000,2000
3,232,213,10,1000,2000
3,234,219,219,1000,2000
3,202,13,222,1000,2000
3,240,5,65,1000,2000
3,25,234,3,1000,2000
3,236,5,164,1000,2000
3,26,229,12,1000,2000
3,225,217,18,1000,2000
3,8,229,216,1000,2000
3,49,0,7,1000,2000
3,12,99,190,1000,2000
3,222,7,226,1000,2000
3,12,221,208,1000,2000
3,4,37,227,1000,2000
3,4,122,48,1000,2000
3,88,181,192,1000,2000
3,1,17,222,1000,2000
3,56,235,19,1000,2000
3,236,15,101,1000,2000
3,13,175,231,1000,2000
3,229,218,17,1000,2000
3,9,74,239,1000,2000
3,10,233,17,1000,2000
3,12,73,227,1000,2000
3,234,3,3,1000,2000
3,7,128,110,1000,2000
3,5,209,241,1000,2000
3,8,61,229,1000,2000
3,237,1,238,1000,2000
3,228,19,19,1000,2000
3,16,228,92,1000,2000
3,243,206,14,1000,2000
3,193,3,220,1000,2000
3,236,7,7,1000,2000
3,115,236,7,1000,2000

-The first line indicates how many lines are to follow. -The first number in the following lines indicates how many data items will be read from the line. (in this example there are 3 data elements on each line) -The last two numbers on the line are times in ms.

My code reads this much of the file and stops.

40
3,241,17,17,1000,2000
3,14,92,223,1000,2000
3,101,229,12,1000,2000
3,16,215,228,1000,2000
3,50,240,11,1000,2000
3,232,213,10,1000,2000
3,234,219,219,1000,2000
3,202,13,222,1000,2000
3,240,5,65,1000,2000
3,25,234,3,1000,2000
3,236,5,164,1000,2000
3,26,229,12,1000,2000
3,225,217,18,1000,2000
3,8,229,216,1000,2000
3,49,0,7,1000,2000
3,12,99,190,1000,2000
3,222,7,226,1000,2000
3,12,221,208,1000,2000
3,4,37,227,1000,2000
3,4,122,48,1000,2000
3,88,181,192,1000,2000
3,1,17,222,1000,2000
3,56,235,19,1000,2000
3,236,15,101,1000,2000
3,13,175,231,1000,2000
3,229,218,17,1000,2000
3,9,74,239,1000,2000
3,10,233,17


other lines are not read

It misses the two time values on this readline call and the code breaks out on the assignment of time2 in the code below on a System.IndexOutOfRangeException since linearry is 5 elements long (the null doesn't nicely convert to an int lol) and looks like....

linearray[0] = "3"
linearray[1] = "10"
linearray[2] = "233"
linearray[3] = "17"
linearray[4] = ""

instead of being 6 elements long like the rest of the lines and looking like....

linearray[0] = "3"
linearray[1] = "10"
linearray[2] = "233"
linearray[3] = "17"
linearray[4] = "1000"
linearray[5] = "2000"

All the previous lines are read fine and linearray contains all the data it should. Am I not using StreamReader correctly?

Here's the code....

            FileStream fs2 = new FileStream(FilePath, FileMode.Open, FileAccess.Read, FileShare.None);
            StreamReader sr = new StreamReader(fs2);

            line = sr.ReadLine();

            numLines = Convert.ToInt32(line);

            line = "";
            for (int i = 0; i < numLines; i++)
            {
                line = sr.ReadLine();
                string[] linearray = line.Split(comma);
                numDataElements[i] = int.Parse(linearray[0]);

                for (int j = 1; j <= numDataElements[i]; j++)
                {
                    readData[i][j] = byte.Parse(linearray[j]);
                }
                //clear the rest of the channels
                for (int j = (numDataElements[i]+1); j <= MAXCHANNELS; j++)
                {
                    data[i][j] = (byte)0;
                }

                time1[i] = Convert.ToInt32(linearray[linearray.Length - 2]);
                time2[i] = Convert.ToInt32(linearray[linearray.Length - 1]);

            }

Thanks!

Do any of your calls to `line = sr.ReadLine();` return null? You should get a null value returned when the end of stream is reached. Also, be sure and dispose of your StreamReader when done (the `using` keyword is your friend here). — Eric J., May 03 '12 at 16:49
You could try ReadToEnd instead of ReadLine to load the entire file at once, just to see if you get different results. — mbeckish, May 03 '12 at 16:50
Start by checking the file, or create a clean one to test with. I suspect some invisible control char after the "17". — H H, May 03 '12 at 17:10
@Eric - no it doesn't ever return null, it just reads less than the entire line. — martinarcher, May 03 '12 at 20:25
@Henk - good idea - that might be worth doing. I will have to re-write my parse logic, but that's no big deal. — martinarcher, May 03 '12 at 20:26
I thought about re-creating the file and seeing if that fixes it. It just seems odd the way it reacts. — martinarcher, May 03 '12 at 20:26
@Eric - I'm not sure I catch your drift about the using keyword. Is this a C# thing? I'm a C programmer trying to get up to speed in a .NEt world. ;-) — martinarcher, May 03 '12 at 20:28
"thought about re-creating the file .." - beware that the creating software might be the problem. You need an exact copy of the file and load it into a Hex viewer. — H H, May 03 '12 at 20:59
You were right on. The file on the SD card has a bunch of NULLs at that point in the file. It is being FTPed to the SD card from an Android device. The file on the flash of the Android device looks fine. Hmmmm...I wonder if it is getting trashed during the FTP transfer? — martinarcher, May 03 '12 at 22:00
FTP has binary and text(?) modes. Binary might be better, even for text. — H H, May 04 '12 at 13:30
@martinarcher +1 for the question. You might ought to post your resolution as an answer and mark it as the answer. — MrWuf, May 26 '12 at 03:02

score 0 · Accepted Answer · answered Oct 03 '16 at 12:32

Just to answer this question, there was nothing wrong with the code above. I found a bug in the FTP server library . It was also using a 512 byte buffer and only writing that buffer is is was FULL. So that second buffer that was partially full was never written out to the incoming file. I fixed the FTP library code and all is good!

StreamReader is not reading my entire file?

1 Answers1