C# - Reading HTTP requests with StreamReader

Question

I am writing a TCP Client and a Server in C# which use manually written HTTP requests to communicate with each other. The trouble I have is with reading from the Network Stream using StreamReader. So far, I have tried many methods but to no avail.

The requests I get from my TCP Client are in in various forms. For updating database, the requests look like this (CRLF is a constant I use to denote "\r\n" string):

HTTP 1.0:

"POST /" + name + " HTTP/1.0" + CRLF + "Content-Length: " + length + CRLF + CRLF + location;

HTTP 1.1:

"POST / HTTP/1.1" + CRLF + hostname + "Content-Length: " + length + CRLF + CRLF + nameLocString;

The requests are of correct form and the client is sending them correctly - I have tested this on a server to which I have access that responded to them without problems.

The problems I have are with my TCP Listener code. To avoid posting the whole code, I will just include the parts of the code that are problematic (found out by debugging).

Server code:

NetworkStream socketStream = new NetworkStream(connection);
StreamReader sr = new StreamReader(socketStream);

string input = ReadAllLinesWithNull(sr); // reading version 1
string input = ReadAllLinesWithEndOfStream(sr);  // reading version 2
string input = ReadAllLinesWithPeek(sr);  // reading version 3
string input = sr.ReadToEnd();  // reading version 4

And the methods used are:

static string ReadAllLinesWithNull(StreamReader sr)
{
    string input;
    string nextLine;
    input = sr.ReadLine();
    while ((nextLine = sr.ReadLine()) != null)
    {
        Console.WriteLine(input);
        input += nextLine;
    }
    sr.Close();
    return input;
}

static string ReadAllLinesWithEndOfStream(StreamReader sr)
{
    string input = "";
    while (!sr.EndOfStream)
    {
        input += sr.ReadLine();
    }
    sr.Close();
    return input;
}

static string ReadAllLinesWithPeek(StreamReader sr)
{
    string input = "";
    while (sr.Peek() >= 0)
    {
        input += sr.ReadLine();
    }
    sr.Close();
    return input;
}

None of these methods for reading worked. With my connection timeouts set, I have been getting IO Exception that it took too long to read/the connection was forcibly closed. I switched off the timeouts and the Read took indefinite amounts of time.

Thanks to using ReadLine()s I was able to single out the place where it ultimately hangs for all versions of protocol and found out that when there is cluster of two CRLFs ("\r\n\r\n"), the Stream Reader is not able to cope with this and gets stuck.

Do you have any suggestions as on how to get around this? I need to use the version with multiple CRLFs as it is in the specification.

If you need any additional information, I will try to supply it as sson as possible.

Can you post code that “is not able to cope with this and gets stuck” because what you have posted works fine for me. Note that `ReadLine` [does not include any terminating returns or linefeeds](https://msdn.microsoft.com/en-us/library/system.io.streamreader.readline%28v=vs.110%29.aspx), so what you have posted will return all the content concatenated with returns and linefeeds removed. — Dour High Arch, Feb 25 '17 at 21:01
@DourHighArch The code I supplied is the code where it is all failing for me. I checked and the Client correctly Writes the HTTP requests mentioned to the stream and flushes them and then when I read using any of the ReadLine methods, I never get anything past the Content-Length: + length + CRLF + CRLF. It either reads indefinitely or throws Exception when I have timeout set. Thanks for the note about ReadLine. I first wanted to get it reading as simple as possible and look at correct reading later. I tested the reading with new simple server (just read and echo) and it had the same problem. — Alexander Rossa, Feb 25 '17 at 21:55

Alexander Rossa · Accepted Answer · 2017-02-27T15:16:00.137

In the end I have found a solution to my problem. Instead of using

static string ReadAllLinesWithPeek(StreamReader sr)
{
    string input = "";
    while (sr.Peek() >= 0)
    {
        input += sr.ReadLine();
    }
    sr.Close();
    return input;
}

I had to use

static string ReadAllLinesWithPeek(StreamReader sr)
{
    string input = "";
    while (sr.Peek() >= 0)
    {
        input += (char) sr.Read();
    }
    return input;
}

I am still not sure why reading the input by lines did not work but when reading it by char at a time, it does.

score 1 · Answer 2 · answered Feb 26 '17 at 21:52

A NetworkStream blocks on a Read operation if there is currently no data available and the other side has not closed that channel yet. TCP by itself has no concept of a message - that problem is to be solved at the HTTP level.

For HTTP you can keep reading until your data contains a \r\n\r\n sequence, which separates the header from the body. How to process the body depends on which headers are present:

Transfer-Encoding: chunked indicates that the sender will send chunks of data and will end with a 0-length chunk
Content-Length should be present when not using chunks, you can then read exactly that many bytes of data
GET requests should not have a body, you can probably assume this if the above headers are not set
Connection: close may be used for responses, indicating that the connection will be closed after all response data is sent

As you can see, StreamReader.ReadLine() would work pretty well on parsing the header, and it is quite suitable for reading chunks too, but it cannot be used for reading a fixed-length body.

I don't know how realiable it would be to read from a stream previously read from by a StreamReader (it can probably read ahead some data to its buffer), but slapping using blocks around them only causes the underlying stream to be closed unless you pick that one constructor overload.

I have been previously wrong, that is why I unaccepted your answer. By following it, my problem just moved further down the line. Thank you for your effort and writing the answer to my question. — Alexander Rossa, Feb 27 '17 at 15:17
I don't mind you unaccepting my answer, however previously you commented that this answer inspired you to call `StreamReader.Close()` instead of using `using` blocks - that's really not the direction I wanted to send you in. If you have questions regarding my answer, you can comment them and I'll update the answer. — C.Evenhuis, Feb 27 '17 at 15:49

C# - Reading HTTP requests with StreamReader

2 Answers2