C# OutOfMemory Exception when reading Post Response

Question

I'm currently working on a simple application that utilizes JSON objects to POST to an API and gets the responding data. However, when I run the POST method, the POST response is so large that I encounter the OutOfMemory Exception.

I'm currently using WebClient and a CookieContainer for the process:

string jsonObject ="...."; //Example JSON string - It's very small

using (var client = new WebClient())
{
     var auth = new NameValueCollection();
       values["username"] = "username";
       values["password"] = "password";

     client.uploadValues(endpoint,auth);

     // This is causing the OutOfMemory Exception
     var response = client.uploadString(endpoint, jsonObject);

}

I have looked into the issue, and have set the property AllowStreamBuffering to be false.

 client.AllowStreamBuffering() = false;

However, I am still encountering the issue, and do not know how to control the POST response.

Update: 7/5/2017

Thanks to @Tim suggestions, I've moved to a stream of the response, however I'm encountering issues regarding the actual response. After writing the JSON (as a string) to the end point with a POST method, the script gets stuck at trying to read the response.

    String endPoint = @"http://example.com/v1/api/";
    String json = @"....";

    HttpWebRequest request = (HttpWebRequest)WebRequest.Create(endPoint);
    request.Method = "POST";
    request.KeepAlive = false;
    request.AllowReadStreamBuffering = false;

    /* Pretend this middle part does the Authorization with username and password. */
    /* I have actually authenticated using the above method, and passed a key to the request */

    //This part POST the JSON to the API
    using (StreamWriter writeStream = new StreamWriter(request.GetRequestStream()))
        {
            writeStream.Write(json);
            writeStream.Flush();
            writeStream.Close();  
        }

    //This bottom part opens up a console, but never reads or loads the data
    HttpWebResponse response = (HttpWebResponse)request.GetResponse();
    StreamReader reader = new StreamReader(response.GetResponseStream());

I'm wondering if the JSON is not encoded probably.

(Side Note: I have looked at writing the response, line by line onto a file, but it's the response that is causing the issue -- http://cc.davelozinski.com/c-sharp/fastest-way-to-read-text-files)

The answer here seems related to your issue: https://stackoverflow.com/questions/15163451/system-outofmemoryexception-was-thrown-webclient-downloadstringasynch — Tim, Jun 18 '17 at 14:58
Hi Tim, thanks for the info. I haven't tried to use the HttpWebRequest for getResponse(), but I will update it here if it's the solution. — John Gail, Jun 18 '17 at 16:45
You might need to use a stream, BeginGetResponse should let you grab the whole response in chunks — Tim, Jun 18 '17 at 20:49

score 0 · Accepted Answer · answered Jul 07 '17 at 01:33

I was able to solve my own problem. For the header, I forgot to encode the JSON, and set the content-type properly for the API.

Here is exactly the same code above for the stream, but with an updated header and bufferedStream for increased efficiency with handling data and memory.

String endPoint = @"http://example.com/v1/api/";
String json = @"....";//Example JSON string

Byte[] jsonData = Encoding.UTF8.GetBytes(json);

StreamWriter file = new StreamWriter(@"File Location");

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(endPoint);
request.Method = "POST";
request.ContentType = "text/plain";
request.KeepAlive = false;
request.AllowReadStreamBuffering = false;
request.ContentLength = jsonData.Length;

/* Pretend this middle part does the Authorization with username and password. */
/* I have actually authenticated using the above method, and passed a key to the request */

 //This part POST the JSON to the API
 Stream writeStream = request.GetRequestStream();
 writeStream.Write(jsonData, 0, jsonData.Length);

 //This conducts the reading/writing into the file
 HttpWebResponse response = (HttpWebResponse)request.GetResponse();
 using (Stream receivedStream = response.GetResponseStream())
 using (BufferedStream bs = new BufferedStream(receivedStream))
 using (StreamReader sr = new StreamReader(bs))
 {
       String s;
       while ((s = sr.ReadLine()) != null)
       {
            file.WriteLine(s);
        }

 }

A lot of the methods here come from Dave Lozinski's blog post regarding memory handling, stream reading, and stream writing (although he uses files instead of streams). It's very useful to go through the post above for help.

C# OutOfMemory Exception when reading Post Response

1 Answers1