HttpWebRequest throws exception in code but not browser

Question

I'm currently writing an API that gets data from a Point of Sale System's web interface. So far, I haven't had any problems logging in and generating reports to get data from until this situation.

In general, I can use the follow method to return an HttpWebRequest object that does the trick for most request to the web server.

private HttpWebRequest DefaultRequestObject(string path)
{
    var request = (HttpWebRequest)WebRequest.Create(_baseUrl + path);
    request.Method = "GET";
    request.Host = _baseUrl.Substring(8); // Cut off the https://
    request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:30.0) Gecko/20100101 Firefox/30.0";
    request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
    request.Headers.Add("Accept-Language", "en-US,en;q=0.5");
    request.Headers.Add("Accept-Encoding", "gzip, deflate");

    request.SendChunked = false;
    request.AllowAutoRedirect = true;

    request.ProtocolVersion = HttpVersion.Version11;

    var sp = request.ServicePoint;
    sp.Expect100Continue = false;
    var prop = sp.GetType().GetProperty("HttpBehaviour", BindingFlags.Instance | BindingFlags.NonPublic);
    prop.SetValue(sp, (byte)0, null);

    request.CookieContainer = _cookieJar;

    if (!String.IsNullOrEmpty(_cookieString))
        request.Headers.Add(HttpRequestHeader.Cookie, _cookieString);

    return request;
}

When I send a GET request, I use the following method:

public MgrngResponse GetContent(string serverPath, Dictionary<string, string> args)
{
    if (!serverPath.StartsWith("/"))
        serverPath = "/~pos/mgrng/" + serverPath;

    var requestString = serverPath + "?" + HttpDataFormat(args);
    var request = DefaultRequestObject(requestString);

    try
    {
        var response = (HttpWebResponse)request.GetResponse();
        var mgrngResponse = new MgrngResponse(response);

        if (!String.IsNullOrEmpty(mgrngResponse.HttpResponse.GetResponseHeader("Set-Cookie")))
            SaveMgrngResponseCookies(mgrngResponse);

        _sessionExpiration = DateTime.Now.AddMinutes(15);
        UpdateStatus(mgrngResponse.Content);

        return mgrngResponse;
    }
    catch (WebException webException)
    {
        using (WebResponse response = webException.Response)
        {
            var httpResponse = (HttpWebResponse)response;
            Console.WriteLine("Error code: {0}", httpResponse.StatusCode);
            using (Stream data = response.GetResponseStream())
            using (var reader = new StreamReader(data))
            {
                string text = reader.ReadToEnd();
                Console.WriteLine(text);
            }
        }
        var eventArgs = new SessionUpdateEventArgs(SessionStatus.ConnectionError, "Unable to GET data");
        RaiseStatusChangeEvent(eventArgs);
        return null;
    }
}

This works well for all of the pages I've attempted so far, but now I'm running into a new problem where when I try to get the response for a particular page, the method throw a WebException with a 500 Internal Server Error.

I used Fiddler to match a browser's request exactly with mine (with the exception of the order of the headers and the cookie values obviously) but I'm still getting the Internal Server Error.

Below are the Raw Requests from Fiddler. The first one is from Firefox and the second one is from my program.

GET https://location.posprovider.com/~pos/mgrng/Report.php?boc_brand=7&csv_delimeter=csv_delimeter_comma&format=text&format1=csv&format1=txt&format1=pdf&format1=html HTTP/1.1
Host: location.posprovider.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:30.0) Gecko/20100101 Firefox/30.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Cookie: USER=af182fda473354eb3199522726ca61c9d5516c95f165cffd63b9522726ca61c9dc714cb52a46278e4399720706ea41e9dc714cb52a46278e4399720706ea41e9dc714cb52a46278e4399720706ea41e9dc714cb52a46278e4399720706ea41e9fc516c950a6607ae63b9522726ca61c9; PHPSESSID=9d7f54f9a1769a3e0572745fe0db3d97
Connection: keep-alive

GET https://location.posprovider.com/~pos/mgrng/Report.php?boc_brand=7&csv_delimeter=csv_delimeter_comma&format=text&format1=csv&format1=txt&format1=pdf&format1=html HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:30.0) Gecko/20100101 Firefox/30.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Host: location.posprovider.com
Cookie: PHPSESSID=16ad21c9d69fe85b3d185ab284f8620b; USER=30dca66f355e0ba89b6eda3c3e822ea24a95e5209e0f90bec94eda3c3e822ea243b5c500582b78cde96efa1c1ea20e8243b5c500582b78cde96efa1c1ea20e8243b5c500582b78cde96efa1c1ea20e8243b5c500582b78cde96efa1c1ea20e826395e520780b58edc94eda3c3e822ea2
Connection: Keep-Alive

I even tried logging into the web interface and then copy and pasting my generated request string into the browser and I got the desired data.

Any ideas?

Pull your programs request into Fiddlers composer and gradually eliminate the remaining differences. What do you find doing that? — usr, Jul 17 '14 at 17:45
@usr - Great advice! Apparently there's something wrong with my cookie line. I copy and pasted the cookie line from the browser's version into mine and it worked. Now I just need to resolve why my cookies aren't correct... — Ben, Jul 17 '14 at 18:35

score 0 · Accepted Answer · answered Jul 17 '14 at 19:21

0

Pull your programs request into Fiddlers composer and gradually eliminate the remaining differences. What do you find doing that?

Great advice! Apparently there's something wrong with my cookie line. I copy and pasted the cookie line from the browser's version into mine and it worked. Now I just need to resolve why my cookies aren't correct...

I agree with that diagnosis. Who knows what's going on inside of the server. Probably some fragile code that expect a very exact cookie string format.

answered Jul 17 '14 at 19:21

usr

168,620
35
240
369

It turns out that I needed to make a request to a manage.php file with some session specific parameters and it returns a new cookie value that allowed me to get the desired result from the Report.php request. – Ben Jul 21 '14 at 22:29

HttpWebRequest throws exception in code but not browser

1 Answers1