I have recently started using libcurl.net with one of my projects as a replacement to the HttpWebRequest and HttpWebResponse classes. The reason I chose to use libcurl.net instead of the managed classes is that libcurl.net mimics the behavior of cURL from PHP and I was porting over some code from PHP. I attempted to use the built-in managed classes, but the CookieContainer class was not capturing all of the cookies correctly from the website that I was trying to capture cookies from. I may end up going back to the managed classes if I can figure out how to capture the cookies correctly.
My PHP script works perfectly fine in capturing cookies so I ported most of the cURL functionality using libcurl.net to my C# project. The problem I'm having is when I have to send more than one request header with the CURLOPT_HTTPHEADER cURL option and I have to use an Slist datatype to pass in more than one header like so:
Slist headers = new Slist();
headers.Append("Content-Type: application/x-www-form-urlencoded");
headers.Append("X-Requested-With: XMLHttpRequest");
easy.SetOpt(CURLoption.CURLOPT_HTTPHEADER, headers);
I sometimes have to fake an AJAX request but it does not seem to pass the X-Requested-With: XMLHttpRequest header with the request as the website I'm scraping does not return any results for these "fake" AJAX requests. If I set the CURLOPT_HTTPHEADER do I need to set the Content-Type header or is that always defaulted to Content-Type: application/x-www-form-urlencoded?
